MinIO, Podman, and Apple Silicon

Will Dinyes -

I've been a Mac guy for a long time.  As in, "last millennium" long time.  As such, I've always been willing to find ways to make software work on my Macs whether it was a good idea or not.  But when it came to servers, I was all in on Linux, which also meant there was a certain insanity of finding working solutions.  The switch to OS X (now macOS) over a decade ago was a very welcome event in my world, because it meant I could use all that Linux scripting goodness directly on my Mac without virtual machines, or dual booting, or, frankly, compromises.  

Then came containers and cloud and all that goes along with it, including that shift in mentality that you can just spin up a server when you need it, and let it go away cleanly when it was done with its work.  MinIO works fantastically well in this environment!  And when I joined MinIO, I was enthusiastic about getting some MinIO containers working on my Mac.

Homebrew Makes It Simple

For those of you not familiar with Homebrew, it's billed as the "missing package manager" for Mac.  I cannot say this loud enough for the developers reading this: get Homebrew.  Do it.  First, all of my notes from this point on are predicated on having Homebrew, and second, it's absolutely the best way to get tons of packages in a simple way that integrates with the OS without hurling a bunch of files all over your system.  Even, and I say this with a great deal of excitement, MinIO itself can be installed from Homebrew, if you want to run bare metal, with a simple brew install minio/stable/minio command.  Everything resides in /opt/homebrew (Apple Silicon) or /usr/local (Intel).  The command to install Homebrew, if you're ready:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

But I'm not here to sing the praises of running MinIO directly on my Mac hardware.  Mainly because that's so simple, I don't think it'd be worth a blog post.  This post is here  mostly because my goal was to have a repeatable process that anyone could use to spin up their own environment quickly on any platform.  And that's where containers come in.

Podman or Docker

Docker is an amazing platform.  In fact, when I first spun up my MinIO container, it was using Docker.  So why not just do a post about running the container on Docker? Well, again, it's easy.  In fact, most of the commands I run below for Podman could easily be done with Docker, simply by swapping the command from one to the other.  Docker is also well integrated with Macs, and in fact has some benefits over Podman when running on a Mac (native containerization being the main one), but I was never one to do things simply.  Back in the 90s, the question wasn't "why not just use Docker?" it was "why not just use Windows?"  Before that it was "why not just use emacs?"  And before that "what have you got against punch cards?"  This is a never-ending cycle, and the answer is always "because." You may as well ask me why I still keep a working Mac Plus around.  Because I like it!

I also really like Podman.  It's well integrated with the Linux kernel, and it just kind of tickles me that to run it on a Mac, I'm actually spinning up a little Linux instance which then runs Podman for me.  The commands are deliberately written to be essentially the Docker commands (in fact, the Podman website advocates simply aliasing "docker" to "podman" and then see who actually notices), making migration from what I was already familiar with very simple.  And, not for nothing, I'm a command line person.  To run Docker on my Mac means Docker Desktop, which I don't actually use day to day, so why not skip what I don't need?

On a Mac, Podman can be installed via Homebrew:

brew install podman

This will install a bunch of supporting software, notably the qemu command.  Once you've got Podman installed, you need to have a Linux instance running.  Your Podman containers are going to be running in a little Fedora VM via qemu.  Now, here's where it gets a little weird.  Because of this extra layer, you need to mount the local drive into the Fedora instance.  Fortunately, there's a flag to allow Podman to use a local drive on a Mac when you initialize the machine:

podman machine init -v /local/path/to/Minio/data:/Minio/data

This is important!  You don't want your data stuck inside the machine that's doing the Podman work, because that machine can (and possibly should) be reaped on a regular basis.  By letting Podman load the data from a local directory, you can access that data whether the container is running or not, and even re-use it in other containers as needed.

Once the base machine is initialized, then run:

podman machine start

This will take a moment, and may even seem like it's hung up on Waiting for VM …  Patience is a virtue, and soon you will see Machine "podman-machine-default" started successfully. During this process, you may also encounter some messages about running rootless (you may ignore this, since MinIO runs fine in a rootless environment, only needing ports 9000 and 9090 to be exposed) or running the Docker API connector (generally useful, so follow the steps, particularly if you're migrating over from Docker in general — not that this post is encouraging that in any way).

Once the Fedora instance that you'll actually run Podman on is up and running, start a MinIO container with:

podman run \
	-p 9000:9000 \
	-p 9090:9090 \
	--name minio \
	-v /Minio/data:/data \
	-e "MINIO_ROOT_USER=admin" \
	-e "MINIO_ROOT_PASSWORD=ifyouusethispasswordsupportwilllaughatyou" \
	quay.io/minio/minio server /data --console-address ":9090"

While this may appear to take over your terminal window, you can in fact hit that Ctrl-C, and get your prompt back, and your container will still be running.  This is a slight difference from Docker, and sort of cuts against the grain of my experience with Linux in general, but, we should all take it in stride.  

Since it's a MinIO container instance running in a Fedora VM on top of our Mac, what we're really doing here is mounting the local filesystem (/local/path/to/Minio/data) into the Fedora-based Podman machine (/Minio/data) and then telling MinIO to mount that as /data. The server then uses that directory to store all of the objects that we upload to MinIO.  

Now we come to a big issue: this results in slow data access because you're mounting a remote disk in the base Linux instance and then asking a Podman container to access it.  Two layers of virtualized disk access may be too slow for MinIO to really function, unless you're using fast hardware.  How fast is fast enough?  Let's put it this way: my 2012 Intel iMac couldn't do it with the data being stored on a USB-connected external drive, but on my 2022 MacBook Pro, with the onboard SSD and M1 Pro chip, MinIO-via-Podman is functional enough to do pretty much everything I'm asking it to, so far.  Of course, there's a lot of daylight between those two Macs: 10 years;  a completely different processor architecture;  vastly different storage speeds.  So, your mileage will absolutely vary.  But if you're storing a significant amount of data in a large number of objects, you may see some timeouts.  What this should tell you is that this whole Podman on Mac exercise is really for development purposes only and should never be allowed off your MacBook into the wild, or worse, production. [Note: the author did not just say that Podman isn't good for production - Podman on an otherwise day-to-day use Mac isn't good for production.]  The benefit is, if you outgrow Podman on your Mac, you can still just take your data (and all your server settings) with you to the next container, because the data is right there on your local filesystem.

There's another good reason not to do this in production: server time drift.  If my MacBook goes to sleep while Podman is running, then my Mac system time will begin to deviate from the Fedora VM system time.  This is a problem because MinIO looks at the time a request is made from the client and compares that to the time when it's being processed by MinIO on the server.  If there's a gap, MinIO refuses to perform the request.  Obviously this is a security measure, and part of MinIO's implementation of S3-compatible storage, but one you may run afoul of in your Podman-based development environment.  Maybe this is something Podman may address in future, syncing the Fedora VM time after some sleep happens, or maybe you can spend some time setting up a proper NTP client on the VM, or maybe it just is what it is, because in a real production environment, the host is unlikely to go to sleep like my MacBook does. In any case, the workaround is to try turning it off and on again, because it's a container, and that's what containers are for.  But shut it all down.  Here are the steps:

podman stop minio
podman machine stop
podman machine start
podman start minio

This works because the podman run command from earlier called the container minio.  If you name it something else, use something else.

At this stage, you have a working MinIO instance that you can send data to, and pull data from.  If you're building a POC, this is probably enough to show off your thoughts, to get that first iteration done, to begin development.  And that's always been my goal: how fast can I get from nothing to something worth showing off.  Once your application matures, though, you will need to pivot to something more robust.

Podman Care and Feeding

At some stage, you're going to want to clean all this up and move on to other things.  This, frankly, is where Podman shines brightest to me.  The usual Docker style commands are all there to show you how to remove your containers and images:

podman ps -a
podman rm
podman images
podman rmi

But what about the Fedora VM?  This command handily shows you what's running:

podman system connection list

And if you want to delete everything, once you've shut down your MinIO instance and the Fedora VM, run:

podman machine rm

This feature right here makes Podman my choice, because there's a lot less mess left lying around.  If you are really looking to clean up, check your home directory, under .config and .local.  All that's likely to be left are some empty directories and maybe the Fedora qcow2 file.  Since Fedora updates regularly, Podman will update this file automatically.  But since Podman is rootless and daemon-less, there's no system services to clean up, no files in protected directories to hunt down and delete, and no system resources being used when you're not actively running the containers.  That's why I like it.  That's why I use it.  It does what I need, and no more.

Another thing I like is that Podman also updates regularly.  Since we're on Homebrew, though, updating that is as simple as running:

brew upgrade

And what about MinIO?  Well that container image moves right in time with all our releases, so it's worth pulling down a new one once in a while!  When?  That's where you want to spend some time with the MinIO image history:

podman image historm minio

Check the CMD ["minio"] entry.  If it's CREATED entry is much more than a week, you're likely not running on the latest build of MinIO!  Pull a new one with:

podman pull quay.io/minio/minio

And then it's just a matter of stopping and starting your MinIO container to get all the new goodness.  What about the data?  It's all still there.  Unless you delete the local directory, your data will remain.

Let me repeat that:

Unless you delete the local directory, your data will remain.

That means you can absolutely migrate to Docker, or bare metal, with ease, when you need to.

Does This Make Sense?

Nope.

I'm serious.  This doesn't make sense.  It's hacky.  It's fragile in some places.  And it's absolutely more work than just installing Docker Desktop.  Or even just running MinIO bare metal on a Mac, which can be done with a simple:

brew install minio/stable/minio

But it's also a challenge, sort of a "because it's there" style bet with myself to see if I could make it work.  And great for a simple, quick, functional development environment that cleans up in a snap.  It works with our command line client, which you can install with:

brew install minio/stable/mc

And at the end of the day, it's got some geek cred.  So why did I do this?  Because I like Macs.  I like Podman.  I love me some MinIO as well, of course.  And I like doing things the hard way sometimes, even if it doesn't make sense.  Tomorrow, I'll go back to writing training for MinIO, and I'll be more focused on the best way, the easy way, the repeatable way, the way that Makes Sense.

But just for today, this was fun.