Decommissioning Server Pools with mc admin decommission

Hardware eventually reaches the end of its useful life and must be taken out of service before failures present risk to data availability and integrity. Data must therefore be removed proactively to prevent disruption to dependent applications. MinIO recently added the ability to retire and upgrade server pools without compromising data availability. While decommissioning is not a commonly requested feature, we are providing it for those of you who are planning across the entire storage lifecycle. When making key purchases, infrastructure teams like to plan in advance, and MinIO has you covered.

This post shows you how to decommission a server pool, view the status of a decommission in-progress or cancel it.

Use Case: Retiring a Server Pool

Let’s say you have a MinIO cluster made up of one server pool with four nodes. This server pool hardware is approaching end-of-life. You purchase a new, second server pool and wish to add it to the cluster to replace the first, while maintaining data availability. The mc admin decommission command will first lock the first server pool as read-only, then drain objects onto the new pool. In a multi-pool MinIO environment, decommissioning spreads data across all pools equally.

We’ll pick up after having installed the hardware and added the new server pool to our MinIO cluster.

Decommission the Original Server Pool

Open a Terminal and use the MinIO Client to begin.

$ mc admin decommission start alias/ http://minio{1...2}/data{1...4}

Check Overall Decommissioning Status

Depending on the amount of data you’re moving, the process may take some time. You can check the current state of decommissioning across all pools with:

$ mc admin decommission status alias/

And you will see something like:

| ID  | Pools                          | Capacity                         | Status  |
| 1st | http://minio{1...2}/data{1...4}| 439 GiB (used) / 561 GiB (total) | Active  |
| 2nd | http://minio{3...4}/data{1...4}| 329 GiB (used) / 421 GiB (total) | Active  |

Check Server Pool Decommissioning Status

You can also see a more detailed status on a per-pool basis, for example:

$ mc admin decommission status alias/ http://minio{1...2}/data{1...4}
Decommissioning rate at 36 MiB/sec [4 TiB/50 TiB]
Started: 1 minute ago

If you request status from a pool that is not decommissioning, you will receive an error:

mc admin decommission status alias/ http://minio{1...2}/data{1...4}
ERROR: This pool is not scheduled for decommissioning currently.

Cancel Server Pool Decommissioning

Optionally, should you decide not to decommission the original server pool, you can cancel the process. This might be used if the load is too high and you want to schedule the decommission for off hours:

mc admin decommission cancel without an argument, lists out any on-going decommission in progress.

| ID  | Pools                          | Capacity                          | Status  |
| 1st | http://minio{1...2}/data{1...4}| 439 GiB (used) / 561 GiB (total)  | Draining|

Please note that cancelling decommission does not make the pool active again in order to avoid having partial namespace across pools.

$ mc admin decommission cancel alias/ http://minio{1...2}/data{1...4}
| ID  | Pools                          | Capacity                          | Status            |
| 1st | http://minio{1...2}/data{1...4}| 439 GiB (used) / 561 GiB (total)  | Draining(Canceled)|

In the unlikely event that decommission fails, the status will indicate decommission as failed:

| ID  | Pools                          | Capacity                          | Status          |
| 1st | http://minio{1...2}/data{1...4}| 439 GiB (used) / 561 GiB (total)  | Draining(Failed)|
| 2nd | http://minio{3...4}/data{1...4}| 329 GiB (used) / 421 GiB (total)  | Active          |

You can restart a cancelled or failed decommission with:

$ mc admin decommission start alias/ http://minio{1...2}/data{1...4}

When is Decommission Complete?

The status command will also tell you when decommissioning is complete:

$ mc admin decommission status alias/
| ID  | Pools                          | Capacity                          | Status    |
| 1st | http://minio{1...2}/data{1...4}| 439 GiB (used) / 561 GiB (total)  | Complete  |
| 2nd | http://minio{3...4}/data{1...4}| 329 GiB (used) / 421 GiB (total)  | Active    |

Remove Server Pool

The final step is to remove the first server pool from your MinIO deployment. Remove this server pool’s address from your MinIO startup command.

On baremetal deployments, if you started MinIO like this for all three server pools:

$ minio server http://minio{1...2}/data{1...4} http://minio{3...4}/data{1...4} http://minio{5...6}/data{1...4} 

then you can remove the first argument to update your MINIO_VOLUMES setting:

$ minio server http://minio{3...4}/data{1...4} http://minio{5...6}/data{1...4}

and then use systemctl restart minio on all of the servers in your deployment in parallel.  

On Kubernetes deployments, MinIO the statefulset specification needs to be modified by changing the command line input for the MinIO container. Once the relevant changes are done proceed to execute kubectl apply -f statefulset.yaml.

On Operator based MinIO deployments you need to modify the tenant.yaml specification and modify the pools: section from two entries to a single entry, once relevant changes are done proceed to execute kubectl apply -f tenant.yaml.

Experience MinIO

You can remove and upgrade hardware non-disruptively on MinIO clusters with a single command. This not only makes life easier for infrastructure admins, it also ensures that your object storage solution can outlive its hardware through the use of MinIO’s flexible server pool architecture.  

You can try this out by downloading MinIO. If you’d like to know more or have any questions,  join our Slack Channel, drop us a note at hello@min.io or use the Ask an Expert button