YouTube Summaries: Object Management

We are back with another educational course in our rapidly expanding repertoire on our YouTube channel. This month, MinIO’s Will Dinyes is discussing Object Management for those who have set up their object store and want to learn more about how to set up an efficient and sustainable data lifecycle management strategy.

This 11-part series spanning just over an hour of content covers object versioning, locking and retention, the ins and outs of lifecycle management, and includes hands-on labs performed in test environments for you to follow along. It is meant as a comprehensive summary, but each video can be watched independently if you just need to brush up on specific topics. By the end of this course, you should have an understanding of versioning your object store, setting parameters to ensure objects can only be accessed and deleted by permissioned users, managing the lifecycle of your objects, and replicating your data.

First up is object versioning, the (usually) first step after creating a bucket. Here Will shows how to keep a history of objects through versioning, the storage implications of doing so, and how to use delete markers to delete objects when you have a versioned bucket. The obvious benefit to versioning is being able to delete or overwrite objects in a bucket while keeping their history, but it’s important to keep in mind that every time an object is written to a versioned bucket storage requirements will expand linearly (the size of the object times the number of times written).

Will also shows how to use delete markers to delete different objects using the specific version ID. Delete markers allow you to mark objects for deletion without actually deleting them when you create the marker, ensuring that any permanent deletions require a very specific action. This overview of versioning is followed by a lab in which he walks you through each of the above actions in a test environment.

Next is object locking, also called retention, to prevent accidental deletion or overwriting. He starts off discussing WORM (write once, read many). “Write once” means that if you upload the same object to the same bucket it will be overwritten or a new version will be created—no modifications will be made to an object once it’s uploaded to MinIO, as it is only written once. “Read many,” on the other hand, means that once an object is in MinIO it is immutable, and the same object will always be read unless it is deleted. This is where object locking comes in. Enabling object locking on a bucket at bucket creation allows you to render an object undeletable, and to create rules about how the object is locked.

The first type of rule is duration-based, which describes either governance mode or compliance mode. Governance mode prevents non-privileged users from deleting an object, while compliance mode is stricter in that it prevents anybody, including the root user, from deleting an object until the retention lock expires. The second type is legal-based, which are not time bound and prevent deletion by anyone until the hold is lifted. These two types of rules can be used in conjunction with each other to ensure nothing gets deleted unless it is entirely approved and necessary. Again, Will follows this video with a lab demonstrating how to perform each of the above actions.

Will’s third topic is object lifecycle management, a two-parter about how to manage objects as your storage grows. In the first, he talks about object expiration, object tiering, and lifecycle options. With object expiration, you can set an object to delete permanently after it reaches a certain age. No delete marker is created, and the object will not be replicated. If you do not want your object deleted, your other option is to have it transition to a different server (tier). This is not replication—the object is simply transitioned—but the object can still be accessed by the original server. Using tiering, you can make sure your most important data is on your fastest servers while maintaining access to all your data.

Part two discusses concerns around versioning and lifecycle management. Since versioning creates more and more objects linearly, as discussed above, it’s important to manage the lifecycle of older versions to maintain clean object storage. Lifecycle management has a different set of flags that allow you to work with older versions, and using both versioning and lifecycle management in conjunction with one another lets you set rules on expiration, transitioning, delete markers and more to make this maintenance simple. As always, Will includes a lab on lifecycle management so you can see exactly how this all works.

Finally, there is another two-parter on object replication, which can be used for disaster recovery and data migration. In the first video, Will goes over bucket replication. Replication can be either client-side or server-side, and can be active-passive, active-active, or multi-site active-active. This all requires versioning, and is an extremely powerful tool—replication is not just limited to objects, but includes locking, delete markers, and most of the other concepts discussed in this course.

In part two, Will goes over site replication, with which you can replicate entire sites and not just buckets. This is where our multi-site active-active replication comes in. You don’t just replicate your objects and buckets, you replicate everything: bucket modifications, locks, encryption, and security credentials and keys. Though it is a bit more complex than simple bucket replication, it is an absolutely invaluable tool for disaster recovery. After this deep dive is—you guessed it—a two-part lab on bucket and site replication.

As the data in your MinIO deployment grows, it is incredibly important to keep good object management in mind to get the most out of object storage—we hope this series helps. As always, feel free to watch our other courses to learn more, or reach out with any questions. For more detailed information about installing, running, and using MinIO in any environment, please refer to our documentation here. To learn more about MinIO or get involved in our community, please visit us at min.io or join our public slack channel. If you want to see the other summaries, you can use the YouTube Summaries tag.