Object Locking, Versioning, Holds and Modes in MinIO

MinIO supports a complete object locking framework offering both Legal Hold and Retention (with Governance and Compliance modes). Object Locking functionality is a requirement for many regulated industries from financial services to healthcare.

Lifecycle management is an increasingly critical element in the data ecosystem. Data is the primary asset in most organizations at this point - more so than the physical assets we normally associate with large enterprises - trains, planes, automobiles or oil rigs. Protecting this data, and managing how long to protect it are complex challenges.

MinIO now offers the complete suite of lifecycle management capabilities. This includes versioning, object locking and the various derivative components. Every use case is different and our goal is to create software and documentation that allow our community to implement these features with confidence.

Our approach models Amazon’s. This is important for a number of reasons. First, for companies seeking compatibility for their applications, the same S3 API calls that work in the public cloud need to work on premises. Second, Amazon took a measured approach to the problem, considered all of the corner cases and built a strong offering. We acknowledge a debt there. Third, given Amazon’s leadership there is strong adoption of this in the approach and most of the issues have been resolved.

Even with Amazon’s leadership, not a lot of companies have implemented this feature. We are pleased to add our name to the list. Given the other attributes of our software, we think this removes any hesitation around adopting MinIO as your private cloud object store.

Having immutability and protection from deletion ensures that your data assets are safe.

Versioning

Versioning allows a user to retain multiple variants of an object in the same bucket. Versioning provides a mechanism to  preserve, retrieve, and restore every version of every object stored in a bucket. Versioning ensures that objects remain available across a range of failures from applications to human error.

Versioning is enabled at the bucket level. When enabled, MinIO automatically creates a unique version ID for the object. There can be multiple versions of the same object.

One of the main benefits of versioning is to protect overwriting it or deleting it unintentionally. This is performed using the concept of a delete marker. When a versioned object is deleted it is not removed permanently. Rather a delete marker is created and becomes the current version of the object. When that object is read it returns a 404 Not Found message.

Objects that have the delete marker can be restored.

Along the same lines, if a versioned object is overwritten, it creates a new version and that becomes the current version. Again, the older version can be restored as needed.

To account for these changes there are now three bucket states in MinIO. Unversioned which is the default, versioning-enabled, or versioning-suspended. Please note that once a bucket is enabled for versioning, that action cannot be undone - only suspended. Versioning is a global setting within the bucket - meaning all objects are now versioned.

Users with appropriate permissions can suspend versioning to stop accruing object versions. This is applicable only for that bucket and not for other buckets. For more information on what happens in this case refer to the documentation.

As with all of MinIO, versioning can be applied using the MinIO Client (mc), the SDK or via the command line.

Versioning is the simplest way to protect data from unintended actions. It does have the effect of adding to bucket sizes as objects are versioned and can lead to a higher number of interdependencies between the objects as well as the risk of hidden object dependencies. This can be mitigated with Lifecycle Management Choices. Still, for most organizations, this is a reasonable tradeoff for data protection.

There are, however, other use cases, that require another level of certainty.

Object Locking, Retention and Legal Holds

Object locking is a key feature for those companies operating in regulatory environments or having forward thinking policies around retention, audit or other compliance related requirements. We will be publishing our final Cohasset Report shortly but have a favorable assessment in hand.

This work builds on our previous feature set for write once, read many (WORM) - extending it as appropriate. As always, we took a minimalist approach - ensuring that it was feature complete without being feature-heavy and sapping performance.

While there are dozens of use cases, they can be effectively reduced to Legal Hold and Retention. Additionally there are modes, either Governance or Compliance and these have certain implications. There is some overlap here so we will do our best to be precise in our descriptions.

Retention

Retention ensures that an object is WORM protected (cannot be deleted or overwritten) for some period of time. It can be expressed in seconds, days or years.

Retention is set at the object level. You can place a retention period on an object version either explicitly or through a bucket default setting. A default lock configuration set at the bucket level does not apply retroactively to versions of objects created before. It only applies to objects that are created afterwards.  When you apply a retention period to an object version explicitly, you specify a Retain Until Date for the object version. The Retain Until Date setting is stored in the object version's metadata and protects the object version until the retention period expires.

When the bucket default setting is used, a Retain Until Date is not specified. Instead, a duration is set (either days or years), for which every object version placed in the bucket should be protected. When a new object is placed in the bucket a Retain Until Date is calculated for the object version by adding the specified duration to the object version's creation timestamp. The object version is then protected exactly as though it was explicitly locked with that retention period on the object version.

When you place a retention period on an object version, a timestamp is stored in the object version's metadata to indicate when the retention period expires. After the retention period expires, the object version can be deleted unless a legal hold was also placed on the object version.

Explicit retention mode and period specified while creating an object version always overrides the bucket default settings.

The retention period can be extended by submitting a new lock request.

There are two types of modes in the Retention framework:

Governance Mode

When setting a retention period for your objects or buckets Governance mode is used when the goal is to protect objects from being deleted by standard users. Having said that, there are users that will retain the permissions required to modify the retention settings or delete the objects. Those users will need the s3:BypassGovernanceRetention permission as well as the DeleteObject permission.

Compliance Mode

Compliance mode is more restrictive - it cannot be undone within the retention period. As a result, Compliance mode should be used when you are certain that you do not want anyone, including the root user, to be able to delete the objects during its retention period.

Legal Hold

Legal hold offers the same protection (WORM) as the retention period but it has no expiration date by default nor can one be set. It is an indefinite hold that can only be removed by an authorized user.

Combo and Versioning

While Retention and Legal Hold prevent objects from deletion or overwriting, they can overlap. An object version can have both retention and legal hold. Retention and legal hold apply to individual object versions. Different versions of a single object can have different retention modes and periods.

Specifically, an object can have a retention period (call it six months) and a legal hold (indefinite). Even after the expiration of the retention period, the object version cannot be deleted because it still carries the legal hold. Only once the legal hold is lifted on that version of the object, can it be deleted.

While objects may have retention or legal hold, they can still be versioned. A copy operation on a version of an object does not carry forward the retention and legal hold settings that may be present on the source to the destination.

Here is a quick walk-through of a Legal Hold and Retention Period:

To use S3 Object Lock, follow these basic steps:

1. Create a new bucket with Object Lock enabled.

./mc mb myminio/testbucket --with-lock

The command listed above creates a bucket that supports object locking.

(Optional) Configure a default retention period for objects placed in the bucket.

/mc bucket lock myminio/testbucket governance 100d

Place the objects that you want to lock in the bucket.

./mc cp /path/to/file myminio/testbucket

Apply a retention period, a legal hold, or both, to the objects that you want to protect.

./mc retention myminio/testbucket/file governance 1y

./mc legalhold myminio/testbucket/file on

Pretty straightforward stuff.  

Conclusion

The addition of these features adds another layer to our ever growing suite of enterprise capabilities. For banks and healthcare companies that were already using MinIO, these capabilities, supported by our Cohasset assessment, provide an additional suite of use cases to use with MinIO, expanding our reach within the organizations.

We encourage you to try out these features for yourself.  We have documentation here and our Slack channel is always on.