An Architect's Guide to the Hybrid Cloud

As we ease into 2021, there are a few technology vectors that are dictating the conversation for IT architects. The dominant one is Kubernetes. Related, and quickly becoming “standard,” is the hybrid cloud.

The challenges inherent in an architect's role are only compounded when planning for the hybrid cloud. First, it is new and marketing has a tendency to outrun facts. Second, it is constantly evolving  —  which requires the architect to have a strong sense of what comes next. Third, organizations are changing and adapting to the trials and tribulations associated with a global pandemic. Finally, this is a long term planning exercise with short term deliverables — one thing we know for sure is that the modern enterprise will not tolerate a technology vacuum. A vacuum, after all, is what launched the multi-cloud phenomena to start with.

Now is an opportunity to bring order to the chaos. At the most fundamental level, the architect must deliver consistency across the various environments. Developer consistency, application consistency, user interface consistency, performance consistency. The list goes on, but the success criteria stays the same with regard to consistency.

This post is going to focus on one element, albeit a very critical element in any hybrid cloud architecture: storage. Before we go any further, we should mention that we are only concerned with object storage in this post. Object storage is the storage class of the cloud and of Kubernetes. That makes it the storage class of the hybrid cloud. File and block systems are legacy at this point — one only needs to look at how it is priced in the public cloud to understand that fact.

From an architect's perspective, it helps to start by defining the playing field. There is a tendency to use the terms ‘public cloud’ and ‘on-prem’ and be done with the definition of the hybrid cloud. The truth, however, is that the hybrid cloud is multi-dimensional.

To deliver a functional hybrid cloud architecture, you need to have a storage strategy that can operate in the following environments.

Public Clouds - This is an increasingly large field, but starts with AWS, Azure, GCP, IBM, Alibaba and Tencent. Your hybrid cloud storage software needs to run everywhere your business runs. Even companies that claim to run on a single cloud don’t — there are always other clouds, you just don’t know about them yet.

Private Cloud - The definition of the private cloud continues to evolve and the role of hybrid cloud storage needs to evolve to support that emerging architecture. The private cloud is a concept, not a place, and the modern private cloud is often found in off-premises data centers, virtual private networks and virtual private clouds. Your hybrid cloud storage needs to run without compromise everywhere your cloud computing infrastructure runs.

The Kubernetes Distributions - Often overlooked, the Kubernetes distributions could be considered a subcategory of the private cloud, but we treat them as separate entities because they don’t lend themselves to a roll-your-own approach.  To run here, your hybrid cloud storage solution needs to be object storage, software defined and cloud native. Options include VMware (Tanzu), HP (Ezmeral), Cisco (IKE), RedHat (OpenShift) and Rancher/SUSE.

The Edge - Also often overlooked, the edge is a critical part of any hybrid cloud architecture. Your hybrid cloud storage solution needs to be lightweight, powerful, cloud native and fast to run at the edge. While the edge has varying levels of importance today, that importance will only grow — and with it the challenge of small objects. Architects designing hybrid systems need to be thinking clearly about the implications of the edge.

The Attributes of Hybrid Cloud Storage

Given these hybrid cloud parameters — object storage deployed across public, private, Kubernetes distros and the edge — what are the attributes of success? I present the following for consideration:

Consistency - As noted earlier, the goal is consistency in the user experience, application performance and developer experience. Roadmaps are nice, but data is bankable. What object storage solutions run across multiple public clouds, across the Kubernetes distributions, and at the edge? Are there elements that would preclude a solution from succeeding in these environments? An appliance, for example, doesn’t lend itself to orchestration. It cannot provide consistency across the environments. Public clouds are another area where consistency is threatened. There is increasing talk from the major players not just about their on-prem offerings, but also their ambitions to run in each other’s clouds. How does that square with their experience when running a service of having complete control over the hardware? Can they really guarantee consistency?

Performance - Performance expands the pool of applications that you can pair with object storage. Almost every modern workload demands performance. If you are not performant you cannot run Spark, Presto, Tensorflow or any of the other AI/ML and big data applications that have come to define the enterprise landscape. Even archival workloads benefit from performance. What enterprise designs a slow restore process?

An architect needs to design not only for performance, but also performance at scale. This is where modern object storage shines. Long known as cheap and slow, new object storage offerings read and write at hundreds of GB/s on standard hardware. Not every workload demands that performance, but every workload wants it. To serve the broadest audience, architects need to design for speed.

Scale - Scale is often misinterpreted to mean the theoretical limit of the system. While object storage is considered to be infinitely scalable, everyone knows that practically this is not the case. Scalability has multiple dimensions. Architects need to consider the operational efficiency of scaling and the bottlenecks that can arise. For example, object stores that use an external metadata database simply don’t scale past a certain point. They are poor choices for large scale infrastructure.

A hybrid cloud object storage solution needs to scale in the same way — no matter the environment — and do so simply, with minimal human interaction and maximum automation.

Software Defined - For an architect thinking about multiple workloads on multiple clouds, (public, private, edge) there is only one answer: software. Multiple environments dictate other heterogeneous hardware. Software abstracts the backend physical storage and is the architect's primary tool in this effort (see Kubernetes). Software defines the user experience, providing flexibility and extensibility.

Cloud Native - For an architect thinking about storage this can be a component that one “gives a pass” to, given how few vendors are actually cloud native. Don’t. Just as a leopard cannot change his/her spots, an appliance vendor does not suddenly become software defined and cloud native. Cloud native is as much a philosophy as it is a collection of technologies and principles. If Kubernetes, containers, microservices, S3 and API were not part of the plan from the beginning, there will always be friction. It should not completely disqualify non-cloud native storage vendors, but it should provide pause. What worked for on-prem doesn’t work for the cloud. What became a key vendor relationship five years ago may not be relevant to the architectures that are emerging.

A FrameworkThis post is designed to provide a framework for enterprise cloud architects looking ahead. The key to any successful planning exercise is to challenge your thinking and to create as much detail as possible around the key components of the plan. Planning for hybrid cloud storage architecture requires exceptional discipline and deep evaluation of previously held beliefs. The payoff for enterprises, however, can be massive — both from a cost savings perspective and a competitive perspective.

Feel free to reach out and engage us on Slack or email us at hello@min.io to continue the conversation.