Re-Imagining Support: From Slack to SUBNET

We approach things differently here at MinIO.

When we started in 2014, we questioned everything about the object storage market as we built our product - thinking more like a data company than a storage company. We did it from scratch, using engineering first principles, an extraordinary attention to detail and a relentless pursuit of simplicity.

We paid careful attention to the features that mattered. Erasure code, bitrot detection, encryption and WORM, continuous replication, identity management, and global federation. If it made it simpler, more scalable or more enterprise-ready we did and did it right.

It has paid off. We now have almost 230M Docker pulls, more than 16.7K GitHub stars, 390+ contributors and 4,800 members of our Slack channel. More than 50% of the Fortune 100 run MinIO inside their organization in configurations that range from S3 gateways to large distributed production instances.  We are deployed in production on five continents supporting dozens of use cases.

Which means we have to support production deployments on five continents and dozens of use cases.

There are two levels of support, the public support channel and the subscription SUBNET channel. The foundations are the same for both, but the experience differs as one might expect.

This post covers the philosophy behind support as well as the mechanics of how it works.

The Public Support Channel

Public support is centered around our public Slack channel and our documentation. As one might imagine, there are no SLAs or response guarantees there, but there is an active presence by MinIO engineers and its broader community. There are more than 4,800 members and, if you are adept at search, your answer is probably there. There is a large amount of traffic here, about half of it incoming help requests (with one off feature requests here and there) and half is responses to those issues.

If not, it is likely in our robust documentation. It is comprehensive, well organized, and easy to understand making it different from anywhere else I have worked. I would estimate that our documentation reduces our support volume by 30% or more.

It is that good.

One reason our documentation is so good is that we are hyper-focused on simplicity. Simple products are easy to explain. And easy to support.

This is massive.

Because of our relentless pursuit of simplicity, we don’t have garbage issues to track down for either our community or SUBNET subscribers. We don’t spend time troubleshooting minutia/corner cases caused by our own software - even though our Open Source license means that we are deployed in countless configurations.

This has allowed us to pursue a different path both on the public side and on the private, subscription side.  

SUBNET Support

SUBNET is our subscription offering. It is more than just support, but support is a big part of it. It is designed for enterprises who have large production deployments on MinIO and want the peace of mind that comes with having some of the best minds in storage, cloud native applications and file systems on speed-dial.

In building our SUBNET support system we took the same approach as we did when we built the object server itself - we questioned everything and built only what was necessary.  It would have been easy to buy an off-the-shelf ticketing/support system - there are literally dozens of them, but that wouldn’t have solved for what we were trying to achieve.

Our goal was a real time communications platform that focuses specifically on rapid resolution through conversation.  

We adapted the model that we employed for public community - leveraging Slack and its conversational interface. We improved upon the responsiveness, the supporting tools and the level of focus. We also drew on the cultural emphasis of engagement at MinIO. There is a strong sense of collective responsibility in this largely engineering oriented organization. The result is that everyone is deeply invested in our customer’s success.

In our model we don’t impose rules or regulations on how our clients open or work on issues.  If you are in the middle of a critical issue, you don’t have five minutes to spend figuring out how to get in touch with someone. You want someone who can solve the problem to be available, engaged and focused on your issue.  Answers are targeted...the people who wrote the code are the ones answering your issues, so there is a huge reduction in probing questions and collecting data.

Sometimes, the issue isn’t critical, but you have spent a lot of time thinking it through and you just need someone to bounce your ideas off of - and again, availability and expertise matter.  We strive to support your whole ecosystem, so we can recommend the implications of implementing MinIO specific to your environment.

Since everything is meant to be a conversation, the model ensures that we understand exactly what problem you are trying to solve - nothing is lost in translation over forms and timezones.

The result is a support system that is unlike what the industry has come to know, but far more successful. What we can do with a handful of people, other organizations do with hundreds.  In over a decade in support, I have never worked with a system this intuitive and simple.  

Other systems concentrate on management and ticket flow.  They require you to spend time on managing tickets, which takes time away from the actual end goal you are trying to accomplish.  Even if the amount of work in managing the tickets is only moderate, you are still forced into a series of context switches that distracts you from what you need to be focusing on. The simplicity of MinIO’s approach ensures that problems, even complex ones get the focused attention they deserve. This has an added benefit, it makes the team far more efficient in solving problems precisely because the engineers solving the problem are not being forced to context switch.

Our approach has its implications.

One is that all hands are on deck all the time and for all the issues. As a result, we pick our customers carefully. Because we are 100% open source and customers can self-support, we structure and price SUBNET to appeal to those customers with large scale deployments of our software. Because of that, SUBNET subscribers tend toward the more sophisticated end of the technical spectrum. When engineering is your support - that leads to higher “love thy customer” metrics and a better experience for both parties.

Another implication is that our approach leads to deeper engagement. Unlike most models, we will look at your code, we will advise you on the optimal implementation. We do it because it helps the customer and we are deeply committed to their success.  This shouldn’t be confused with services or with private extension development - we don’t engage in those, even for seven figure sums.

This approach focuses on outcomes over metrics. Did the customer solve her/his problem? Was it to their satisfaction? Outcomes drive great metrics, but those are not the end game. It takes some getting used to for some customers. They (well to be honest, procurement teams) value SLAs and turnaround guarantees. Even they come around when we implement fixes or roll new releases (again, not custom) in a fraction of the time it takes traditional organizations.

The results are very promising.  Our average response times are under an hour, with many issues getting attention in less than 5 minutes.

Again, speed is a metric, but it is not always the best metric. Many issues require careful thought and deep engagement - getting them right is the right metric, not getting them closed so they return elsewhere.

Some in the industry will claim that our model isn’t scalable that it you need a massive support organization as you go. Those companies don’t relentlessly pursue simplicity like we do. Our marketing guy is here because he was able to get MinIO running in less than ten minutes despite limited experience with object storage or distributed systems. He knew it was different.

This scales because everything is about minimalism (simplicity) and everything is an engineering problem. As we grow, we will grow engineering, not support. The collective ownership gene will be something we optimize for in our hiring.

We are incredibly excited about what lies ahead for us. Our growth trajectory means we will face new challenges. It will stretch us and challenge us, but I think our underlying philosophy, culture and commitment to the conversation will carry us forward.

Feel free to join our Slack channel to ask me about the approach. My handle is @Eco.