MinIO Packet Pushers Podcast: Tom Lyon, NFS Must Die.

We really like the team over at Packet Pushers. Their podcast is one of the best in the industry, and they cover technology from the top of the stack to the bottom. We recently had the opportunity to sponsor the legendary Tom Lyon for an interview with Ethan Banks and Drew Conry-Murray.

The team at Packet Pushers was intrigued by Tom’s recent talk entitled “NFS (Network File System) Must Die.” Lyon has had a long career in the IT industry, including founding multiple companies, being an early employee at Sun Microsystems and contributing to the UNIX kernel. He knows a thing or two about file systems—their positives and negatives. We highly recommend the full podcast but will do our best to give you an abridged version in case you prefer the written word. 

Lyon kicks us off with a look at the three types of cloud storage: block storage, file storage and object storage. He explains that object storage is essentially the only truly cloud-native approach, largely pioneered by AWS to create a straightforward system that can scale globally. 

These days, there's a significant amount of cloud-native activity happening on-prem as well. It is often called the cloud operating model. The machine-to-user ratio is now at least a thousand times greater than it used to be. Systems need to operate consistently, scaling seamlessly to fit into cloud-native environments. 

So, what is wrong with NFS? NFS was a very carefully selected set of tradeoffs chosen 40 years ago. The whole paradigm doesn’t work in the cloud age. It encourages multiple users to access shared writeable dataspaces. 

Lyon states, “One of the most important lessons we’ve learned in distributed computing is that shared mutable data is a really bad idea. Things get out of sync. You can’t control or synchronize multiple actions. Everyone has a slightly different view of the data. So, it's a huge mess and you can't really build strong semantics on top of that.”

Another big problem is that everyone wants slightly different semantics. According to the CAP theorem, a network can’t have consistency and availability with a lack of partitioning, therefore leaving you to choose between consistency and availability. 

Lyon explains, “In the NFS days, we discovered this the hard way and had to do hard mounts to get consistency and soft mounts to get availability,” leaving you to choose, “are you going to piss off this group or that group?” 

In Lyon’s talk at a recent NLUUG (an association of professional UNIX and Linux users in the Netherlands) conference, he explains that “people are still investing hugely in NFS. There’s a whole new line of companies like HammerSpace, Weka and Vast Data that all rely on this kind of thing. Amazon has a cloud-managed NFS offering.” 

He asked the crowd, “Any using that? Yeah, don’t do that.” 

Banks points out, “You’re arguing that NFS must die, but it has had an awfully long run as we pointed out. NFS is everywhere… it’s baked into everything.” 

“But it’s never used at scale,” Lyon explains. 

To achieve the scale required in cloud deployments, systems must be 100% reliable. The trouble with NFS is that it requires a “human babysitting things when things go wrong.” Ultimately, when you need to operate at scale, NFS and similar file-oriented protocols are not the way to go. 

Banks asks Lyon to define the size of the scale he is talking about. 

The standout feature of NFS is its ability to allow access to large datasets with network-oblivious programs, providing access to explore data without needing to copy it first. But how do you achieve this at cloud scale? For example, how can I use grep on a dataset that’s publicly available in AWS in the US while I'm accessing it from China? That’s the kind of cloud-scale capability I’m referring to. AND how do you make it really easy?

Lyon also introduces his current project he calls Beyond File Sharing—BYFS, for short—an open source project in which he is looking for collaborators. This project is quite different from S3 as it leverages existing file systems and OverlayFS with existing protocols. 

“You need a block protocol for writable storage, but as soon as you freeze it, you can snapshot it to your object storage system, and that gives you worldwide access in a fairly trivial way. But you're snapshotting, again, at the file system level, so it's a much more granular thing. So, you can use this with object storage, except for that first writable layer, where you either want local writable storage or something coming over NVME over fabrics or something like that.” 

Lyon leaves us with the following, “Object storage is the way to do cloud storage. It’s cheap. It’s fast enough. File sharing, in my mind, is just an abomination left over from the enterprise world.” 

Powerful stuff from a guy who knows. As evangelists of object storage as primary storage, you won’t be surprised to learn that we agree wholeheartedly.