IOT, IIOT and Storage at the Edge

IoT applications have been exploding for some time, from the bleeding edge of the network to the factory floor, but now they are reaching critical mass. They already generate lots of data and that amount is increasing. Some of this data is processed locally but much of it needs to be sent someplace for further analysis.

In many cases, the data also needs to be archived for compliance or security purposes.

Deploying a local S3 object store can help reliably store, automatically upload, and manage this data - both at the edge, in-flight and at rest in the final analysis.

Below we discuss a few IoT applications that could profit from deploying local S3 object stores.

Oil & Gas Exploration

We are familiar with one company that has survey ships that record seismic data to disks inserted in PCs/servers, which they manually extract, and when back in port, insert into portside servers to upload data.

There are several problems with this. All it takes is one mistake, like, extracting the wrong disk, dropping a data disk, incorrectly inserting it, etc. any of which will lose the survey data. Moreover, disk drives can die and do so on a regular basis (and die more often when man-handled). Using an S3 object storage to record and manage data onboard can avoid many of these problems. For example,

• S3 buckets (of seismic recordings) can be automatically copied to other buckets for redundancy once saved.
• MinIO’s sophisticated approach to erasure coding ensures data protection, even in the face of drive failure.
• S3 buckets can be automatically replicated offsite or under API control, which can be easily triggered once back in port.
  

Furthermore, where necessary, seismic object data can be processed on ship, with appropriate hardware and software, which could lead to better targeted, follow-on surveys or dynamically adjusting the survey in real time. Having your data in the same, standard object format, both on ship and offsite can make processing faster and easier in both places. Given that MinIO is the object store of choice for AI/ML workloads, this seems like an easy choice.

While slightly different, several of the world’s largest cruise line operators are doing just this - with MinIO onboard and AWS S3 as the final destination.

Video Surveillance

While it seems cliche to call out video surveillance given it has been the poster child for edge storage use cases since the inception of the edge storage use case - the fact of the matter is that there is more and more of it every day. Take an airport for example - you cannot walk 10 meters without passing a camera. But it is not just airports, it is everywhere. Cameras get cheaper, so there are more of them. Resolution gets better so there are larger and larger objects. The traditional approach used DAS, Blu-ray disks, or other media to record video data, but that model is changing. For the Blinks and Nests of the world, they have excellent bandwidth on the home’s network, so that can go straight to the cloud. Further, with the cost of storage plummeting and the emergence of really beefy NVMe drives, the need to overwrite data is reduced considerably.

There is also the AI/ML angle. That video data can prove useful, but you need lots of it. Whether for customer traffic patterns, or intelligence services pattern-of-life analysis, the more you have the better the end result. Again, MinIO is the standard here when it comes to object storage.

As discussed previously, S3 object erasure coding fault tolerance plus its automated object replication/copying advantages can easily be applied to video surveillance data. Moreover, another feature of some S3 object replication is that one can limit bandwidth consumption to retain more bandwidth for other (customer) activities.

Furthermore with MinIO’s data protection mechanisms, video cannot be easily accidentally or intentionally overwritten.

Smart Vehicles (cars, tractors, trucks, drones, buses & ships)

As more and more vehicles add autonomous driving capabilities, sensor packages are being upgraded to higher resolution sensors and more of them. As a result, the amount of data these devices are starting to create, daily, is staggering.

Given the amount of data - the manufacturers are faced with a decision on what to keep and what to discard. This is somewhat of a false choice perpetuated by outdated approaches. While keeping everything forever on a constrained storage capacity is not realistic, the options are increasingly good. First, the manufacturer can process the data in-situ and only send up the anomalies or key metadata. Second, the manufacturer can track with the scale of the modern NVMe drives - capable of 20+ TB per drive. Third, the manufacturer can take a blended approach that transmits metadata of interest while keeping other data until an acceptable upload can be found (acceptable meaning the bandwidth cost/storage tradeoff is achieved).

Apart from all of these basic use cases, new applications can be developed that require real-time data like V2V communication (vehicle-to-vehicle) https://www.nhtsa.gov/technology-innovation/vehicle-vehicle-communication  or the Car to Car consortium https://www.car-2-car.org/ . All these require high-speed and light-weight storage -the sort that MinIO provides.

5G-6G Cell Towers

Cell towers for 5G are becoming full-fledged, edge data centers. 5G delivers not only higher data bandwidth, lower latency but also mission critical reliability and availability communications services required by some emerging AR and VR applications.

As a result, there is far more data running over the cell towers to the backhaul networks and back again. And it’s not just limited to user traffic. All 5G services need to log information and telemetry data about service use. Yet more data. Furthermore, to sustain ultra-low latency requirements for some 5G apps, data may need to be cached at the edge, increasing the need for automation and reliable data storage at the cell tower to manage this influx of data.

This drives unprecedented requirements for storage at the edge. That storage has to be software defined, it also has to be lightweight yet robust. It has to be performant, both from a throughput perspective and a latency perspective. Finally it has to be able to handle unstructured data. That all points to an object store like MinIO.

And of course, 6G is only 3-5 years away. Everything said about 5G data just goes up, by an order of magnitude, when 6G comes out.

Summary

There may be some IoT applications that generate small amounts of data today and throughout their active deployment, but most won’t. Everywhere we look, more and more sophisticated IoT applications are being deployed. And that increased sophistication relies on more data generation not less.

Managing all that data onsite, to not lose it and have it copied/replicated reliably elsewhere, can be done by hand with time and money. Or one could embed an S3 object store directly into your IoT application and have it reliably store, automatically copy and replicate all this data for you.

The choice is yours.