Supercharge TileDB Engine with MinIO

MinIO makes a  powerful primary TileDB backend because both are built for performance and scale. MinIO is a single Go binary that can be launched in many different types of cloud and on-prem environments. It's very lightweight, but also feature-packed with things like replication and encryption, and it provides integrations with various applications. MinIO is the perfect companion for TileDB because of its industry-leading performance and scalability. MinIO is capable of tremendous performance – we’ve benchmarked it at 325 GiB/s (349 GB/s) on GETs and 165 GiB/s (177 GB/s) on PUTs with just 32 nodes of off-the-shelf NVMe SSDs – and is used to build data lakes/lake houses with analytics and AI/ML workloads.

TileDB is used to store data in a variety of applications, such as Genomics, Geospatial, Biomedical Imaging, Finance, Machine Learning, and more. The power of TileDB stems from the fact that any data can be modeled efficiently as either a dense or a sparse multi-dimensional array, which is the format used internally by most data science tooling. By storing your data and metadata in TileDB arrays, you abstract all the data storage and management pains, while efficiently accessing the data with your favorite programming language or data science tool via our numerous APIs and integrations.

Set Up TileDB

Let’s dive in and create some test data using TileDB

Install the TileDB pip module, which should also install the numpy dependency.

Create a test array by running the below Python script, name it tiledb-demo.py.

Run the script

This will create a directory called tiledb_demo to store the actual data.

You can continue using it as is but it's no bueno if everything is local because if the local disk or node fails then you lose your entire data. Let's do something fun, like reading this same data from a MinIO bucket instead.

Migrating Data to MinIO Bucket

We’ll start by pulling mc in our docker ecosystem and then using play.min.io to create the bucket.

Pull mc docker image

Test with MinIO Play by listing all the buckets

Create a bucket to move our local TileDB data to, name it tiledb-demo.

Copy the contents of the tiledb_demo data directory to the MinIO tiledb-demo bucket

List the contents of tiledb-demo to make sure the data has been copied

Note: The MinIO Client (mc), or any S3 compatible client, only copies non-empty folders. The reason for this is that in the object storage world the data is organized based on bucket prefixes, so non-empty folders are not needed. In a future blog we’ll dive deeper into how data is organized with prefixes and folders. Hence, you see only these 3 folders and not the rest that we saw in the local folder.

Now let’s try to read the same data directly from the MinIO bucket using the Python code below, name the file tiledb-minio-demo.py.

The output should look familiar

We've read from MinIO, next let's see how we can write the data directly in a MinIO bucket, instead of copying it to MinIO from an existing source.

Writing Directly to the MinIO Bucket

So far we’ve shown you how to read data that already exists, either in local storage or an existing bucket. But if you wanted to start fresh by writing directly to MinIO from the get-go, how would that work? Let’s take a look.

The code to write data directly to the MinIO bucket is the same as above except with two line changes.

The path to the MinIO bucket where TileDB data is stored must be updated to tiledb_minio_demo (instead of tiledb_demo).

We’ll use the tiledb.from_numpy function, as we did earlier with local storage, to create the array to store in the MinIO bucket.

After making these 2 changes, run the script and you should see the output below

If you run the script again it will fail with the below error because it will try to write again.

Just comment out the following line and you can re-run it multiple times.

Check the MinIO Play bucket to make sure the data is in there as expected

There you go, getting data into MinIO is that simple. Did you get the same results as earlier? You should have, but if you didn't there are a few things you can check out.  

Common Pitfalls

We’ll look at some common errors you might encounter while trying to read/write to MinIO.

If your access key and secret key are incorrect, you should expect to see an error message like below

Next, you need to ensure the hostname and port are correct, without a proper endpoint these are the errors you would encounter

Incorrect Hostname:

Incorrect Port:

Last but not least, one of the most cryptic errors I’ve seen is the following

After a ton of debugging it turns out that if you are connecting using http but the MinIO server has TLS activated then you will see the above error. Just be sure the connection scheme is set to the right configuration, in this case, config["vfs.s3.scheme"] = "https".

Racks on Racks on Racks

There is a rap song (you can search for it) where they rap about having stacks on stacks on stacks of *cough* cash. But there is another rap song where they claim they have so many stacks of cash that they can’t be called “stacks” anymore, they are now “racks”. Essentially when your stacks get so big and so high you need racks on racks on racks to store your stacks of cash.

This is an apt comparison because your stacks of data mean as much (or more) to you as the stacks of cash they're rapping about. If only there was something like MinIO to keep all your objects – physical or virtual – safe and readily accessible.

With MinIO in the mix, you can easily scale TileDB to multiple racks across multiple datacenters with relative ease. You also get all the features that make MinIO great like Security and Access Control, Tiering, Object Locking and Retention, Key Encryption Service (KES), among others right out of the box. By having all your data in MinIO, you decrease required storage complexity and therefore realize considerable savings on data storage costs, while at the same time running MinIO on commodity hardware provides the best possible performance-to-cost ratio. MinIO supercharges your TileDB engine with industry-leading performance that makes querying a joy.

We’ve added the code snippets used in this blog to a git repository. If you have any questions on how to connect MinIO to TileDB or migrate data into MinIO be sure to reach out to us on Slack!