Master Full Text Search with MeiliSearch on MinIO

Nitish Tiwari -

Text data is a big and continuously growing category of data that’s being generated, stored and analyzed every day. Different applications generate text data in the form of events, logs and notifications. Businesses, facing growing volumes of IOT and machine-generated text data, are always looking for faster and more accurate search tools. There are many applications to search text data and they require high-performance object storage to store and search text at scale. Providing rapid text search is a great value add for any team.

In this post, we’ll learn how to leverage MinIO and MeiliSearch to build a large scale storage and search platform. First, we’ll set up MeiliSearch, enable snapshots and then use the MinIO server as the high throughput, scalable, storage platform for MeiliSearch snapshots. We’ll use MinIO Client (mc) to store these snapshots to MinIO, so that MeiliSeach can be restored from any point in time.

MeiliSearch is a RESTful search API server. Simply put, MeiliSearch allows users to upload arbitrary text data, indexes the data and allows lightning fast search on the data set uploaded.

MinIO is well-suited as an object store for snapshots because it is fast, scalable and straightforward to install and manage. At 100MB, MinIO is lightweight and efficient, so it’s as easy to install MinIO locally to learn about and develop on new technologies as it is to use it to build your own self-service object storage platform.


Text search platforms typically take snapshots of data and indices, and MeiliSearch is not different. MinIO supports moving, replicating and versioning snapshots. In this demo, we’ll use mirroring to move MeiliSearch snapshots to MinIO on a periodic basis. With versioning enabled, MinIO will ensure multiple versions of snapshots are stored properly and it is even possible to rewind to older snapshots.

Prerequisites

We’ll need MinIO and MeiliSearch configured and running before we learn how to backup the MeiliSearch snapshots to MinIO.

Install MeiliSearch

MeiliSearch is available as a self contained binary for major operating systems. Download and start MeiliSearch:

# Install MeiliSearch
curl -L https://install.meilisearch.com | sh
# Launch MeiliSearch
./meilisearch --schedule-snapshot=true --snapshot-interval-sec 3600

With this, MeiliSearch will start with snapshots scheduled to be generated every hour in the default snapshot location (i.e. snapshots directory inside the MeiliSearch root directory).

Install MinIO and MinIO Client

To backup the MeiliSearch snapshots to a MinIO Server, we’ll use the `mc mirror` command. You’ll need a working MinIO Server as the mirror target. First, set up a MinIO server. In this blog post, we’ll use the publicly hosted MinIO Play server as the backup target.

Once you have the MinIO Server up and running, download and install the MinIO Client (mc).

Back Up Snapshots to MinIO

To back up snapshots we’ll use a MinIO versioned bucket with lock enabled so that the snapshot can be restored to a specific point in time as required. Create the bucket and enable versioning:

mc mb --with-lock play/meilisearch-snap
mc version enable play/meilisearch-snap

Now, start an mc mirror session that will keep copying the MeiliSearch snapshot every time it is  generated in the MinIO Server bucket that we just created:

mc mirror --watch /path/to/meilisearch/snapshots/ play/meilisearch-snap/

Note that the MeiliSearch snapshot is named as data.ms.snapshot . That’s the file we’ll be concerned with in the backup bucket.

Restore MeiliSearch from a Snapshot

In case you want to restore MeiliSearch from a specific snapshot, you can download a version of that snapshot from MinIO server using:

mc ls play/meilisearch-snap --versions
mc cp --vid 77decd94-ac3d-409f-8abb-934c7b157f75 play/meilisearch-snap/data.ms.snapshot ./

Once you have the specific snapshot version downloaded, you can restore MeiliSearch using

meilisearch --import-snapshot mySnapShots/data.ms.snapshot

This will start MeiliSearch from the point in time where the snapshot was created.

MeiliSearch and MinIO For Fast Reliable Text Search

Voila! Super simple, blazingly fast.

Hopefully we have demonstrated how easy and efficient it is to pair highly scalable object storage and a search platform for text data, using MinIO and MeiliSearch. This pattern should look familiar to readers. It is similar in may ways to our recent post on databases on object storage. The big takeaway is that disaggregation continues to move up the stack and applications are outsourcing the storage to S3 (in this case MinIO) and focusing on query processing or search (compute). Combined together these systems offer a great way to build a best-of-breed, scalable text search platform.

Get started by downloading MinIO and MeileSearch. If you have questions, please refer to MinIO Documentation or ask on our Slack channel.