Using Elasticsearch snapshots with Minio repository

on 16 July 2016

This article shows you how to configure Elasticsearch to store snapshots and metadata in Minio. Minio is an object storage server built for cloud application developers and devops and compatible with Amazon S3 object storage.

The current aws cloud plugin of Elasticsearch has been build with an older version of the Amazon SDK, causing several issues. We’ve created a pull request to bump the aws sdk for Elasticsearch 5 here.

A patchfile to update the aws sdk for older versions can be found here.

Compiled plugins for Elasticsearch 2.3.4 and 2.4 can be found at this repository. These contain the bumped version of aws sdk together with a fix for double hashing.

To build the updated aws cloud plugin of Elasticsearch:

Custom building

Elasticsearch 2.x

git clone https://github.com/elastic/elasticsearch/
git reset — hard v2.3.4

Update pom.xml references to your Elasticsearch version. This is because otherwise during installation the plugin will complain about an incompatible version.

sed -i”.bak” s/”2.3.4-SNAPSHOT”/”2.3.4"/g **pom.xml

Apply the patch containing the newer SDK version, and fix the double hashing issue.

curl https://gist.githubusercontent.com/nl5887/95352af849089e69150bdf7129a83c92/raw/c36b172ea2c69a3e566194af59c9c3c8ff104010/aws-sdk-upgrade.patch|git apply -

Compile cloud aws plugin.

mvn package -pl plugins/cloud-aws

Elasticsearch 5.x

Elasticsearch 5 is using gradle. You can compile the updated plugin using:

gradle build -p plugins/repository-s3

Plugin install

We have to install the plugin so it can be used by Elasticsearch.

Plugin install on Elasticsearch 2.x (2.3.4)

sudo bin/plugin remove cloud-awssudo bin/plugin install file:/{path to elasticsearch}/plugins/cloud-
aws/target/releases/cloud-aws-2.3.4.zip

Plugin install on Elasticsearch 5.x

sudo bin/elasticsearch-plugin remove repository-s3sudo bin/elasticsearch-plugin install file:/plugins/repository-s3/build/distributions/repository-s3–5.0.0-alpha5-SNAPSHOT.zip

Configuration

Configure Elasticsearch to use V4 signing for S3, by adding the following to elasticsearch.yml.

cloud:    
    aws:        
        s3:            
            signer: "AWS4SignerType"

Now you should restart Elasticsearch. The new configuration and the installed plugins will become active.

With the following command you can verify the installed and active plugins.

curl http://127.0.0.1:9200/_cat/plugins?v

Working with snapshots

First we need to create a new bucket in Minio where the snapshots can be put../mc mb myminio/elasticsearch

Create repository:

Now we’ll configure a Elasticsearch repository using Minio as a backend.

curl -X POST http://127.0.0.1:9200/_snapshot/my_minio_repository — 
data ‘{ 
“type”: “s3”, 
“settings”: { 
“bucket”: “elasticsearch”, 
“region”: “us-east-1”, 
“endpoint”: “http://127.0.0.1:9000",  
“access_key”: “AJBCFEV8M5Q8XIQPRITQ”, 
“secret_key”: “TBPnPHamh6r7ypXACfO4Nxz59PjE+3SanplAZDzq”, 
“protocol”: “http” 
    }
        }’

Retrieve repository info

To confirm the repository has been created succesfully, we’ll use the get repository.

curl -X GET http://127.0.0.1:9200/_snapshot/my_minio_repository?pretty
{ “my_minio_repository” : { 
“type” : “s3”, 
“settings” : { 
“bucket” : “elasticsearch”, 
“endpoint” : “http://127.0.0.1:9000", 
“protocol” : “http”, 
“region” : “us-east-1” 
    } 
    }
}

Create snapshot

We’ll create a snapshot and wait for completion of it.

curl -X PUT 
http://127.0.0.1:9200/_snapshot/my_minio_repository/snapshot_1/\?wait_for_completion=true
{“snapshot”:{“snapshot”:”snapshot_1",”version_id”:2030399,”version”:”2.3.3",”indices”:[“minio”],”state”:”SUCCESS”,”start_time”:”2016–07–18T14:58:05.492Z”,”start_time_in_millis”:1468853885492,”end_time”:”2016–07–18T14:58:36.526Z”,”end_time_in_millis”:1468853916526,”duration_in_millis”:31034,”failures”:[],”shards”:{“total”:41,”failed”:0,”successful”:41}}}

Verify snapshot status

Retrieve the snapshot status.

curl -X GET http://127.0.0.1:9200/_snapshot/my_minio_repository/snapshot_1/{“snapshots”:[{“snapshot”:”snapshot_1",”version_id”:2030399,”version”:”2.3.3",”indices”:[“minio”],”state”:”SUCCESS”,”start_time”:”2016–07–18T14:58:05.492Z”,”start_time_in_millis”:1468853885492,”end_time”:”2016–07–18T14:58:36.526Z”,”end_time_in_millis”:1468853916526,”duration_in_millis”:31034,”failures”:[],”shards”:{“total”:41,”failed”:0,”successful”:41}}]}

Restore snapshot

Restore an existing index, make sure the index has been closed.

curl -X POST http://127.0.0.1:9200/_snapshot/my_minio_repository/snapshot_1/_restore{“accepted”:true}

References: