Publish Minio events via Elasticsearch

Introduction

Minio server supports Amazon S3 compatible bucket event notification for following targets AMQP, Elasticsearch , Redis, nats.io, PostgreSQL and Apache Kafka. Part 2 of this blog series covers sending bucket notifications using Elasticsearch server.

Steps to Configure Elasticsearch

I used Ubuntu Linux 16.04, latest Minio server binary and Elasticsearch 2.4.0on my laptop for this setup.

Note: Elasticsearch requires at least Java 7. It is recommended you use the Oracle JDK.

Install Elasticsearch

$ curl -L -O https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/tar/elasticsearch/2.4.0/elasticsearch-2.4.0.tar.gz
$ tar -xvf elasticsearch-2.4.0.tar.gz
$ cd elasticsearch-2.4.0/bin
$ ./elasticsearch
[2016-09-09 16:17:24,733][INFO ][node                     ] [Kirigi] version[2.4.0], pid[4422], build[ce9f0c7/2016-08-29T09:14:17Z]
[2016-09-09 16:17:24,734][INFO ][node                     ] [Kirigi] initializing ...
[2016-09-09 16:17:25,632][INFO ][plugins                  ] [Kirigi] modules [reindex, lang-expression, lang-groovy], plugins [], sites []
[2016-09-09 16:17:25,679][INFO ][env                      ] [Kirigi] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [3.5gb], net total_space [7gb], spins? [possibly], types [ext4]
...
...

Note: Ensure you have curl package installed.

Steps to Configure Minio

Latest Minio server binary can be downloaded from https://minio.io/downloads/

$ wget https://dl.minio.io/server/minio/release/linux-amd64/minio
$ chmod +x minio
In this case “myphotos” is used as my data directory for Minio server.
$ ./minio server myphotos
Endpoint: http://10.1.10.150:9000 http://127.0.0.1:9000
AccessKey: 7I6R5G576YI641GS9J9F
SecretKey: SuycBIe+O/s5zXxU9w+N4wkXHpBCKa2H6Ptlrc8c
Region: us-east-1
...
...

The default location of Minio server configuration file is ~/.minio/config.json. Update the Elasticsearch configuration block in ‘config.json’ as follows.

Restart the Minio server to reflect config changes made above. “bucketevents” is the index used by Elasticsearch in this example.

Enable bucket notification using Minio client

Step 1: Download and install Minio client

$ wget https://dl.minio.io/client/mc/release/linux-amd64/mc
$ chmod 755 mc

Step 2: Add Minio server host alias information

Configure Minio client with access and secret keys pointing to the Minio server.

$ ./mc config host add myminio http://localhost:9000 7I6R5G576YI641GS9J9F SuycBIe+O/s5zXxU9w+N4wkXHpBCKa2H6Ptlrc8c

Step 3: Enable bucket notification

In this example we will enable bucket events only when JPEG images are uploaded or deleted from ‘images’ bucket on ‘myminio’ server. Here ARN value is arn:minio:sqs:us-east-1:1:elasticsearch. To understand more about ARN please follow AWS ARN documentation.

$ ./mc mb myminio/images
$ ./mc events add  myminio/images arn:minio:sqs:us-east-1:1:elasticsearch --suffix .jpg
$ ./mc events list myminio/images
arn:minio:sqs:us-east-1:1:elasticsearch s3:ObjectCreated:*,s3:ObjectRemoved:* Filter: suffix=”.jpg”

Step 4: Testing on Elasticsearch

Open another terminal and upload a JPEG image into “images” bucket.

$ ./mc cp myphoto.jpg myminio/images

You should see Elasticsearch has created a new index.

$ curl -XGET '127.0.0.1:9200/_cat/indices?v'

health status index pri rep docs.count docs.deleted store.size 
pri.store.size

yellow open   bucketevents  5   1          1            0      7.8kb          7.8kb

Use curl to view contents of “bucketevents” index.

$ curl -XGET '127.0.0.1:9200/bucketevents/_search?pretty=1'
{
  "took" : 3,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 1.0,
    "hits" : [ {
      "_index" : "bucketevents",
      "_type" : "event",
      "_id" : "AVcRVOlwe-uNB1tfj6bx",
      "_score" : 1.0,
      "_source" : {
        "Records" : [ {
          "eventVersion" : "2.0",
          "eventSource" : "aws:s3",
          "awsRegion" : "us-east-1",
          "eventTime" : "2016-09-09T23:42:39.977Z",
          "eventName" : "s3:ObjectCreated:Put",
          "userIdentity" : {
            "principalId" : "minio"
          },
          "requestParameters" : {
            "sourceIPAddress" : "10.1.10.150:52140"
          },
          "responseElements" : { },
          "s3" : {
            "s3SchemaVersion" : "1.0",
            "configurationId" : "Config",
            "bucket" : {
              "name" : "images",
              "ownerIdentity" : {
                "principalId" : "minio"
              },
              "arn" : "arn:aws:s3:::images"
            },
            "object" : {
              "key" : "myphoto.jpg",
              "size" : 200436,
              "sequencer" : "1472CC35E6971AF3"
            }
          }
        } ]
      }
    } ]
  }
}
view rawelasticsearch hosted with ❤ by Git

curl output above states that an Elasticsearch index has been successfully created with notification contents.

For further questions and comments join our Slack chat at: https://slack.minio.io