Publish Minio events via NATS

Introduction

Minio server supports Amazon S3 compatible bucket event notification for following targets AMQP, Elasticsearch , Redis, nats.io, PostgreSQL and Apache Kafka. Part 4 of this blog series covers sending bucket notifications using NATS.

Steps to Configure NATS

I used Ubuntu Linux 16.04, gnatsd and nats package. It requires go package installed, for installation steps follow https://docs.minio.io/docs/how-to-install-golang

$ go get github.com/nats-io/gnatsd
$ go get github.com/nats-io/nats

Start NATS server.

$ gnatsd --user yourusername --pass yoursecret
[1510] 2016/10/12 06:14:48.706544 [INF] Starting nats-server version 0.9.4
[1510] 2016/10/12 06:14:48.707441 [INF] Listening for client connections on 0.0.0.0:4222
[1510] 2016/10/12 06:14:48.708332 [INF] Server is ready

Steps to configure Minio

Latest Minio server binary can be downloaded from https://minio.io/downloads/

$ wget https://dl.minio.io/server/minio/release/linux-amd64/minio
$ chmod +x minio
In this case “myphotos” is used as my data directory for Minio server.
$ ./minio server myphotos
Endpoint: http://10.1.10.150:9000 http://127.0.0.1:9000
AccessKey: 7I6R5G576YI641GS9J9F
SecretKey: SuycBIe+O/s5zXxU9w+N4wkXHpBCKa2H6Ptlrc8c
Region: us-east-1
...
...

The default location of Minio server configuration file is~/.minio/config.json. Update the NATS configuration block in ‘config.json’ as follows

"nats": {
            "1": {
                "enable": true,
                "address": "0.0.0.0:4222",
                "subject": "bucketevents",
                "username": "yourusername",
                "password": "yoursecret",
                "token": "",
                "secure": false,
                "pingInterval": 0
            }
        },

Restart the Minio server to reflect config changes made above. “bucketevents” is the subject used by NATS in this example.

Enable bucket notification using Minio client

Step 1: Download and install Minio client

$ wget https://dl.minio.io/client/mc/release/linux-amd64/mc
$ chmod 755 mc

Step 2: Add Minio server host alias information

Configure Minio client with access and secret keys pointing to the Minio server.

$ ./mc config host add myminio http://localhost:9000 7I6R5G576YI641GS9J9F SuycBIe+O/s5zXxU9w+N4wkXHpBCKa2H6Ptlrc8c

Step 3: Setup bucket notification

In this example we will enable bucket events only when JPEG images are uploaded or deleted from ‘images’ bucket on ‘myminio’ server. Here ARN value is arn:minio:sqs:us-east-1:1:nats. To understand more about ARN please follow AWS ARN documentation.

$ ./mc mb myminio/images
$ ./mc events add  myminio/images arn:minio:sqs:us-east-1:1:nats --suffix .jpg
$ ./mc events list myminio/images
arn:minio:sqs:us-east-1:1:nats s3:ObjectCreated:*,s3:ObjectRemoved:* Filter: suffix=”.jpg”

Step 4: Testing on NATS

Using this program below we can log the bucket notification added to NATS.

package main

// Import Go and NATS packages
import (
  "runtime"
  "log"
  "github.com/nats-io/nats"
)

func main() {

    // Create server connection
    natsConnection, _ := nats.Connect("nats://yourusername:yoursecret@localhost:4222")
    log.Println("Connected")

    // Subscribe to subject
    log.Printf("Subscribing to subject 'bucketevents'\n")
    natsConnection.Subscribe("bucketevents", func(msg *nats.Msg) {

      // Handle the message
      log.Printf("Received message '%s\n", string(msg.Data) + "'")
  })

  // Keep the connection alive
  runtime.Goexit()
}

$ go run nats.go
2016/10/12 06:39:18 Connected
2016/10/12 06:39:18 Subscribing to subject 'bucketevents'

Open another terminal and upload a JPEG image into “images” bucket.

$ ./mc cp myphoto.jpg myminio/images

Below terminal running nats.go program prints event notification on console.

$ go run nats.go 
2016/10/12 06:51:26 Connected
2016/10/12 06:51:26 Subscribing to subject 'bucketevents'
2016/10/12 06:51:33 Received message '{"EventType":"s3:ObjectCreated:Put","Key":"images/myphoto.jpg","Records":[{"eventVersion":"2.0","eventSource":"aws:s3","awsRegion":"us-east-1","eventTime":"2016-10-12T13:51:33Z","eventName":"s3:ObjectCreated:Put","userIdentity":{"principalId":"minio"},"requestParameters":{"sourceIPAddress":"[::1]:57106"},"responseElements":{},"s3":{"s3SchemaVersion":"1.0","configurationId":"Config","bucket":{"name":"images","ownerIdentity":{"principalId":"minio"},"arn":"arn:aws:s3:::images"},"object":{"key":"myphoto.jpg","size":56060,"eTag":"1d97bf45ecb37f7a7b699418070df08f","sequencer":"147CCD1AE054BFD0"}}}],"level":"info","msg":"","time":"2016-10-12T06:51:33-07:00"}

For further questions and comments join our Slack chat at: https://slack.minio.io