Introducing Webhooks for MinIO

MinIO provides integration with a range of backend systems which enables you to build a complete solution for your projects. The team has recently integrated a pull request from the community to add Webhook support and wanted to tell you a bit about it.

First we will look at what Webhooks are, some examples of what they can connect with, then run through an example end-to-end of how to configure a Thumbnail generator with MinIO.

Webhooks are the de facto standard for API communication over the Internet. In technical terms a webhook is where a server application sends a notification to another system as a HTTP POST with a JSON payload. To consume MinIO webhooks in your own application you just need to create a HTTP/s endpoint and edit your ~/.minio/config.json file.

In this post we will to highlight a use-case for automatically creating thumbnails in near-realtime without the need for polling.

Other Webhook uses with MinIO:

• Invoke an AWS Lambda function
• Kick off a Continuous Integration (CI)
• Notify your team over Slack, Flowdock or HipChat

Benefits of webhooks

Up until recently the only way to get events from MinIO over HTTP was by polling which means creating a long-running task on your computer to checks for new messages in a loop. Polling has its benefits in that it’s easy to setup and doesn’t require an HTTP endpoint, but it can also be wasteful with resources.

Well designed APIs allow their users to receive a stream of events in real-time as they happen and now MinIO officially supports: NATS, Kafka, Redis, PostgreSQL and webhooks (HTTP/s). For more on the range of notifcations check out the docs: MinIO Notification docs.

Use-case: Thumbnail generator

Before we go ahead and explore how to configure your webhook endpoint let’s talk about a common use-case.

Imagine that you have a team uploading photos to your server in full-resolution, but that these need to be resized to thumbnails for an online catalog. MinIO Developer Harsha has put together an example project in Node.js called Thumbnailer. It listens for S3 put events with a file-extension of “.jpg” and then fetches, resizes and re-uploads the associated file.

1. Install Thumbnailer

Open a Terminal then create two new buckets for the code and install “thumbnailer”. (You will also need git and nodejs)

$ mc mb myminio/images 
$ mc mb myminio/images-processed
$ git clone https://github.com/minio/thumbnailer/
$ npm install

Before starting the thumbnailer code we need to edit the config file at config/default.json. Add the configuration for your server and then start the code.

2. Start Thumbnailer

$ NODE_ENV=webhook node thumbnail-webhook.js
Webhook listening on all interfaces at port 3000!

3. Configure a webhook ARN for your Minio Server

The server component of MinIO is configured through a file at ~/.minio/config.json — here you can configure one of the supported queues or notifcation systems. We will edit this file and add our HTTP/s endpoint, but first make sure that you are running the latest version of MinIO server and client.

Edit your ~/.minio/config.json file and add the following section:

"webhook": {
  "1": {
    "enable": true,
    "endpoint": "http://localhost:3000/"
}

The endpoint can be set to any site and TCP port on your network or the Internet with either an HTTP or HTTPs scheme. The endpoint needs to be live and reachable when you re/start your MinIO server.

4. See it in action

Set-up a notifcation for the webhook ARN:

$ mc events add myminio/images arn:minio:sqs:us-east-1:1:webhook — events put — suffix .jpg

In a separate Terminal upload a .jpg photo to the images bucket on your MinIO instance:

$ mc cp san-francisco.jpg myminio/images

Wait a few moments, then check the bucket’s contents with mc ls — you will see a thumbnail appear in the images-processed bucket.

$ mc ls myminio/images-processed
san-francisco-thumbnail.jpg

Over to you

You may already consume a number of webhooks in your application. MinIO now enables you to get notifications in near real-time about activity on your S3 buckets.

You can test out MinIO in no time with Docker:

docker run minio/minio

Head over to the project homepage for more instructions. And if you’d like to know more or just get connected, please join the team on the Slack community below:

• MinIO Slack