Enterprise-Grade Cloud Storage with NGINX Plus and MinIO

Designing Object Storage

Almost all applications need storage, but different apps need and use storage in particular ways. Take for example, a document store: it might not need to serve frequent read requests when small, but needs to scale as time progresses. Another application, such as an image gallery, needs to both satisfy requests quickly and scale with time.

These nuances make storage setup tough. However, everything is not gloomy — with the advent of object storage as the default way to store unstructured data, HTTP has become the default communication mode, standardizing how applications communicate with storage.

Still, the question remains: How do you build an object storage setup that is tailored for your application requirements, but still flexible?

Because object storage involves HTTP servers and clients, it makes sense to have a versatile web server like NGINX Plus in front to handle HTTP traffic. A lightweight object storage server such as Minio can be used to provide scalable storage at the backend. The flexibility of such a system is the key to creating an enterprise‑grade service.

With NGINX Plus, administrators can not only load balance incoming traffic — they can cache, throttle, terminate SSL/TLS, and even filter the traffic based on various parameters. Minio, on the other hand, offers a lightweight object storage server that is compatible with Amazon S3.

Minio is best suited for storing unstructured data such as photos, videos, log files, backups, and VM and container images. Minio server is light enough to be bundled with the application stack, similar to Node.js, Redis, and MySQL. Minio also supports a distributed mode, letting you pool multiple drives — even on different machines — into a single object storage server.

In this post we’ll explore some of the features of NGINX Plus in various use cases and learn how to combine them with Minio to set up a production‑grade, highly scalable, highly available, and stable object storage system.

NGINX Plus as a Reverse Proxy and Load Balancer

NGINX Plus is well known as a reverse proxy server. But why does one need a reverse proxy for Minio? Let’s look at some of the use cases:

• With an NGINX Plus reverse proxy in front of one or more Minio servers, you have the freedom to move Minio server instances to different machines/locations over time, without having to update clients or applications.
• NGINX Plus can load balance incoming traffic and spread it evenly across distributed Minio server instances.
• An NGINX Plus proxy can be part of a highly available object storage setup with Minio, using the Minio Client (mc) mirror command.

NGINX Plus reverse proxies client traffic by passing requests to a backend server, which is listening on the URL specified by the proxy_pass directive. In the following configuration snippet, a standalone Minio instance is running on localhost, so it’s available at http://localhost:9000. All requests coming on port 80 to the top‑level directory (/) at www.example.com are passed to Minio. NGINX Plus explicitly sets the Host header to its value in the original request.

server {
    listen 80;
    server_name www.example.com;
location / {
        proxy_set_header Host $http_host;
        proxy_pass       http://localhost:9000;
    }
}

If you have multiple Minio servers, load balance traffic among them by listing them in an upstream configuration block and referencing the upstream group in the proxy_pass directive:

upstream minio_servers {
    server minio-server-1:9000;
    server minio-server-2:9000;
}
server {
    listen 80;
    server_name www.example.com;
location / {
        proxy_set_header Host $http_host;
        proxy_pass       http://minio_servers;
    }
}

For more details about setting up NGINX or NGINX Plus as a proxy for Minio, see the Minio documentation.

SSL/TLS Termination

With HTTPS now becoming the default protocol for much web traffic, it makes sense to deploy an HTTPS server, rather than simply an HTTP server, for Minio. It’s fairly easy to set up NGINX Plus as an HTTPS server. You need a SSL/TLS certificate to get started, Let’s Encrypt provides free SSL/TLS certificates and integrates with NGINX Plus.

The next step is to edit the NGINX Plus configuration file. Here you need to specify the ssl parameter to the listen directive in the server block, then specify the files containing the server certificate and private key:

server {
    listen 80;
    server_name www.example.com;
    return 301 https://www.example.com$request_uri;
}
server {
    listen              443 ssl;
    server_name         www.example.com;
ssl_certificate     www.example.com.crt;
    ssl_certificate_key www.example.com.key;
    ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers         HIGH:!aNULL:!MD5;
location / {
        proxy_set_header Host $http_host;
        proxy_pass       http://localhost:9000;
    }
}

For more information about SSL/TLS termination, see the NGINX Plus Admin Guide.

Caching

Object storage servers are not known for their speed, but that doesn’t have to mean slow responses to clients. When you enable caching on the NGINX Plus server, it saves frequently accessed data which it can return to the client immediately without having to forward the request to the backend server.

Here is how it works. The NGINX Plus web cache sits in between a client and Minio, saving a copy of each requested content file. When a client requests content that is stored in the cache, NGINX returns it directly, without contacting Minio. This both improves response time to the client and reduces the load on the Minio server.

You set up the NGINX Plus cache for Minio with the proxy_cache_path and proxy_cache directives. The proxy_cache_path directive sets the location and configuration of the cache, and the proxy_cache directive activates it. For details, see A Guide to Caching with NGINX and NGINX Plus.

proxy_cache_path /path/to/cache levels=1:2 keys_zone=my_cache:10m max_size=10g inactive=60m
                 use_temp_path=off;
server {
...
    location / {
        proxy_cache      my_cache;
        proxy_set_header Host $http_host;
        proxy_pass       http://localhost:9000;
    }
}

Throttling

Sometimes you need to throttle requests for business or security reasons. With NGINX Plus, you can limit the available bandwidth, number of requests, or number of connections.

To throttle bandwidth, use the limit_rate directive. This example limits download speed to 200 KB per second:

server {
    ...
    location /images/ {
        limit_rate 200k;
        ...
    }
}

For request throttling, use the limit_req and limit_req_zone directives, as in this example which limits each unique IP address to 10 requests per second while allowing for bursts of 20 requests.

limit_req_zone $binary_remote_addr zone=my_req_limit:10m rate=10r/s;
server {
    ...
    location /images/ {
        limit_req zone=my_req_limit burst=20;
        ...
    }
}

To throttle the number of connections, use the limit_conn and limit_conn_zone directives. This example limits each unique IP address to 5 simultaneous connections.

limit_conn_zone $binary_remote_addr zone=my_conn_limit:10m;
server {
    ...
    location /images/ {
        limit_conn my_conn_limit 5;
        ...
    }
}

For more details, see the NGINX Plus Admin Guide.

Summary

In this post we demonstrated the use of several NGINX Plus features for load balancing in — particular, for load balancing in front of a Minio object storage server. The combination of NGINX Plus and Minio allows you to set up a flexible object storage server tailored for your application requirements.

This post was originally published on Nginx Blog.

We hangout on Slack, meet us here: slack.minio.io

While you’re at it, help us understand your use case and how we can help you better! Fill out our best of Minio deployment form (takes less than a minute), and get a chance to be featured on the Minio website and showcase your Minio private cloud design to Minio community.