Store your Node.js application logs to Minio

Logging remains one of the major concerns among developers. Typically, various components like the web servers, databases store logs at different locations. When something goes wrong, developers need to look all around to find the relevant logs.

Wouldn’t it be nice to have your application logs automatically stored at a predefined location. You can do this with Fluentd and Minio. In this post, we’ll see how to setup Fluentd and Minio to automatically create logs from your Node.js application and store them on your Minio server. But first, a brief introduction to Fluentd.

Quick note, I am using Ubuntu 15.10 for demo purposes in this post. If you’re using other OS, please use the corresponding Fluentd commands. You can find that information from the links I have added wherever relevant.

Fluentd

Fluentd is an open source data collection platform that lets you unify the data collection process. It decouples data sources from backend systems by providing a unified logging layer in between, allowing developers to easily generate and store logs from various applications to their preferred location.

Get started

To begin with, lets create logs from a Node.js application and store them to local file system. Once this is successfully done, we’ll move on to setting up the log file storage destination to a Minio server.

• Install Fluentd — You can either install a stable distribution of Fluentd, called td-agent (made available by Treasure Data, the company backing Fluentd) or the Ruby based daemon. If you’re are confused, read the differences between Fluentd and td-agent here.

I am using Ubuntu 15.10, and installed the td-agent using

curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-trusty-td-agent2.sh | sh

If you’re using other platforms, you’ll need to use the corresponding script. Read detailed installation instructions here.

Once successfully installed, you should be able to launch the Fluentd daemon using

$ /etc/init.d/td-agent start

Check the status using

$ /etc/init.d/td-agent status

Modify the Fluentd configuration file — Let us now set up the Fluentd forward input plugin to accept logs from the Node.js application. Note that the plugin itself is included in the core, so we’ll just need to edit the configuration file. Open the file “/etc/td-agent/td-agent.conf” and set the source tag as following

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

Here, we just set the input type to forward and the port as 24224 to listen for incoming connections. Note that this should be only used in your private network, as opening port 24224 on the internet would leave your system vulnerable to attacks. For public network use, consider using the in_secure_plugin.

• Configuring the Node.js application to send logs to Fluentd — We’ll now see how to log events from your Node.js application. To do so, you’ll need to include fluent-logger as a dependency in your Node.js application. In my sample app, I did it like this:

Package.json

{
  "name": "node-example",
  "version": "0.0.1",
  "dependencies": {
    "express": "2.5.9",
    "fluent-logger": "0.1.0"
  }
}

Next, I called logger.configure() and logger.emit() to set up logging in my web app.

app.js

var express = require('express');
var app = express.createServer(express.logger());

var logger = require('fluent-logger');
logger.configure('fluentd.test', {host: 'localhost', port: 24224});

app.get('/', function(request, response) {
  logger.emit('follow', {from: 'userA', to: 'userB'});
  response.send('Hello World!');
});
var port = process.env.PORT || 3000;
app.listen(port, function() {
  console.log("Listening on " + port);
});

This sample app configures the logger to send logs to localhost port 24224 (same port that Fluentd config is expecting), and then, whenever “/” is accessed, a log entry saying “from userA to userB” is made.

You can access this app on your browser at http://localhost:3000, after you execute the node module by running

$ node app.js

If everything is fine, you should be able to access the logs at /var/log/td-agent/td-agent.log.

We’re yet only halfway there. In the next steps we’ll see how to set up Minio server and client and then configure Fluentd to store the logs to our own Minio server.

• Set up Minio — We’ll now set up our own Minio server that will serve as the home for all the logs generated by the Node.js application. Download Minio server and the Minio client from here. Then navigate to the folder where you downloaded both the applications and type

$ chmod +x minio
$ ./minio --help
$ ./minio server <Folder_That_You_Wish_To_Be_Root>

This will generate the access key and secret key, you’ll need these to login to your Minio server.

Next, configure Minio Client:

$ chmod 755 mc
$ ./mc config host add myminio http://localhost:9000 <Access_Key> <Secret_Key>

Then create a bucket using the Minio client, this will be the destination for inbound logs from Fluentd. (You can also do this via the Minio server web interface.)

$ mc mb myminio/NodeAppLogs

• Setup Fluentd to send logs to Minio — Finally, we’ll configure Fluentd so that it sends logs across to Minio server instead of the local file system. To do this, you need to open the Fluentd configuration file available here “/etc/td-agent/td-agent.conf” (same file that we edited in the second step). Once you have the file opened, look for the <match> tag and replace it with

<match>
 @type s3
 aws_key_id <minio_access_key>
 aws_sec_key <minio_secret_key>
 s3_bucket <target_bucket>
 s3_endpoint <minio_server_URL>
 path logs/
 force_path_style true
 buffer_path /var/log/td-agent/s3
 time_slice_format %Y%m%d%H%M
 time_slice_wait 10m
 utc
 buffer_chunk_limit 256m
</match>

You just need to change the minio_access_key, minio_secret_key, target_bucketand minio_server_URL fields. Once done, save the file and restart Fluentd by

$ sudo /etc/init.d/td-agent restart

If everything is fine, you’ll be able to see your Node.js logs in Minio server

While you’re at it, help us understand your use case and how we can help you better! Fill out our best of Minio deployment form (takes less than a minute), and get a chance to be featured on the Minio website and showcase your Minio private cloud design to Minio community.