MinIO Audit Logs in ElasticSearch in Kubernetes

AJ AJ on AI/ML 22 May 2024

Whether you are on-prem or in the Cloud, you want to ensure you set up your tools and processes in a homogenous way. No matter where the infrastructure is being accessed you want to ensure the tools used to interact with the various pieces of infrastructure are similar to the other regions.

With that in mind, having insight into your storage infrastructure is paramount when it comes to deploying your own MinIO object storage infrastructure. When any unexpected activity is happening in your storage infrastructure you would want to surface those logs quickly and in a succinct manner. This tutorial will give you a full overview of how you can surface MinIO audit logs in ElasticSearch so they can be searchable.

Setup Configuration

Deploy MinIO operator on a Kubernetes cluster

kubectl apply -k github.com/minio/operator/

We need to deploy the tenant with ElasticSearch WebHook enabled

Encode your user and password in base64

# Encode your user and password in base64

$ echo -n 'user:password' | base64

dXNlcjpwYXNzd29yZA==

Create the namespace where the secret will be located

kubectl create namespace tenant-lite

Create the secret using the base64 user/password generated earlier. Create a file called secret.yaml and save the contents in there.

apiVersion: v1

kind: Secret

metadata:

type: Opaque

stringData:

token: "Basic dXNlcjpwYXNzd29yZA=="

Apply the secret as follows

k apply -f secret.yaml

Let’s configure the webhook so that it can fetch the credentials from the secret

apiVersion: minio.min.io/v2

kind: Tenant

metadata:

namespace: minio-tenant

spec:

env:

- name: MINIO_AUDIT_WEBHOOK_ENABLE_TESTING

value: "on"

- name: MINIO_AUDIT_WEBHOOK_ENDPOINT_TESTING

value: "http://<HOST>:9200/ajtest/ajtest2"

- name: MINIO_AUDIT_WEBHOOK_AUTH_TOKEN_TESTING

valueFrom:

secretKeyRef:

key: token

optional: false # value coming from secret

- name: MINIO_ROOT_USER

value: minio

- name: MINIO_ROOT_PASSWORD

value: minio123

Install the tenant using Kustomize

kubectl apply -k github.com/minio/operator/examples/kustomization/tenant-lite

Lets curl the index to see the output

curl -u <USER>:<PASSWORD> -X GET "<HOST>:9200/ajtest/_search?pretty" -H 'Content-Type: application/json'

You should expect to see audit logs similar to below

[TRUNCATED]

"remotehost" : "10.244.4.2",

"requestID" : "17329E78E6E41E8E",

"userAgent" : "MinIO (linux; arm64) madmin-go/0.0.1",

"requestQuery" : {

"key" : "audit_webhook:storage-lite-log-search-api"

"requestHeader" : {

"Authorization" : "AWS4-HMAC-SHA256 Credential=minio/20221220//s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=2ae3c3d8693619c4407723146091ae3b884827bc6dc8b9d12c9380f4a50db3b7",

"User-Agent" : "MinIO (linux; arm64) madmin-go/0.0.1",

"X-Amz-Content-Sha256" : "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",

"X-Amz-Date" : "20221220T214413Z"

"responseHeader" : {

"Accept-Ranges" : "bytes",

"Content-Length" : "242",

"Content-Security-Policy" : "block-all-mixed-content",

"Content-Type" : "application/json",

"Server" : "MinIO",

"Strict-Transport-Security" : "max-age=31536000; includeSubDomains",

"Vary" : "Origin,Accept-Encoding",

"X-Amz-Request-Id" : "17329E78E6E41E8E",

"X-Content-Type-Options" : "nosniff",

"X-Xss-Protection" : "1; mode=block"

}

[TRUNCATED]

{

"_index" : "ajtest",

"_type" : "ajtest2",

"_id" : "ADB-MYUBuuSsBbj2iI88",

"_score" : 1.0,

"_source" : { }

}

]

}

The output will be very long so we truncated the output so it's legible.

Final Thoughts

As you can see, it's very important to take notice of these audit logs quickly. In fact you can set Alerts via your monitoring system or directly via ElasticSearch to send you alerts based on certain logs so you can be on top of your MinIO storage infrastructure.

More importantly, you need to think about the cloud as an operating model rather than a location. Meaning ensure no matter where you deploy your tools and infrastructure, whether it’d be Hybrid Cloud, Private Cloud, On-Prem, you want to make sure you send your MinIO audit logs to a log processor, doesn’t have to be ElasticSearch. This way no matter where you deploy MinIO you ensure it gets monitored every step of the way.

If you have any questions on how to get your MinIO audit logs into ElasticSearch be sure to reach out to us on Slack!