CI/CD Deploy with MinIO distributed cluster on Kubernetes

CI/CD Deploy with MinIO distributed cluster on Kubernetes

Welcome to the third and final installment of our MinIO and CI/CD series. So far, we’ve discussed the basics of CI/CD concepts and how to build MinIO artifacts and how to test them in development. In this blog post, we’ll focus on Continuous Delivery and MinIO. We’ll show you how to deploy a MinIO cluster in a production environment using infrastructure as code to ensure anyone can read the resources installed and apply version control to any changes.

MinIO is very versatile and could be installed in almost any environment. MinIO conforms to multiple use cases for developers to have the same environment on a laptop that they work in production using the CI/CD concepts and pipelines we discussed. We showed you previously how to install MinIO as a docker container and even as a systemd service. Today we’ll show you how to deploy MinIO in distributed mode in a production Kubernetes cluster using an operator. We’ll use Terraform to deploy the infrastructure first, then we’ll deploy the required MinIO resources.

MinIO Network

First we’ll use Terraform to build the basic network needed for our infrastructure to get up and running. We are going to set up a VPC networking with 3 basic commonly used networking types. Within that network we’ll launch a Kubernetes cluster where we can deploy our MinIO workloads. The structure of our Terraform modules would look something like this

modules
├── eks
│   ├── main.tf
│   ├── outputs.tf
│   └── variables.tf
└── vpc
    ├── main.tf
    ├── outputs.tf
    └── variables.tf

https://github.com/minio/blog-assets/tree/main/ci-cd-deploy/terraform/aws/modules

In order for the VPC to have different networks each subnet requires a unique non overlapping subnet. These subnets are split into CIDR blocks. For a handful, this is pretty easy to calculate, but for many subnets like we have here, Terraform provides a handy function cidrsubnet() to split the subnets for us based on a larger subnet we provide, in this case 10.0.0.0/16.

variable "minio_aws_vpc_cidr_block" {
  description = "AWS VPC CIDR block"
  type        = string
  default     = "10.0.0.0/16"
}

variable "minio_aws_vpc_cidr_newbits" {
  description = "AWS VPC CIDR new bits"
  type        = number
  default     = 4
}

vpc/variables.tf#L1-L11

Define the VPC resource in Terraform. Any subnet created will be based on this VPC.

resource "aws_vpc" "minio_aws_vpc" {

  cidr_block           = var.minio_aws_vpc_cidr_block
  instance_tenancy     = "default"
  enable_dns_hostnames = true

}

vpc/main.tf#L1-L7

Set up 3 different networks: Public, Private and Isolated.

The Public Network with Internet Gateway (IGW) will have inbound and outbound internet access with a public IP and an Internet Gateway.

variable "minio_public_igw_cidr_blocks" {
  type = map(number)
  description = "Availability Zone CIDR Mapping for Public IGW subnets"

  default = {
    "us-east-1b" = 1
    "us-east-1d" = 2
    "us-east-1f" = 3
  }
}

vpc/variables.tf#L15-L24

The aws_subnet resource will loop 3 times creating 3 subnets in the public VPC

resource "aws_subnet" "minio_aws_subnet_public_igw" {

  for_each = var.minio_public_igw_cidr_blocks

  vpc_id            = aws_vpc.minio_aws_vpc.id
  cidr_block        = cidrsubnet(aws_vpc.minio_aws_vpc.cidr_block, var.minio_aws_vpc_cidr_newbits, each.value)
  availability_zone = each.key

  map_public_ip_on_launch = true
}

resource "aws_route_table" "minio_aws_route_table_public_igw" {

  vpc_id = aws_vpc.minio_aws_vpc.id

}

resource "aws_route_table_association" "minio_aws_route_table_association_public_igw" {

  for_each       = aws_subnet.minio_aws_subnet_public_igw

  subnet_id      = each.value.id
  route_table_id = aws_route_table.minio_aws_route_table_public_igw.id
}

resource "aws_internet_gateway" "minio_aws_internet_gateway" {

  vpc_id = aws_vpc.minio_aws_vpc.id

}

resource "aws_route" "minio_aws_route_public_igw" {
  route_table_id         = aws_route_table.minio_aws_route_table_public_igw.id
  destination_cidr_block = "0.0.0.0/0"
  gateway_id             = aws_internet_gateway.minio_aws_internet_gateway.id
}

vpc/main.tf#L11-L46

The Private Network with NAT Gateway (NGW) will have outbound network access, but no inbound network access, with a private IP address and NAT Gateway.

variable "minio_private_ngw_cidr_blocks" {
  type = map(number)
  description = "Availability Zone CIDR Mapping for Private NGW subnets"

  default = {
    "us-east-1b" = 4
    "us-east-1d" = 5
    "us-east-1f" = 6
  }
}

vpc/variables.tf#L26L-L35

The aws_subnet resource will loop 3 times creating 3 subnets in the private VPC

resource "aws_subnet" "minio_aws_subnet_private_ngw" {

  for_each = var.minio_private_ngw_cidr_blocks

  vpc_id            = aws_vpc.minio_aws_vpc.id
  cidr_block        = cidrsubnet(aws_vpc.minio_aws_vpc.cidr_block, var.minio_aws_vpc_cidr_newbits, each.value)
  availability_zone = each.key
}

resource "aws_route_table" "minio_aws_route_table_private_ngw" {

  for_each = var.minio_private_ngw_cidr_blocks

  vpc_id = aws_vpc.minio_aws_vpc.id
}

resource "aws_route_table_association" "minio_aws_route_table_association_private_ngw" {

  for_each = var.minio_private_ngw_cidr_blocks

  subnet_id      = aws_subnet.minio_aws_subnet_private_ngw[each.key].id
  route_table_id = aws_route_table.minio_aws_route_table_private_ngw[each.key].id
}

resource "aws_eip" "minio_aws_eip_nat" {

  for_each = var.minio_private_ngw_cidr_blocks

  vpc = true
}

resource "aws_nat_gateway" "minio_aws_nat_gateway" {

  for_each = var.minio_private_ngw_cidr_blocks

  subnet_id     = aws_subnet.minio_aws_subnet_public_igw[each.key].id
  allocation_id = aws_eip.minio_aws_eip_nat[each.key].id

  depends_on    = [aws_internet_gateway.minio_aws_internet_gateway]
}

resource "aws_route" "minio_aws_route_private_ngw" {

  for_each = var.minio_private_ngw_cidr_blocks

  route_table_id         = aws_route_table.minio_aws_route_table_private_ngw[each.key].id
  destination_cidr_block = "0.0.0.0/0"
  nat_gateway_id         = aws_nat_gateway.minio_aws_nat_gateway[each.key].id
}

vpc/main.tf#L50-L98

Finally, we create an Isolated and Air-gapped network with neither outbound nor inbound internet access. This network is completely air gapped with only a private IP address.

variable "minio_private_isolated_cidr_blocks" {
  type = map(number)
  description = "Availability Zone CIDR Mapping for Private isolated subnets"

  default = {
    "us-east-1b" = 7
    "us-east-1d" = 8
    "us-east-1f" = 9
  }
}

vpc/variables.tf#L37-L46

The aws_subnet resource will loop 3 times creating 3 subnets in the isolated/air-gapped VPC

resource "aws_subnet" "minio_aws_subnet_private_isolated" {

  for_each = var.minio_private_isolated_cidr_blocks

  vpc_id            = aws_vpc.minio_aws_vpc.id
  cidr_block        = cidrsubnet(aws_vpc.minio_aws_vpc.cidr_block, var.minio_aws_vpc_cidr_newbits, each.value)
  availability_zone = each.key
}

resource "aws_route_table" "minio_aws_route_table_private_isolated" {

  vpc_id = aws_vpc.minio_aws_vpc.id

}

resource "aws_route_table_association" "minio_aws_route_table_association_private_isolated" {

  for_each = aws_subnet.minio_aws_subnet_private_isolated

  subnet_id      = each.value.id
  route_table_id = aws_route_table.minio_aws_route_table_private_isolated.id
}

vpc/main.tf#L102-L123

MinIO Kubernetes Cluster

Create a Kubernetes cluster on which we’ll deploy our MinIO cluster. The minio_aws_eks_cluster_subnet_ids will be provided by the VPC that we’ll create. Later, we’ll show how to stitch all this together in the deployment phase.

variable "minio_aws_eks_cluster_subnet_ids" {
  description = "AWS EKS Cluster subnet IDs"
  type        = list(string)
}

variable "minio_aws_eks_cluster_name" {
  description = "AWS EKS Cluster name"
  type        = string
  default     = "minio_aws_eks_cluster"
}

variable "minio_aws_eks_cluster_endpoint_private_access" {
  description = "AWS EKS Cluster endpoint private access"
  type        = bool
  default     = true
}

variable "minio_aws_eks_cluster_endpoint_public_access" {
  description = "AWS EKS Cluster endpoint public access"
  type        = bool
  default     = true
}

variable "minio_aws_eks_cluster_public_access_cidrs" {
  description = "AWS EKS Cluster public access cidrs"
  type        = list(string)
  default     = ["0.0.0.0/0"]
}

eks/variables.tf#L1-L28

Note: In production you probably don’t want to have public access to the Kubernetes API endpoint because it could become a security issue as it will open up control of the cluster.

You will also need a couple of roles to ensure the Kubernetes cluster can communicate properly via the networks we’ve created, and those are defined at eks/main.tf#L1-L29. The Kubernetes cluster definition is as follows

resource "aws_eks_cluster" "minio_aws_eks_cluster" {
  name = var.minio_aws_eks_cluster_name
  role_arn = aws_iam_role.minio_aws_iam_role_eks_cluster.arn

  vpc_config {
    subnet_ids              = var.minio_aws_eks_cluster_subnet_ids
    endpoint_private_access = var.minio_aws_eks_cluster_endpoint_private_access
    endpoint_public_access  = var.minio_aws_eks_cluster_endpoint_public_access
    public_access_cidrs     = var.minio_aws_eks_cluster_public_access_cidrs
  }

  depends_on = [
    aws_iam_role.minio_aws_iam_role_eks_cluster,
  ]

}

eks/main.tf#L31-L46

The cluster takes in the API requests made from commands like kubectl, but there’s more to it than that – the workloads need to be scheduled somewhere. This is where a Kubernetes cluster node group is required. Below, we define the node group name, the type of instance and the desired group size. Since we have 3 AZs, we’ll create 3 nodes one for each of them.

variable "minio_aws_eks_node_group_name" {
  description = "AWS EKS Node group name"
  type        = string
  default     = "minio_aws_eks_node_group"
}

variable "minio_aws_eks_node_group_instance_types" {
  description = "AWS EKS Node group instance types"
  type        = list(string)
  default     = ["t3.large"]
}

variable "minio_aws_eks_node_group_desired_size" {
  description = "AWS EKS Node group desired size"
  type        = number
  default     = 3
}

variable "minio_aws_eks_node_group_max_size" {
  description = "AWS EKS Node group max size"
  type        = number
  default     = 5
}

variable "minio_aws_eks_node_group_min_size" {
  description = "AWS EKS Node group min size"
  type        = number
  default     = 1
}

eks/variables.tf#L30-L58

You need a couple of roles to ensure the Kubernetes node group can communicate properly, and those are defined at eks/main.tf#L48-L81. The Kubernetes node group (workers) definition is as follows:

resource "aws_eks_node_group" "minio_aws_eks_node_group" {
  cluster_name    = aws_eks_cluster.minio_aws_eks_cluster.name
  node_group_name = var.minio_aws_eks_node_group_name
  node_role_arn   = aws_iam_role.minio_aws_iam_role_eks_worker.arn
  subnet_ids      = var.minio_aws_eks_cluster_subnet_ids
  instance_types  = var.minio_aws_eks_node_group_instance_types

  scaling_config {
    desired_size = var.minio_aws_eks_node_group_desired_size
    max_size     = var.minio_aws_eks_node_group_max_size
    min_size     = var.minio_aws_eks_node_group_min_size
  }

  depends_on = [
    aws_iam_role.minio_aws_iam_role_eks_worker,
  ]

}

eks/main.tf#L83-L100

This configuration will launch a control plane with worker nodes in any of the 3 VPC networks we configured. We’ll show later the kubectl get no output once the cluster is launched.

MinIO Deployment

By now, we have all the necessary infrastructure in code form. Next, we’ll deploy these resources and create the cluster on which we’ll deploy MinIO.

Install Terraform using the following command

brew install terraform

Install aws CLI using the following command

brew install awscli

Create an AWS IAM user with the following policy. Note the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY after creating the user.

Set environmental variables for AWS, as they will be used by `terraform` and awscli.

$ export AWS_ACCESS_KEY_ID=<access_key>
$ export AWS_SECRET_ACCESS_KEY=<secret_key>

Create a folder called hello_world in the same directory as modules using the structure below

.

├── hello_world

│   ├── main.tf

│   ├── outputs.tf

│   ├── terraform.tfvars

│   └── variables.tf

├── modules

│   ├── eks

│   └── vpc


https://github.com/minio/blog-assets/tree/main/ci-cd-deploy/terraform/aws/hello_world

Create a file called terraform.tfvars and set the following variable

hello_minio_aws_region  = "us-east-1"

Create a file called main.tf and initialize the terraform AWS provider and S3 backend. Note that the S3 bucket needs to exist beforehand. We are using S3 backend to store the state so that it can be shared among developers and CI/CD processes alike without dealing with trying to keep local state in sync across the org.

terraform {
  required_version = ">= 1.0"

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.31.0"
    }
  }

  backend "s3" {
    bucket = "aj-terraform-bucket"
    key    = "tf/aj/mo"
    region = "us-east-1"
  }

}

provider "aws" {
  region  = var.hello_minio_aws_region
}

hello_world/main.tf#L1-L21

Setting the backend bucket and key as variables is not supported, so those values need to be hard coded.

Call the VPC module from main.tf and name it hello_minio_aws_vpc

module "hello_minio_aws_vpc" {
  source = "../modules/vpc"

  minio_aws_vpc_cidr_block   = var.hello_minio_aws_vpc_cidr_block
  minio_aws_vpc_cidr_newbits = var.hello_minio_aws_vpc_cidr_newbits

  minio_public_igw_cidr_blocks       = var.hello_minio_public_igw_cidr_blocks
  minio_private_ngw_cidr_blocks      = var.hello_minio_private_ngw_cidr_blocks
  minio_private_isolated_cidr_blocks = var.hello_minio_private_isolated_cidr_blocks

}

hello_world/main.tf#L23-L33

These are the variables required by vpc module

hello_minio_aws_vpc_cidr_block = "10.0.0.0/16"

hello_minio_aws_vpc_cidr_newbits = 4

hello_minio_public_igw_cidr_blocks = {
  "us-east-1b" = 1
  "us-east-1d" = 2
  "us-east-1f" = 3
}

hello_minio_private_ngw_cidr_blocks = {
  "us-east-1b" = 4
  "us-east-1d" = 5
  "us-east-1f" = 6
}

hello_minio_private_isolated_cidr_blocks = {
  "us-east-1b" = 7
  "us-east-1d" = 8
  "us-east-1f" = 9
}

hello_world/terraform.tfvars#L3-L22

Once the VPC has been created, the next step is to create the Kubernetes cluster. The only value we will use from the VPC creation is minio_aws_eks_cluster_subnet_ids. We’ll use the private subnets created by the VPC

module "hello_minio_aws_eks_cluster" {
  source = "../modules/eks"

  minio_aws_eks_cluster_name                    = var.hello_minio_aws_eks_cluster_name
  minio_aws_eks_cluster_endpoint_private_access = var.hello_minio_aws_eks_cluster_endpoint_private_access
  minio_aws_eks_cluster_endpoint_public_access  = var.hello_minio_aws_eks_cluster_endpoint_public_access
  minio_aws_eks_cluster_public_access_cidrs     = var.hello_minio_aws_eks_cluster_public_access_cidrs
  minio_aws_eks_cluster_subnet_ids              = values(module.hello_minio_aws_vpc.minio_aws_subnet_private_ngw_map)
  minio_aws_eks_node_group_name                 = var.hello_minio_aws_eks_node_group_name
  minio_aws_eks_node_group_instance_types       = var.hello_minio_aws_eks_node_group_instance_types
  minio_aws_eks_node_group_desired_size         = var.hello_minio_aws_eks_node_group_desired_size
  minio_aws_eks_node_group_max_size             = var.hello_minio_aws_eks_node_group_max_size
  minio_aws_eks_node_group_min_size             = var.hello_minio_aws_eks_node_group_min_size

}

hello_world/main.tf#L37-L51

These are the variables required by EKS module

hello_minio_aws_eks_cluster_name = "hello_minio_aws_eks_cluster"
hello_minio_aws_eks_cluster_endpoint_private_access = true
hello_minio_aws_eks_cluster_endpoint_public_access = true
hello_minio_aws_eks_cluster_public_access_cidrs = ["0.0.0.0/0"]
hello_minio_aws_eks_node_group_name = "hello_minio_aws_eks_node_group"
hello_minio_aws_eks_node_group_instance_types = ["t3.large"]
hello_minio_aws_eks_node_group_desired_size = 3
hello_minio_aws_eks_node_group_max_size = 5
hello_minio_aws_eks_node_group_min_size = 1

hello_world/terraform.tfvars#L24-L32

Finally we’ll apply the configuration. While still in the hello_world directory run the following terraform commands. This will take about 15-20 minutes to get the entire infrastructure up and running. Towards the end, you should see an output similar to below:

$ terraform init


…TRUNCATED…


$ terraform apply


…TRUNCATED…


hello_minio_aws_eks_cluster_name = "hello_minio_aws_eks_cluster"
hello_minio_aws_eks_cluster_region = "us-east-1"

…TRUNCATED…

Finished: SUCCESS

Update your --kubeconfig default configuration to use the cluster we just created using aws eks command. The --region and --name are available from the previous output.

$ aws eks --region us-east-1 update-kubeconfig \
    --name hello_minio_aws_eks_cluster

Check to verify that you can get a list of nodes

$ kubectl get no
NAME                           STATUS   ROLES    AGE    VERSION
ip-10-0-105-186.ec2.internal   Ready    <none>   3d8h   v1.23.9-eks-ba74326
ip-10-0-75-92.ec2.internal     Ready    <none>   3d8h   v1.23.9-eks-ba74326
ip-10-0-94-57.ec2.internal     Ready    <none>   3d8h   v1.23.9-eks-ba74326

Next, install EBS drivers so gp2 PVCs can mount. We are using gp2 because this is the default storage class supported by AWS.

Set credentials for the AWS secret using the same credentials used for awscli

kubectl create secret generic aws-secret \
    --namespace kube-system \
    --from-literal "key_id=${AWS_ACCESS_KEY_ID}" \
    --from-literal "access_key=${AWS_SECRET_ACCESS_KEY}"

Apply the EBS drivers resources:

$ kubectl apply -k "github.com/kubernetes-sigs/aws-ebs-csi-driver/deploy/kubernetes/overlays/stable/?ref=release-1.12"

Your Kubernetes cluster should be ready now.

Now we’re ready to deploy MinIO. First, clone the MinIO repository

$ git clone https://github.com/minio/operator.git

Since this is AWS, we need to update the storageClassName to gp2. Open the following file and update any references from storageClassName: standard to storageClassName: gp2. Each MinIO tenant has its own tenant.yaml that contains the storageClassName configuration. Based on the tenant you are using, be sure to update the storageClassName accordingly.

$ vim ./operator/examples/kustomization/base/tenant.yaml

Apply the resources to Kubernetes to install MinIO

$ kubectl apply -k operator/resources

$ kubectl apply -k operator/examples/kustomization/tenant-lite

Wait at least 5 minutes for the resources to come up, then verify that MinIO is up and running.

$ kubectl -n tenant-lite get po -o wide
NAME                                           READY   STATUS    RESTARTS      AGE   IP            NODE                           NOMINATED NODE   READINESS GATES
storage-lite-log-0                             1/1     Running   0             17m   10.0.94.169   ip-10-0-94-57.ec2.internal     <none>           <none>
storage-lite-log-search-api-66f7db97f5-j268m   1/1     Running   3 (17m ago)   17m   10.0.93.40    ip-10-0-94-57.ec2.internal     <none>           <none>
storage-lite-pool-0-0                          1/1     Running   0             17m   10.0.88.36    ip-10-0-94-57.ec2.internal     <none>           <none>
storage-lite-pool-0-1                          1/1     Running   0             17m   10.0.104.48   ip-10-0-105-186.ec2.internal   <none>           <none>
storage-lite-pool-0-2                          1/1     Running   0             17m   10.0.71.81    ip-10-0-75-92.ec2.internal     <none>           <none>
storage-lite-pool-0-3                          1/1     Running   0             17m   10.0.94.183   ip-10-0-94-57.ec2.internal     <none>           <none>
storage-lite-prometheus-0                      2/2     Running   0             15m   10.0.85.181   ip-10-0-94-57.ec2.internal     <none>           <none>

If you notice the above output, each storage-lite-pool- is on a different worker node. Two of them share the same node because we have 3 nodes, but that is okay because we only have 3 availability zones (AZs). Basically there are 3 nodes in 3 AZs and 4 MinIO pods with 2 PVCs each which is reflected in the status 8 Online below.

$ kubectl -n tenant-lite logs storage-lite-pool-0-0

…TRUNCATED…

Status:         8 Online, 0 Offline.
API: https://minio.tenant-lite.svc.cluster.local
Console: https://10.0.88.36:9443 https://127.0.0.1:9443

Documentation: https://min.io/docs/minio/linux/index.html

You will need the TCP port of the MinIO console; in this case it is 9443.

$ kubectl -n tenant-lite get svc | grep -i console
storage-lite-console   ClusterIP   172.20.26.209   <none>        9443/TCP   6s

With this information, we can set up Kubernetes port forwarding. We chose port 39443 for the host, but this could be anything, just be sure to use this same port when accessing the console through a web browser.

$ kubectl -n tenant-lite port-forward svc/storage-lite-console 39443:9443

Forwarding from 127.0.0.1:39443 -> 9443

Forwarding from [::1]:39443 -> 9443

Access MinIO Operator Console through the web browser using the following credentials:

URL: https://localhost:39443

User: minio

Password: minio123

You now have a fully production setup of a distributed MinIO cluster. Here is how you can automate it using Jenkins

Here is the execute shell command in text format

export PATH=$PATH:/usr/local/bin

cd ci-cd-deploy/terraform/aws/hello_world/
terraform init
terraform plan
terraform apply -auto-approve

Final Thoughts

In these past few blogs of the CI/CD series we’ve shown you how nimble and flexible MinIO is. You can build it into anything you want using Packer and deploy it in VMs or Kubernetes clusters wherever it is needed. This allows your developers to have as close to a production infrastructure as possible in their development environment, while at the same time leveraging powerful security features such as Server Side Object Encryption and managing IAM policies for restricting access to buckets.

In a production environment, you might want to restrict the IAM user to a specific policy but that really depends on your use cases. For demonstration purposes, we kept things simple with a broad policy, but in production you would want to narrow it down to specific resources and groups of users. In a later blog we’ll show some of the best practices on how to design your infrastructure for different AZs and regions.

Would you like to try automating the kubectl part as well with Jenkins instead of applying manually? Let us know what type of pipeline you’ve built using our tutorials for planning, deploying, scaling and securing MinIO across the multicloud, and reach out to us on our Slack and share your pipelines!