Welcome to the third and final installment of our MinIO and CI/CD series. So far, we’ve discussed the basics of CI/CD concepts and how to build MinIO artifacts and how to test them in development. In this blog post, we’ll focus on Continuous Delivery and MinIO. We’ll show you how to deploy a MinIO cluster in a production environment using infrastructure as code to ensure anyone can read the resources installed and apply version control to any changes.
MinIO is very versatile and could be installed in almost any environment. MinIO conforms to multiple use cases for developers to have the same environment on a laptop that they work in production using the CI/CD concepts and pipelines we discussed. We showed you previously how to install MinIO as a docker container and even as a systemd service. Today we’ll show you how to deploy MinIO in distributed mode in a production Kubernetes cluster using an operator. We’ll use Terraform to deploy the infrastructure first, then we’ll deploy the required MinIO resources.
MinIO Network
First we’ll use Terraform to build the basic network needed for our infrastructure to get up and running. We are going to set up a VPC networking with 3 basic commonly used networking types. Within that network we’ll launch a Kubernetes cluster where we can deploy our MinIO workloads. The structure of our Terraform modules would look something like this
In order for the VPC to have different networks each subnet requires a unique non overlapping subnet. These subnets are split into CIDR blocks. For a handful, this is pretty easy to calculate, but for many subnets like we have here, Terraform provides a handy function cidrsubnet() to split the subnets for us based on a larger subnet we provide, in this case 10.0.0.0/16.
variable "minio_aws_vpc_cidr_block" { description = "AWS VPC CIDR block" type = string default = "10.0.0.0/16" } variable "minio_aws_vpc_cidr_newbits" { description = "AWS VPC CIDR new bits" type = number default = 4 }
The Private Network with NAT Gateway (NGW) will have outbound network access, but no inbound network access, with a private IP address and NAT Gateway.
variable "minio_private_ngw_cidr_blocks" { type = map(number) description = "Availability Zone CIDR Mapping for Private NGW subnets" default = { "us-east-1b" = 4 "us-east-1d" = 5 "us-east-1f" = 6 } }
Finally, we create an Isolated and Air-gapped network with neither outbound nor inbound internet access. This network is completely air gapped with only a private IP address.
variable "minio_private_isolated_cidr_blocks" { type = map(number) description = "Availability Zone CIDR Mapping for Private isolated subnets" default = { "us-east-1b" = 7 "us-east-1d" = 8 "us-east-1f" = 9 } }
Create a Kubernetes cluster on which we’ll deploy our MinIO cluster. The minio_aws_eks_cluster_subnet_ids will be provided by the VPC that we’ll create. Later, we’ll show how to stitch all this together in the deployment phase.
Note: In production you probably don’t want to have public access to the Kubernetes API endpoint because it could become a security issue as it will open up control of the cluster.
You will also need a couple of roles to ensure the Kubernetes cluster can communicate properly via the networks we’ve created, and those are defined at eks/main.tf#L1-L29. The Kubernetes cluster definition is as follows
The cluster takes in the API requests made from commands like kubectl, but there’s more to it than that – the workloads need to be scheduled somewhere. This is where a Kubernetes cluster node group is required. Below, we define the node group name, the type of instance and the desired group size. Since we have 3 AZs, we’ll create 3 nodes one for each of them.
variable "minio_aws_eks_node_group_name" { description = "AWS EKS Node group name" type = string default = "minio_aws_eks_node_group" } variable "minio_aws_eks_node_group_instance_types" { description = "AWS EKS Node group instance types" type = list(string) default = ["t3.large"] } variable "minio_aws_eks_node_group_desired_size" { description = "AWS EKS Node group desired size" type = number default = 3 } variable "minio_aws_eks_node_group_max_size" { description = "AWS EKS Node group max size" type = number default = 5 } variable "minio_aws_eks_node_group_min_size" { description = "AWS EKS Node group min size" type = number default = 1 }
You need a couple of roles to ensure the Kubernetes node group can communicate properly, and those are defined at eks/main.tf#L48-L81. The Kubernetes node group (workers) definition is as follows:
This configuration will launch a control plane with worker nodes in any of the 3 VPC networks we configured. We’ll show later the kubectl get no output once the cluster is launched.
MinIO Deployment
By now, we have all the necessary infrastructure in code form. Next, we’ll deploy these resources and create the cluster on which we’ll deploy MinIO.
Install Terraform using the following command
brew install terraform
Install aws CLI using the following command
brew install awscli
Create an AWS IAM user with the following policy. Note the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY after creating the user.
Set environmental variables for AWS, as they will be used by `terraform` and awscli.
Create a file called terraform.tfvars and set the following variable
hello_minio_aws_region = "us-east-1"
Create a file called main.tf and initialize the terraform AWS provider and S3 backend. Note that the S3 bucket needs to exist beforehand. We are using S3 backend to store the state so that it can be shared among developers and CI/CD processes alike without dealing with trying to keep local state in sync across the org.
Once the VPC has been created, the next step is to create the Kubernetes cluster. The only value we will use from the VPC creation is minio_aws_eks_cluster_subnet_ids. We’ll use the private subnets created by the VPC
Finally we’ll apply the configuration. While still in the hello_world directory run the following terraform commands. This will take about 15-20 minutes to get the entire infrastructure up and running. Towards the end, you should see an output similar to below:
Update your --kubeconfig default configuration to use the cluster we just created using aws eks command. The --region and --name are available from the previous output.
$ kubectl get no NAME STATUS ROLES AGE VERSION ip-10-0-105-186.ec2.internal Ready <none> 3d8h v1.23.9-eks-ba74326 ip-10-0-75-92.ec2.internal Ready <none> 3d8h v1.23.9-eks-ba74326 ip-10-0-94-57.ec2.internal Ready <none> 3d8h v1.23.9-eks-ba74326
Next, install EBS drivers so gp2 PVCs can mount. We are using gp2 because this is the default storage class supported by AWS.
Set credentials for the AWS secret using the same credentials used for awscli
Now we’re ready to deploy MinIO. First, clone the MinIO repository
$ git clone https://github.com/minio/operator.git
Since this is AWS, we need to update the storageClassName to gp2. Open the following file and update any references from storageClassName: standard to storageClassName: gp2. Each MinIO tenant has its own tenant.yaml that contains the storageClassName configuration. Based on the tenant you are using, be sure to update the storageClassName accordingly.
$ vim ./operator/examples/kustomization/base/tenant.yaml
Apply the resources to Kubernetes to install MinIO
If you notice the above output, each storage-lite-pool- is on a different worker node. Two of them share the same node because we have 3 nodes, but that is okay because we only have 3 availability zones (AZs). Basically there are 3 nodes in 3 AZs and 4 MinIO pods with 2 PVCs each which is reflected in the status 8 Online below.
With this information, we can set up Kubernetes port forwarding. We chose port 39443 for the host, but this could be anything, just be sure to use this same port when accessing the console through a web browser.
$ kubectl -n tenant-lite port-forward svc/storage-lite-console 39443:9443 Forwarding from 127.0.0.1:39443 -> 9443 Forwarding from [::1]:39443 -> 9443
Access MinIO Operator Console through the web browser using the following credentials:
URL: https://localhost:39443
User: minio
Password: minio123
You now have a fully production setup of a distributed MinIO cluster. Here is how you can automate it using Jenkins
Here is the execute shell command in text format
export PATH=$PATH:/usr/local/bin
cd ci-cd-deploy/terraform/aws/hello_world/
terraform init
terraform plan
terraform apply -auto-approve
Final Thoughts
In these past few blogs of the CI/CD series we’ve shown you how nimble and flexible MinIO is. You can build it into anything you want using Packer and deploy it in VMs or Kubernetes clusters wherever it is needed. This allows your developers to have as close to a production infrastructure as possible in their development environment, while at the same time leveraging powerful security features such as Server Side Object Encryption and managing IAM policies for restricting access to buckets.
In a production environment, you might want to restrict the IAM user to a specific policy but that really depends on your use cases. For demonstration purposes, we kept things simple with a broad policy, but in production you would want to narrow it down to specific resources and groups of users. In a later blog we’ll show some of the best practices on how to design your infrastructure for different AZs and regions.
Would you like to try automating the kubectl part as well with Jenkins instead of applying manually? Let us know what type of pipeline you’ve built using our tutorials for planning, deploying, scaling and securing MinIO across the multicloud, and reach out to us on our Slack and share your pipelines!
Erasure Code Calculator
Reference Hardware
