How to Install MinIO in Distributed Mode on Azure

Cesar Celis Hernandez -

Enterprises love Azure and they love MinIO on Azure. According to Flexera’s 2022 State of the Cloud Report, enterprises run more workloads on Azure than AWS. A significant portion of the more than one million public cloud deployments of MinIO run in Azure.

We built a managed application in Azure Marketplace that can be deployed in a few clicks. Deploying MinIO as a managed app is a streamlined experience that offloads the effort from you through automations and transparent integrations with Azure components. It’s the quickest way to get started developing applications using MinIO object storage on Azure.

We also know that you, like us, want the flexibility to do it your way. So, here’s how to Install MinIO on Azure virtual machines-  and customize it all to you and your environment’s specifications.

This is a guide on how to deploy MinIO on Azure. We will create four virtual machines, each with four 1 TiB Premium SSD drives. Then we will SSH into our virtual machines and install MinIO in distributed mode.

Prerequisites:

  1. You need an Azure account, by either providing a credit card or by getting the account from your company/administrator.
  2. Install Putty on Windows or use Terminal on Mac, or use a similar program to connect via ssh to the virtual machine we are going to create for this tutorial.

Steps:

  1. Create 4 virtual machines in Azure:

2.  Under Basics, Project Details, Subscription, enter a new resource group  or select and existing resource group for this virtual machine:

We’re going to use the Standard_D2s_v3 instance to deploy for this tutorial. Depending on your use case, you might want to deploy DS3_v2 or D8s_v3 because they have more vCPUs, more RAM and a bigger IOPS values as shown in Azure Portal when displaying the most used machines (image below), this will allow you to save objects faster and latency will be reduced; having a better throughput for MinIO and other applications. You can use the Azure Calculator to estimate the price you will pay based on the time your virtual machine will be used.

3. Then allow administrator access over SSH connection via port 22:

Click Next: Disks to advance.

4. Select the Storage option, where you can configure the type of disks. Choose Premium SSD for performance-sensitive workloads, Standard SSD for web servers and other applications, or Standard HDD for backup or non-critical access. Then, attach 4 disks to each machine.

Click the “Next: Networking >” button to advance.

5. Add your virtual machine to a new or existing virtual network and IP subnet. Note that all virtual machines in your MinIO cluster must use the same subnet so that they can communicate between each other:

6. If you want, you can add tags to your virtual machines to help you identify and work with them. Click on Tags and enter the name and value as applicable.

7. Finally, click Review + create. Review your settings. Then click Create.

8. Generate and download a new private key.

Sit back and watch deployment.

9. You will now have 4 virtual machines to use when deploying MinIO in distributed mode.

10.  Once our VMs are ready, we can start with the MinIO deployment. To use SSH to connect to your VMs, first add your them to your local PEM file (Windows instructions, Mac/Linux instructions) ):

ssh -i <your-pem-file>  azureuser@<the-ip-address>

11.  Append to each virtual machine the hosts to /etc/hosts

<IP Address> host1

<IP Address> host2

… And so on and so forth

In the example PEM file below, the four bottom lines are the lines that you need to append to each hos. You can get the IP Address from the ifconfig command:

127.0.0.1 localhost

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

172.16.0.4 host1
172.16.0.5 host2
172.16.0.6 host3
172.16.0.7 host4

12.  Once connected to your VMs, you will need to mount the disks, format the disks, create directories, download MinIO and execute it. You must do this for each endpoint.

You can use script below to format the disks:

#!/bin/bash
#set -x

mount_device(){
DEVICE="/dev/$1"
VOLUME_NAME=$2
MOUNT_POINT=$3

format_device() {
  echo "Formatting $DEVICE"
  mkfs.xfs -imaxpct=25 -f -L $MOUNT_POINT $DEVICE
}
check_device() {
  if [ -f "/etc/systemd/system/$MOUNT_POINT.mount" ]; then
    echo "Device $MOUNT_POINT ($DEVICE) exists"
    echo "No actions required..."
  else
    echo "$MOUNT_POINT.mount was not found, creating volume"
    format_device
  fi
}
check_mount() {
  if [ -f "/etc/systemd/system/$MOUNT_POINT.mount" ]; then
    echo "Found $MOUNT_POINT.mount in /etc/systemd/system/"
    echo "No actions required..."
  else
    echo "$MOUNT_POINT.mount was not found in /etc/systemd/system/ adding it"
    mkdir -p /$MOUNT_POINT
   
    echo "[Unit]" >> /etc/systemd/system/$MOUNT_POINT.mount
    echo "Description=Mount System Backups Directory" >> /etc/systemd/system/$MOUNT_POINT.mount
    echo "" >> /etc/systemd/system/$MOUNT_POINT.mount
    echo "[Mount]" >> /etc/systemd/system/$MOUNT_POINT.mount
    echo "What=LABEL=$MOUNT_POINT" >> /etc/systemd/system/$MOUNT_POINT.mount
    echo "Where=/$MOUNT_POINT" >> /etc/systemd/system/$MOUNT_POINT.mount
    echo "Type=xfs" >> /etc/systemd/system/$MOUNT_POINT.mount
    echo "Options=noatime" >> /etc/systemd/system/$MOUNT_POINT.mount
    echo "" >> /etc/systemd/system/$MOUNT_POINT.mount
    echo "[Install]" >> /etc/systemd/system/$MOUNT_POINT.mount
    echo "WantedBy=multi-user.target" >> /etc/systemd/system/$MOUNT_POINT.mount

    systemctl enable $MOUNT_POINT.mount
    systemctl start $MOUNT_POINT.mount
  fi
}
LABEL=$(blkid -L $VOLUME_NAME)
check_device
check_mount
systemctl daemon-reload
}

for i in `lsblk -d | grep -v NAME | grep -v nvme0 | awk '{print $1}'`; do
  mnt_point=`echo $i | sed -e 's/nvme/disk/g' -e 's/n1//g'`
  mount_device $i $i $mnt_point;
done

Once the drives are mounted and formatted, you can create data directories and install MinIO. When complete, you should see 16 drives online, recall that we have 4 disks per VM and 4 VMs, so 4x4=16.

azureuser@cesar-celis-2:~$ mkdir data1
azureuser@cesar-celis-2:~$ mkdir data2
azureuser@cesar-celis-2:~$ mkdir data3
azureuser@cesar-celis-2:~$ mkdir data4

azureuser@cesar-celis-2:~$ wget https://dl.min.io/server/minio/release/linux-amd64/minio

azureuser@cesar-celis-2:~$ chmod +x minio

azureuser@cesar-celis-2:~$ MINIO_ROOT_USER=admin MINIO_ROOT_PASSWORD=password MINIO_CI_CD=1 sudo ./minio server http://host{1...4}/home/azureuser/data{1...4} --console-address ":9001"

All MinIO sub-systems initialized successfully

Status:         16 Online, 0 Offline. 

API: http://10.5.0.5:9000  http://127.0.0.1:9000     

RootUser: admin 

RootPass: password 


Console: http://10.5.0.5:9001 http://127.0.0.1:9001   

RootUser: admin 

RootPass: password 


Command-line: https://docs.min.io/docs/minio-client-quickstart-guide

   $ mc alias set myminio http://10.5.0.5:9000 admin password


Documentation: https://docs.min.io

13.  Create an extra connection to run mc, the MinIO client. MC includes a rich set of S3-compatible commands that can be used to manage and troubleshoot your MinIO deployment.

Let’s measure the speed of MinIO with our new feature, SpeedTest. This will run a quick, automated series of drive speed measurements on all drive on all nodes. First we’ll create an alias to make it easier to manage our MinIO deployment..

azureuser@cesar-celis:~$ wget https://dl.min.io/client/mc/release/linux-amd64/mc

--2022-03-09 23:54:30--  https://dl.min.io/client/mc/release/linux-amd64/mc

Resolving dl.min.io (dl.min.io)... 138.68.11.125, 178.128.69.202

Connecting to dl.min.io (dl.min.io)|138.68.11.125|:443... connected.

HTTP request sent, awaiting response... 200 OK

Length: 23318528 (22M) [application/octet-stream]

Saving to: ‘mc’


mc                                             100%[==================================================================================================>]  22.24M  22.8MB/s    in 1.0s    


2022-03-09 23:54:32 (22.8 MB/s) - ‘mc’ saved [23318528/23318528]


azureuser@cesar-celis:~$ chmod +x mc 

azureuser@cesar-celis:~$ 

azureuser@cesar-celis:~$ 


azureuser@cesar-celis:~$ ./mc alias set myminio http://172.16.0.7:9000 minioadmin minioadmin

mc: Configuration written to `/home/azureuser/.mc/config.json`. Please update your access credentials.

mc: Successfully created `/home/azureuser/.mc/share`.

mc: Initialized share uploads `/home/azureuser/.mc/share/uploads.json` file.

mc: Initialized share downloads `/home/azureuser/.mc/share/downloads.json` file.

Added `myminio` successfully.

azureuser@cesar-celis:~$ ./mc support perf drive myminio/

┌────────────┬───────────────────────┬──────────┬──────────┐

│ Node       │ Path                  │ Read     │ Write    │

│ host2:9000 │ /home/azureuser/data1 │ 53 MiB/s │ 50 MiB/s │

│ host2:9000 │ /home/azureuser/data2 │ 50 MiB/s │ 48 MiB/s │

│ host2:9000 │ /home/azureuser/data3 │ 50 MiB/s │ 49 MiB/s │

│ host2:9000 │ /home/azureuser/data4 │ 49 MiB/s │ 50 MiB/s │

└────────────┴───────────────────────┴──────────┴──────────┘

┌────────────┬───────────────────────┬──────────┬──────────┐

│ Node       │ Path                  │ Read     │ Write    │

│ host3:9000 │ /home/azureuser/data1 │ 50 MiB/s │ 50 MiB/s │

│ host3:9000 │ /home/azureuser/data2 │ 50 MiB/s │ 48 MiB/s │

│ host3:9000 │ /home/azureuser/data3 │ 49 MiB/s │ 49 MiB/s │

│ host3:9000 │ /home/azureuser/data4 │ 48 MiB/s │ 50 MiB/s │

└────────────┴───────────────────────┴──────────┴──────────┘

┌────────────┬───────────────────────┬──────────┬──────────┐

│ Node       │ Path                  │ Read     │ Write    │

│ host4:9000 │ /home/azureuser/data1 │ 50 MiB/s │ 49 MiB/s │

│ host4:9000 │ /home/azureuser/data2 │ 50 MiB/s │ 49 MiB/s │

│ host4:9000 │ /home/azureuser/data3 │ 48 MiB/s │ 50 MiB/s │

│ host4:9000 │ /home/azureuser/data4 │ 48 MiB/s │ 50 MiB/s │

└────────────┴───────────────────────┴──────────┴──────────┘

┌────────────┬───────────────────────┬──────────┬──────────┐

│ Node       │ Path                  │ Read     │ Write    │

│ host1:9000 │ /home/azureuser/data1 │ 50 MiB/s │ 48 MiB/s │

│ host1:9000 │ /home/azureuser/data2 │ 50 MiB/s │ 50 MiB/s │

│ host1:9000 │ /home/azureuser/data3 │ 50 MiB/s │ 50 MiB/s │

│ host1:9000 │ /home/azureuser/data4 │ 49 MiB/s │ 50 MiB/s │

└────────────┴───────────────────────┴──────────┴──────────┘

Deploy MinIO on Azure Today

MinIO is now installed and you have verified performance on Azure virtual machines. We used Premium SSD, but you can use any storage medium or tier across storage media. Now you can use your preferred interface to create buckets and further configure your deployment.

If you have any questions, ping us on hello@min.io or join our Slack community.