We wrote the HCP-to-MinIO tool to help our customers migrate data from HCP object storage to MinIO. The tool can be freely downloaded from its GitHub repo.
We developed this tool to make life easier for the many customers who came to us as they transitioned away from their HCP object storage environments. The reasons are varied. Most often, it is to move to a platform that is software-defined as well as cloud-and-Kubernetes-native. These are requirements for the modern application stack that our customers use. The second most common reason is to improve performance-at-scale for mixed object sizes, both very small and very large. What our customers tell us echoes the larger trend in the market – dedicated storage appliances are being put out to pasture en masse as enterprises embrace software-defined S3-compatible object storage for greater flexibility and cloud neutrality.
During the migration process, we recommend not writing to HCP and writing only to MinIO. Customers usually start treating MinIO as the primary object store immediately. While objects are being migrated to MinIO,
getobject calls are first made to MinIO, and if the object does not exist in MinIO, then the object is read from the HCP object store. Once all the objects are copied to MinIO, this fall back option can be removed from the code. While copying objects, the tool maintains the modified timestamp of the objects as it was in HCP.
The tool is first used to make a list of the objects stored in HCP object storage. Then, the list is used to download the objects from HCP object storage and copied over to MinIO. The tool has two commands (
migrate) to accomplish this.
list command produces a list of objects stored in HCP and writes it to a file on a local drive. The list file is then split into multiple smaller list files and the HCP-to-MinIO tool is then executed from multiple client machines. This way multiple nodes are used to download the objects from HCP and migrate the data to MinIO, allowing customers to saturate the network and migrate as much data as possible in the shortest amount of time.
Once the migration is done, applications will use S3-compatible APIs to talk to MinIO. Data is now available for cloud-native analytics and AI/ML frameworks.
Usage for the
migrate commands are given below. Download the HCP-to-MinIO tool to your workstation, then run
list, followed by
list: List objects in HCP namespace and download to disk
NAME: hcp-to-minio list - List objects in HCP namespace and download to disk USAGE: hcp-to-minio list --auth-token --namespace-url --host-header --dir FLAGS: --auth-token value, -a value authorization token for HCP --namespace-url value, -n value namespace URL path, e.g https://namespace-name.tenant-name.hcp-domain-name/rest --host-header value host header for HCP --data-dir value, -d value path to work directory for tool --insecure, -i disable TLS certificate verification --log, -l enable logging --debug enable debugging --help, -h show help EXAMPLES: 1. List objects in HCP namespace https://hcp-vip.example.com and download list to /tmp/data $ hcp-to-minio list --a "HCP bXl1c2Vy:3f3c6784e97531774380db177774ac8d" --host-header "HOST:s3testbucket.tenant.hcp.example.com" \ --namespace-url "https://hcp-vip.example.com" --dir "/tmp/data"
migrate - Migrate objects from HCP object store to MinIO
NAME: hcp-to-minio migrate - Migrate objects from HCP object store to MinIO USAGE: hcp-to-minio migrate --auth-token --namespace-url --host-header --data-dir FLAGS: --auth-token value, -a value authorization token for HCP --namespace-url value, -n value namespace URL path, e.g https://namespace-name.tenant-name.hcp-domain-name/rest --host-header value host header for HCP --data-dir value, -d value path to work directory for tool --bucket bucket name on HCP --input-file file that contains the list of objects to be migrated from HCP to MinIO --insecure, -i disable TLS certificate verification --log, -l enable logging --debug enable debugging --help, -h show help
You can set the following configuration prior to beginning migration:
$ export MINIO_ACCESS_KEY=<Your-MinIO-Access-Key> $ export MINIO_SECRET_KEY=<Your-MinIO-Secret-Key> $ export MINIO_ENDPOINT=https://<Your-MinIO-IP-Address>:9000 $ export MINIO_BUCKET=newbucket # optional, if unspecified HCP bucket name is used
Create the temporary directory to house the list:
$ mkdir /tmp/data # temporary dir where output of listing is stored.
$ hcp-to-minio migrate --namespace-url https://finance.europe.hcp.example.com --auth-token "HCP bXl1c2Vy:3f3c6784e97531774380db177774ac8d" --host-header "s3testbucket.sandbox.hcp.example.com" --data-dir /mnt/data --bucket s3testbucket --input-file /tmp/data/to-migrate.txt
Migrate to MinIO Today
Upgrading your infrastructure and application stack can be challenging. We created the HCP-to-MinIO migration tool to make it easier for you and your organization to embrace the technologies that are driving the cloud forward – the S3 API and Kubernetes. It’s time to break free from on-premise storage appliances and experience the fastest and most-scalable object storage on the planet.