More MinIO Data Options - The MinIO FTP/SFTP Server

It is strange to think about how long the File Transfer Protocol (FTP) has been around. First developed in 1971 by Abhay Bhushan, a student at MIT, it was initially used to transfer files between computers on the ARPANET. Needless to say it evolved - with authentication, encryption, and compression features. It remains a widely used protocol for transferring files over the Internet, particularly in cases where large files or large numbers of files need to be transferred quickly and reliably.

While the S3 API is fundamentally superior, it lacks the bare bones simplicity of FTP/SFTP and the ability to run in highly constrained environments. As a result, MinIO has added support for FTP and SFTP into the MinIO Server. With this new functionality, the user (or application) thinks they are talking to an FTP server when in fact they are talking to MinIO. The advantages should be apparent - it becomes seamless to move data onto MinIO and from the application/user perspective everything is essentially the same - from policies, security etc.

Currently supported FTP/SFTP operations are as follows:

MinIO supports following FTP/SFTP based protocols to access and manage data.

• Secure File Transfer Protocol (SFTP) – Defined by the Internet Engineering Task Force (IETF) as an extended version of SSH 2.0, allowing file transfer over SSH and for use with Transport Layer Security (TLS) and VPN applications.
• File Transfer Protocol over SSL/TLS (FTPS) – Encrypted FTP communication via TLS certificates.
• File Transfer Protocol (FTP) – Defined by RFC114 originally, and replaced by RFC765 and RFC959 unencrypted FTP communication
  • This is NOT recommended

What is covered in this release:

• All IAM Credentials are allowed access excluding rotating credentials, rotating credentials are not allowed to login via FTP/SFTP ports. You must use S3 API port for if you are using rotating credentials.
• Access to bucket(s) and object(s) are governed via IAM policies associated with the incoming login credentials.
• Allows authentication and access for all
  • Built-in IDP users and their respective service accounts
  • LDAP/AD users and their respective service accounts
  • OpenID/OIDC service accounts
• On versioned buckets, FTP/SFTP only operates on the latest objects. If you need to retrieve an older version you must use an S3 API client such as mc.
• All features currently used by your buckets will work “as is” without any changes
  -SSE (Server Side Encryption)
  -Replication (Server Side Replication)

Essentially, this should be a seamless protocol addition from the developer/administrator’s perspective.

Prerequisites and Notes

This functionality is available beginning with RELEASE.2023-04-20T17-56-55Z.

It is assumed users have already been created and configured with relevant access policies. MinIO recommends starting with a basic "readwrite" canned policy to test all the operations before finalizing what level of restrictions are needed for a user.

There are no "admin:*" operations needed for FTP/SFTP access to the bucket(s) and object(s). They may be skipped for restrictions.

Important note: FTP/SFTP is not enabled by default. It needs to be enabled on setup. Further, there is no admin functionality available via FTP, this must be done through the command line.

Usage

Start MinIO in a distributed setup, with FTP/SFTP enabled.

minio server http://server{1...4}/disk{1...4}
--ftp="address=:8021" --ftp="passive-port-range=30000-40000" \
--sftp="address=:8022" --sftp="ssh-private-key=/home/miniouser/.ssh/id_rsa"
...
...

The following example shows connecting via ftp client using minioadmin credentials, and list a bucket named runner:

ftp localhost -P 8021
Connected to localhost.
220 Welcome to MinIO FTP Server
Name (localhost:user): minioadmin
331 User name ok, password required
Password:
230 Password ok, continue
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> ls runner/
229 Entering Extended Passive Mode (|||39155|)
150 Opening ASCII mode data connection for file list
drwxrwxrwx 1 nobody nobody            0 Jan  1 00:00 chunkdocs/
drwxrwxrwx 1 nobody nobody            0 Jan  1 00:00 testdir/
...

The following example shows how to list an object and download it locally via ftp client:

ftp> ls runner/chunkdocs/metadata
229 Entering Extended Passive Mode (|||44269|)
150 Opening ASCII mode data connection for file list
-rwxrwxrwx 1 nobody nobody           45 Apr  1 06:13 chunkdocs/metadata
226 Closing data connection, sent 75 bytes
ftp> get
(remote-file) runner/chunkdocs/metadata
(local-file) test
local: test remote: runner/chunkdocs/metadata
229 Entering Extended Passive Mode (|||37785|)
150 Data transfer starting 45 bytes
45        3.58 KiB/s
226 Closing data connection, sent 45 bytes
45 bytes received in 00:00 (3.55 KiB/s)
...

Notes on SFTP

If there are certificates enabled on the server, SFTP can be used. Provided the FTP client is capable, TLS can be used to secure the transmission. This will require an additional flag for the private key.

The following example shows connecting via sftp client using minioadmin credentials, and list a bucket named runner:

sftp -P 8022 minioadmin@localhost
minioadmin@localhost's password:
Connected to localhost.
sftp> ls runner/
chunkdocs  testdir

The following example shows how to download an object locally via sftp client:

sftp> get runner/chunkdocs/metadata metadata
Fetching /runner/chunkdocs/metadata to metadata
metadata     
100%  226    16.6KB/s   00:00
sftp>

Advanced options

Change default FTP port

Default port 8021 can be changed via

--ftp="address=:3021"

Change FTP passive port range

By default FTP requests that the OS provide a free port automatically, however you may want to restrict this to specific ports in certain restricted environments via

--ftp="passive-port-range=30000-40000"

Change default SFTP port

The default port 8022 can be changed via

--sftp="address=:3022"

TLS (FTP)

Unlike SFTP server, FTP server is insecure by default. To operate under TLS mode, you need to provide certificates via

--ftp="tls-private-key=path/to/private.key" --ftp="tls-public-cert=path/to/public.crt"

NOTE: if MinIO distributed setup is already configured to run under TLS, FTP will automatically use the relevant certificates from the server certificate chain, this is mainly to add simplicity of setup. However if you wish to terminate TLS certificates via a different domain for your FTP servers you may choose the above command line options.

Limitations

We noted earlier that Append and Rename are not support functions.

Given the simplicity of FTP - versioning is also problematic. Versioning is not supported by FTP and the FTP clients don’t know what it actually is - but buckets will generally have multiple versions of an object. As a result, the same object with a new name will be a new object. The same object with the same name will be a new version - but FTP has no concept of that, it needs to be managed via S3. It can move the data, but it doesn’t have the sophistication of S3 to understand certain concepts.

Finally, as noted, there are no admin functions available over FTP.

Summary

FTP/SFTP are exceptionally long lived protocols and they continue to find new use cases and utility well into their 50s. We have made it simple to use FTP to get data into MinIO and to get it out.

More importantly, FTP/SFTP functionality will work seamlessly with your existing MinIO deployment - just upgrade and go feeding you MinIO buckets with all the data you had to previously transform into MinIO.

We have a great video on the subject here. We have our GitHub page here. We are available to support the community here and obviously, if you are a commercial customer you can find us 24/7/365 on SUBNET.