A number of 3rd party clients are available for transfer of data into and from Object Storage, what makes s5cmd useful is its speed and clear syntax. This article will provide the basics for getting started with the s5cmd tool.
Pre-requisites
- s5cmd available from the Github project - https://github.com/peak/s5cmd
- the aws cli must already be installed
available from Amazon at https://aws.amazon.com/cli/ - a Zadara Object Instance
- a user account in the Object Instance, with relevant privileges to the storage (the users S3 Access key and secret will be used in the AWS profile to authorise access to the instance)
Overview
The s5cmd like the similar s3cmd requires that you have configured an aws named profile or profiles ( if using multiple accounts or object stores), this article does not cover profile configuration.
You must have an accessible Zadara Object instance and the URL details for the instance.
A suitable user account with any container/bucket name and privileges to create/remove folder and data objects is required.
Usage
The s5cmd tool itself has it's own help option to get you started.
$ s5cmd --help
Example commands
list bucket & sub-folder content;
s5cmd --profile myprofile \
--endpoint-url https://vsa-00000123-public-my-cloud-01.zadarazios.com \
ls s3://mybucket/folder01/
s5cmd --profile myprofile \
--endpoint-url https://vsa-00000123-public-my-cloud-01.zadarazios.com
mb s3://mybucket
s5cmd --profile myprofile \
--endpoint-url https://vsa-00000123-public-my-cloud-01.zadarazios.com \
cp myfile.txt s3://mybucket/folder01/
s5cmd --profile myprofile \
--endpoint-url https://vsa-00000123-public-my-cloud-01.zadarazios.com \
cp *.txt s3://mybucket/folder02/
s5cmd --profile myprofile \
--endpoint-url https://vsa-00000123-public-my-cloud-01.zadarazios.com \
run backup_files.cmd
Where the file "backup_files.cmd" contains;
mb s3://mybackup
cp *.bak s3://mybackup/
Concurrency / Parallel jobs by setting a workers count;
s5cmd --profile myprofile \
--endpoint-url https://vsa-00000123-public-my-cloud-01.zadarazios.com \
--numworkers 15 \
cp *2023.rpt s3://mybucket/reports-2023/
The above shows a brief overview of what this versatile command tool can offer more information is available online at https://github.com/peak/s5cmd