I have been looking far and wide for a solution for docker swarm persistent volume shared among multiple hosts and nothing seemed to be easily deployable, reasonnably fast and reliable.
Glusterfs is slow as hell on every hardware I’ve tried to run it, Ceph seems very complicated to set up for a simple swarm setup.
Now I’ve begun experimenting with Rexray S3FS docker plugin and it seems quite interesting.
The only issue I’ve had is that I’m using Scaleway for hosting and I could not find a way to make it work with their S3 implementation.
I was missing a crucial piece of the puzzle: you need to specify the S3FS_REGION option, which is not mentionned in the scaleway docs, so the command line looks like this:
docker plugin install --grant-all-permissions rexray/s3fs:latest \
S3FS_ACCESSKEY=XXXXXXXXXX\
S3FS_SECRETKEY=YYYYYYYYYYYYY \
S3FS_ENDPOINT=https://s3.fr-par.scw.cloud \
S3FS_REGION=fr-par \
S3FS_OPTIONS="allow_other,nonempty,use_path_request_style,url=https://s3.fr-par.scw.cloud"
Also, it seems that the plugin tries to delete existing buckets, so I’ve created a separate account for docker S3 volumes, just to be safe.