swarm – A geek's thoughts

I have been looking far and wide for a solution for docker swarm persistent volume shared among multiple hosts and nothing seemed to be easily deployable, reasonnably fast and reliable.

Glusterfs is slow as hell on every hardware I’ve tried to run it, Ceph seems very complicated to set up for a simple swarm setup.

Now I’ve begun experimenting with Rexray S3FS docker plugin and it seems quite interesting.

The only issue I’ve had is that I’m using Scaleway for hosting and I could not find a way to make it work with their S3 implementation.

I was missing a crucial piece of the puzzle: you need to specify the S3FS_REGION option, which is not mentionned in the scaleway docs, so the command line looks like this:

docker plugin install --grant-all-permissions rexray/s3fs:latest \
S3FS_ACCESSKEY=XXXXXXXXXX\
S3FS_SECRETKEY=YYYYYYYYYYYYY \
S3FS_ENDPOINT=https://s3.fr-par.scw.cloud \
S3FS_REGION=fr-par \
S3FS_OPTIONS="allow_other,nonempty,use_path_request_style,url=https://s3.fr-par.scw.cloud"

Also, it seems that the plugin tries to delete existing buckets, so I’ve created a separate account for docker S3 volumes, just to be safe.

I’ve been trying to get Homeassistant working on swarm for a few months now, but the thing that was preventing me from moving to swarm was the Homeassistant requirement to use host networking on the container.
I had tried many things but I finally got everything to work with the macvlan driver. I configured the homeassistant service with two networks:

One macvlan network, giving it an IP address in my iot network
One overlay network, giving traefik access to homeassistant

The macvlan network is configured as follows. I had to create a local macvlan network on each member of the swarm, with a subnet swarm is free to choose an IP from:

1st node:
docker network create –config-only –subnet=192.168.2.0/24 –gateway=192.168.2.1 -o parent=eth0 –ip-range 192.168.2.232/29 macvlan_local
2nd node:
docker network create –config-only –subnet=192.168.2.0/24 –gateway=192.168.2.1 -o parent=eth0 –ip-range 192.168.2.240/29 macvlan_local
3rd node:
docker network create –config-only –subnet=192.168.2.0/24 –gateway=192.168.2.1 -o parent=eth0 –ip-range 192.168.2.248/29 macvlan_local

Then I create a swarm scoped network:

docker network create -d macvlan –scope swarm –config-from macvlan_local macvlan_swarm

Then I can deploy the whole stack, here is the docker-compose file:
(the wait-for-it config is a script that prevents homeassistant from starting before mqtt, it’s not necessary but quite useful)

version: '3.7'

configs:
  wait-for-it:
    file: /srv/docker/homeassistant/wait-for-it/wait-for-it.sh

services:
  homeassistant:
    image: homeassistant/armhf-homeassistant:0.86.4
    configs:
      - source: wait-for-it
        target: /wait-for-it.sh
        mode: 755
    networks:
      - webgateway
      - macvlan_swarm
    command: ["/wait-for-it.sh", "rasper:1883", "--", "python", "-m", "homeassistant", "--config", "/config"]
    volumes:
      - /srv/docker/homeassistant/config:/config
      - /etc/localtime:/etc/localtime:ro
    deploy:
      labels:
        - "traefik.backend=hassio"
        - "traefik.frontend.rule=Host:myhomeassistanthostname"
        - "traefik.port=8123"
        - "traefik.enable=true"
        - "traefik.docker.network=webgateway"
        - "traefik.default.protocol=http"

  mosquitto:
    image: eclipse-mosquitto
    volumes:
      - /srv/docker/mosquitto/config:/mosquitto/config
      - /etc/localtime:/etc/localtime:ro
      - /srv/docker/mosquitto/data/mosquitto-data:/mosquitto/data
      - /srv/docker/mosquitto/data/mosquitto-log:/mosquitto/log
    ports:
      - "1883:1883"
      - "9001:9001"

  nodered:
    image: nodered/node-red-docker:rpi-v8
    deploy:
      labels:
        - "traefik.backend=nodered"
        - "traefik.frontend.rule=Host:mynoderedhostname"
        - "traefik.port=1880"
        - "traefik.enable=true"
        - "traefik.docker.network=webgateway"
        - "traefik.default.protocol=http"
    networks:
      - webgateway
    volumes:
      - /srv/docker/nodered:/data
    environment:
      - TZ:Europe/Brussels


volumes:
  mosquitto-data:
  mosquitto-log:

networks:
  webgateway:
    external: true
  macvlan_swarm:
    external: true
  hostnet:
    external: true
    name: host