Deploy docker-compose using Drone CI

Here I will show the simplest and fastest way to automate the deployments for web app with any complexity level. We will use a secure SSH connection between the Drone CI host and the Application host. In short, the way is next:

We will create a docker-compose description that would deliver resources and apps on any machine.
Order VPS (or dedicated) host with public IP, ssh access, and open application ports.
Tweak SSH settings on the VPS to allow multisession connections for docker deploys.
Create bash script which creates ssh private/public key pair inside of deployer-container, then exports DOCKER_HOST=ssh://[email protected]<host IP> and runs docker-compose up.

All instructions are explained below.

We used Ubuntu 18.04 OS for my Amazon EC2 t3a.small instance (you can use any hosting). Less than 2 GB might cause memory and hanging up issues.

You need to have the next TCP ports open:

22 for SSH
all ports that your application are listening on (e.g. 80, 443)

If you use Amazon EC2, all ports are closed by default, but you can open ports in Security Group configuration. To do it on instance launch dialog:

EC2 Security Group step

Many VPS/Dedicated hosting providers have all ports open by default.

Login by keypair

ssh -i ~/xx.pem ubuntu@xxxxx

Edit file: sudo nano /etc/ssh/sshd_config

Add lines to the end (These settings are required to properly work with docker builds):

PermitRootLogin yes
MaxSessions 500
MaxStartups 500

Run service sshd restart to apply changes✅.

Add your local public ssh key 🔑 content (cat ~/.ssh/id_rsa.pub, use ssh-keygen on your localhost, if you have no such file) to the next file on the remote host (add to beginning with one line break after):

sudo vim /root/.ssh/authorized_keys

By doing this you allow direct connection using your ssh key with a command from your OS user, like this:

ssh [email protected]<APP_HOST_IP>

Prepare docker host

Now on remote host Install docker:

sudo apt update && sudo apt remove docker docker-engine docker.io && sudo apt install docker.io

Prepare repository:

As an example, we will show the config for the Django project.

`/.drone.yml`

kind: pipeline
name: default
type: docker
concurrency:
  limit: 1
steps:
- name: slack-begin
  image: plugins/slack
  settings:
    webhook:
      from_secret: slack_webhook
    username: Drone
    template: >
      {{repo.name}}/{{build.branch}} - Started #{{build.number}} "${DRONE_COMMIT_MESSAGE}" by {{build.author}} (<{{build.link}}|Open>)

- name: build
  environment:
    VAULT_MASTER_SSH_PRIV_KEY:
      from_secret: VAULT_MASTER_SSH_PRIV_KEY
    VAULT_MASTER_SSH_PUB_KEY:
      from_secret: VAULT_MASTER_SSH_PUB_KEY

    VAULT_MASTER_PORTAINER_PASSHASH:
      from_secret: VAULT_MASTER_PORTAINER_PASSHASH

    VAULT_MASTER_DB_PASS:
      from_secret: VAULT_MASTER_DB_PASS
    VAULT_MASTER_SECRET_KEY:
      from_secret: VAULT_MASTER_SECRET_KEY
      
  image: devforth/drone-builder
  commands:
  - cd deploy && /bin/bash build.sh

- name: slack-end
  image: plugins/slack
  settings:
    webhook:
      from_secret: slack_webhook
    username: Drone
    template: >
      {{repo.name}}/{{build.branch}} - #{{build.number}} {{uppercasefirst build.status}} after {{since build.started}} (<{{build.link}}|Open>)

This drone file uses 5 Drone Secrets. VAULT_MASTER_SSH_PRIV_KEY and VAULT_MASTER_SSH_PUB_KEY should be copied from your PC to Drone Vault.

VAULT_MASTER_PORTAINER_PASSHASH - pass for portainer (docker admin panel), you can remove it from config if you don't need portainer.

👆TO GENERATE for VAULT_MASTER_PORTAINER_PASSHASH use next command (on any Linux/Mac with docker or WSL2 on windows:

docker run --rm httpd:2.4-alpine htpasswd -nbB admin 'yourPortainerPpass' | cut -d ":" -f 2

Other secrets are defined to improve application security (better to store all values in drone vault)

`/deploy/build.sh`

#!/bin/bash

branch=$(git branch | grep \* | cut -d ' ' -f2)
branch=`echo $branch | sed 's/feature\///g'`
echo "Building branch $branch"

if [ -z "$VAULT_MASTER_SSH_PRIV_KEY" ]; then
    echo "Running not on Drone (localy)"
else
    mkdir -p ~/.ssh/
    echo "$VAULT_MASTER_SSH_PRIV_KEY" > ~/.ssh/id_rsa
    echo "$VAULT_MASTER_SSH_PUB_KEY" > ~/.ssh/id_rsa.pub
    chmod 600 ~/.ssh/id_rsa
    chmod 644 ~/.ssh/id_rsa.pub
    eval `ssh-agent -s`
    ssh-add ~/.ssh/id_rsa
    echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
fi

    
if [ "$branch" = "master" ]; then
  HOST_DOMAIN=<APP_HOST_IP>
  # add more environemnts for branches here, e.g.:
  # elif [ "$branch" = "develop" ]; then
else
  echo "No configuration for branch $branch"
  exit -1
fi

export DOCKER_HOST=ssh://[email protected]$HOST_DOMAIN   

# cleanup cache from previous builds
docker builder prune -a -f
docker container prune -f
docker rmi $(docker images -f "dangling=true" -q) || true

# build and run db image (migration will connect to db)
docker-compose -p stack-$branch -f docker-compose.yml -f assets/variables_$branch.yml up -d --build db

# build django app contanier
docker-compose -p stack-migrate-$branch -f docker-compose.yml -f assets/variables_$branch.yml build --compress django
docker-compose -p stack-migrate-$branch -f docker-compose.yml -f assets/variables_$branch.yml run --no-deps --workdir="/code/" django bash -c 'pipenv run python manage.py collectstatic --noinput'
docker-compose -p stack-migrate-$branch -f docker-compose.yml -f assets/variables_$branch.yml run --no-deps --workdir="/code/" django pipenv run python manage.py migrate

docker-compose -p stack-$branch -f docker-compose.yml -f assets/variables_$branch.yml up -d --build --remove-orphans 
docker-compose -p stack-$branch restart nginx

stack-migrate-$branch is used to reduce downtime during redeploys (migrating and collecting static actually performed without old container stopping).

/deploy/docker-compose.yml

version: '3.5'

services:
  db:
    network_mode: host
    image: mysql:5.7
    restart: always
    command: --default-authentication-plugin=mysql_native_password --character-set-server=utf8mb4 --collation-server=utf8mb4_bin
    volumes:
      - v-db-data:/var/lib/mysql
  
  django:
    network_mode: host
    build: ../src/
    command: /bin/bash -c "cd /code/ && pipenv run gunicorn --worker-tmp-dir /dev/shm conf.wsgi:application --bind 0.0.0.0:8001 --workers 5 --worker-connections=1000"
    # working_dir: /root/
    restart: always
    volumes:
      - v-django:/code/serve_static

  nginx:
    network_mode: host
    build: nginx
    restart: always
    volumes:     
      - type: volume
        source: v-django
        target: /static/django
        read_only: true
        volume:
          nocopy: true
    
    command: /bin/bash -c "envsubst '$$CERT_NAME $$DOMAIN $$BASIC_AUTH_ENABLE' < /etc/nginx/conf.d/conf.template > /etc/nginx/conf.d/default.conf && nginx -g 'daemon off;'"

  portainer:
    image: portainer/portainer:1.24.1-alpine
    network_mode: host
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - v-portainer-data:/data

volumes:
  v-portainer-data:
    name: v-portainer-data
  v-db-data:
    name: v-db-data
  v-django:
    name: v-django

network_mode: host is recommended for better networking performance (no docker virtual DNS names resolution)

`/deploy/assets/variables_master.yml`

This file is used to configure variables per one environment.

version: '3.5'

services:
  db:
    environment:
      - MYSQL_ROOT_PASSWORD=${VAULT_MASTER_DB_PASS}
      - MYSQL_DATABASE=db_live
  
  nginx:   
    environment: 
      # - BASIC_AUTH_ENABLE=1
      - DOMAIN=yur_site.com
      - CERT_NAME=yur_site_cert

  django:
    environment:
      - ENV_NAME=live
      - DEBUG=False
      - SECRET_KEY=${VAULT_MASTER_SECRET_KEY}
      - DOMAIN=hinty.io
      - DATABASE_URL=mysql://root:${VAULT_MASTER_DB_PASS}@127.0.0.1:3306/db_live?charset=utf8mb4

  portainer:
    command: --admin-password=${VAULT_MASTER_PORTAINER_PASSHASH}