Skip to content

Webhooks setup

Utilising webhook by Adnan Hajdarević and inspired by blogs by Anson Vandoren.

The components consist of: the webhook binary, a hooks file, a trigger script and a system service to keep the binary running in the background.

Webhook install & configuration

The hooks file tells webhook how to handle incoming requests, and which script it should run if it receives an incoming request that matches the pre-arranged criteria, including a shared 'secret'. The scripts can include items such as Telegram notifications.

Install the webhook binary onto server:

Install webhook

sudo apt install webhook

Create a webhooks directory on the system and create a JSON file for the hooks:

Setup hooks

sudo mkdir /opt/webhook && sudo nano /opt/webhook/hooks.json

Set relevant trigger rules (such as the branch being pushed to). A v4 UUID is used as a 'secret' and can be generated at https://www.uuidgenerator.net/

/opt/webhook/hooks.json 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
[
    {
        "id": "redeploy",
        "execute-command": "/opt/webhook/triggerscript.sh",
        "command-working-directory": "/opt/webhook",
        "pass-arguments-to-command":
        [
            {
                "source": "payload",
                "name": "head_commit.message"
            },
            {
                "source": "payload",
                "name": "pusher.name"
            },
            {
                "source": "payload",
                "name": "head_commit.id"
            }
        ],
        "trigger-rule":
        {
            "and":
            [
                {
                    "match":
                    {
                        "type": "payload-hash-sha1",
                        "secret": "<insert_UUID_here>",
                        "parameter":
                        {
                            "source": "header",
                            "name": "X-Hub-Signature"
                        }
                    }
                },
                {
                    "match":
                    {
                        "type": "value",
                        "value": "refs/heads/main",
                        "parameter":
                        {
                            "source": "payload",
                            "name": "ref"
                        }
                    }
                }
            ]
        }
    }
]

Now create a script and make it executable once saved:

Install webhook

sudo nano /opt/webhook/triggerscript.sh && sudo chmod +x /opt/webhook/triggerscript.sh
/opt/webhook/triggerscript.sh 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
#!/bin/bash -e
# Note the '-e' in the line above. This is required for error trapping implemented below.
# Original author Annson Van Doren https://ansonvandoren.com/posts/telegram-notification-on-deploy/
# Adapted by AJR July 2022

# Repo name on GitHub - **BE SURE TO USE SSH RATHER THAN HTTPS**
REMOTE_REPO=git@github.com:EDIflyer/linux-notes.git
# A place to clone the remote repo so the static site generator can build from it; can't use $HOME as runs as root
WORKING_DIRECTORY=/home/<user>/repositories/linux-notes
# Location (server block) where the Nginx container looks for content to serve
PUBLIC_WWW=/var/www/docs.alanjrobertson.co.uk/html
# Backup folder in case something goes wrong during this script
BACKUP_WWW=/home/<user>/docs/backup_html
# Domain name so Hugo can generate links correctly
MY_DOMAIN=docs.alanjrobertson.co.uk

# Set up Telegram
TOKEN=INSERT_TOKEN_HERE
CHAT_ID=INSERT_CHAT_ID_HERE
BOT_URL="https://api.telegram.org/bot$TOKEN/sendMessage"

# Send messages to Telegram bot
function send_msg () {
    # Use "$1" to get the first argument (desired message) passed to this function
    # Set parsing mode to HTML because Markdown tags don't play nice in a bash script
    # Redirect curl output to /dev/null since we don't need to see it
    # (it just replays the message from the bot API)
    # Redirect stderr to stdout so we can still see an error message in curl if it occurs
    curl -s -X POST $BOT_URL -d chat_id=$CHAT_ID -d text="$1" -d parse_mode="HTML" > /dev/null 2>&1
}

# These parameters are passed by the webhook to the script - see the hooks.json `pass-arguments-to-command` section
commit_message=$1
pusher_name=$2
commit_id=$3

# If something goes wrong, put the previous verison back in place
function cleanup {
    ERROR=$?
    echo "A problem occurred. Reverting to backup."
    rsync -aqz --del $BACKUP_WWW/ $PUBLIC_WWW
    rm -rf $WORKING_DIRECTORY

    # Use $? to get the error message that caused the failure
    send_msg "<b>Deployment of $MY_DOMAIN failed:</b> $ERROR"
}

# Call the cleanup function if this script exits abnormally. The -e flag
# in the shebang line ensures an immediate abnormal exit on any error
trap cleanup EXIT

# Clear out the working directory
rm -rf $WORKING_DIRECTORY
# Make a backup copy of current website version
# --mkpath flag causes destination directories to be created
rsync -avz --mkpath $PUBLIC_WWW/ $BACKUP_WWW

# Clone the new version from GitHub
git clone $REMOTE_REPO $WORKING_DIRECTORY

send_msg "<i>Successfully cloned Github repo for $MY_DOMAIN</i>
<code>Message: $commit_message</code>
<code>Commit ID: $commit_id</code>
<code>Pushed by: $pusher_name</code>"

# Delete old version
rm -rf $PUBLIC_WWW/*
# Have mkdocs-material generate the new static HTML directly into the public WWW folder
# Save the output to send to Telegram
mkdocs_response=$(docker run --rm -i -v $WORKING_DIRECTORY:/docs custom/mkdocs-material build)
cp -r $WORKING_DIRECTORY/site/* $PUBLIC_WWW
# Send response to bot as a fenced code block to preserve formatting
send_msg "<pre>$mkdocs_response</pre>"

# All done!
send_msg "<b>Deployment successful!</b>"

# Clear out working directory
rm -rf $WORKING_DIRECTORY
# Exit without trapping, since everything went well
trap - EXIT

Now create the folder from named in triggerscript.sh for Github downloads:

Create download directory

mkdir ~/repositories/<REPOSITORYNAME>

Webhook service creation

Create a service:

Create download directory

sudo nano /opt/webhook/webhooks.service
/etc/systemd/system/webhooks.service 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# NOTES
# -----
# Install to systemd folder:
# sudo cp webhooks.service /etc/systemd/system/webhooks.service
# sudo systemctl daemon-reload
#
# Can then use:
# sudo systemctl enable webhooks --now
# sudo systemctl status webhooks
# sudo systemctl stop webhooks
#
[Unit]
Description=Webhook receiver service
ConditionPathExists=/usr/bin/webhook
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/webhook -hooks /opt/webhook/hooks.json -verbose -hotreload -port 9001
Restart=on-failure

[Install]
WantedBy=default.target

Copy across, enable and start the webhook service then check status:

Enable & start webhooks.service

sudo cp /opt/webhook/webhooks.service /etc/systemd/system/webhooks.service && \
sudo systemctl daemon-reload && \
sudo systemctl enable webhooks --now && \
sudo systemctl status webhooks

Nginx Proxy Manager and firewall setup

Create a new webhook site in NPM:

  • Domain Names: webhook.(servername).(tld) (e.g., webhook.alanjrobertson.co.uk)
  • Scheme: http
  • Forward Hostname/IP: 172.17.0.1 (this is the Docker host, ie the server itself)
  • Forward Port: 9001
  • Block Common Exploits: True
  • Request a new SSL certificate with Force SSL, HTTP/2 Support, HSTS Enabled, HSTS Subdomains all checked

Reconfigure firewall

ufw must be reconfigured to allow communication from Docker to the host, otherwise the incoming webhook requests will just time out:

Enable & start webhook.service

sudo ufw allow from 172.19.0.0/16 to any && \
sudo ufw reload && \
sudo ufw status

Github setup

Go to the Settings > Webhooks page on Github for the relevant repository and click 'Add webhook'

  • Payload URL: https://webhook.(servername).(tld)/hooks/redeploy (e.g., https://webhook.alanjrobertson.co.uk/hooks/redeploy)
  • Content type: application/json
  • Secret: the UUID previously generated and placed in the webhooks.json file
  • Enable SSL verification: True
  • Which events would you like to trigger this webhook?: Just the push event
  • Active: True

You should receive confirmation of a successful ping and a HTTP/200 response. Note there is a tab called Recent Deliveries in the Github webhook management screen that shows the status of recent webhook messages and lets them be resent.

Install git locally and connect to GitHub

We now need to install git on the server and also use an SSH key to connect to our GitHub account.

Setup SSH key on server and copy to GitHub account

cd ~
ssh-keygen -t ed25519 -C "<EMAIL ADDRESS>"
Enter github_sync when prompted for filename to save the key. This will then create a private key called github_sync and a public key called github_sync.pub
Then copy private key to the .ssh sub-directory in home directory: 
cp github_sync ~/.ssh/github_sync
We then need to add the public key to our GitHub account. The easiest way is to run cat github_sync.pub and copy the output to the clipboard the paste into the Settings > Access > SSH and GPG keys section of GitHub.

Warning

Remember to backup the Github public/private keypair that has just been created

Install git and set up syncing

sudo apt install git -y
mkdir repositories
Now edit the config file in the .ssh directory: 
nano ~/.ssh/config
and add these lines: 
Host github.com
IdentityFile ~/.ssh/github_sync
Run git configuration 
git config --global user.name "<username>"
git config --global user.email "<email>"
Now test the connection: 
ssh -T git@github.com
Once the GitHub pulic key fingerprint is accepted there should be a confirmation message of Hi username! You've successfully authenticated, but GitHub does not provide shell access.

Root vs standard user

Remember the triggerscript will be run as root user, therefore in addition to the above you need to copy these credentials across to the root user otherwise you will get an authentication error from Github when trying to pull down the repository. 

sudo cp ~/.ssh/config /root/.ssh && \
sudo cp ~/.ssh/github_sync /root/.ssh

Final triggerscript setup

A few pre-requisites are required for the triggerscript:

Install rsync and setup backup directory

sudo apt install rsync -y && \
mkdir -p ~/docs/backup_html

You can check webhook status as above or the full journal:

View logs for webhooks.service

sudo journalctl -u webhooks

A push to the repository should now cause Github to send a webhook payload and trigger the script to pull down the repository and rebuild the site.