Archive:Zabbix

Zabbix is used to monitor our infrastructure of external and internal servers, websites and things.

Installation

We are using the docker images from monitoringartist.

For a persistent database we are using a separate docker volume.

docker volume create --name zabbix-db-storage

The storage is available on the host in

sudo ls -al /var/lib/docker/volumes/zabbix-db-storage/_data

We are using a dedicated bridge network (in opposite to the default one). Docker provides an internal DNS for custom bridge networks which makes it easier to use hostnames between the docker containers.

docker network create --driver bridge zabbix_nw

Now let's start the database instance:

docker run \
    -d \
    --name zabbix-db \
    -v /tmp:/backups \
    -v /etc/localtime:/etc/localtime:ro \
    -v zabbix-db-storage:/var/lib/mysql \
    --network=zabbix_nw \
    --env="MARIADB_USER=zabbix" \
    --env="MARIADB_PASS=<somepw>" \
    --env="DB_innodb_buffer_pool_size=512M" \
    monitoringartist/zabbix-db-mariadb

And finally we can start Zabbix itself:

docker run \
    -d \
    --name zabbix \
    -p 8080:80 \
    -p 10051:10051 \
    -v /etc/localtime:/etc/localtime:ro \
    -v /srv/zabbix-scripts/alertscripts/:/usr/local/share/zabbix/alertscripts/ \
    -v /srv/zabbix-scripts/externalscripts/:/usr/local/share/zabbix/externalscripts/ \
    --env="VIRTUAL_HOST=zabbix.intern.munichmakerlab.de,zefix.intern.munichmakerlab.de" \
    --env="ZS_Timeout=10" \
    --env="ZS_DBHost=zabbix-db" \
    --env="ZS_DBUser=zabbix" \
    --env="ZS_DBPassword=<somepw>" \
    --env="XXL_zapix=true" \
    --env="XXL_grapher=true" \
    --env="XXL_apiuser=Admin" \
    --env="XXL_apipass=zabbix" \
    --env="PHP_date_timezone=Europe/Berlin" \
    monitoringartist/zabbix-xxl:latest

Important: We have to attach this docker container to both networks (the default and the custom one). The reason we are running it in both is, that the nginx-proxy (we are using it as there are couple of web-services running on the docker host) can access the default one only).

/usr/bin/docker network connect zabbix_nw zabbix

SSH tunnel to external servers

We do not want to expose the Zabbix agent on our external servers. Therefore we use an ssh tunnel. We created a custom docker container, which starts an autossh. Run this container per external server:

git clone https://github.com/siedi/zabbix-autossh.git

Generate the keys (how to deploy them on the servers, see the readme in the git repo):

ssh-keygen -t rsa -b 4096 -f ./id_rsa -C "zabbixagent"

Build the docker image, which includes the new pub key

docker build -t siedi/zabbix-autossh .

And run it for our two servers:

docker run -d --network=zabbix_nw --name jupiter-tunnel -t -i siedi/zabbix-autossh zabbixagent@jupiter.munichmakerlab.de
docker run -d --network=zabbix_nw --name mars-tunnel -t -i siedi/zabbix-autossh zabbixagent@mars.munichmakerlab.de

In Zabbix you can connect to the agents on these servers by using the dns name "jupiter-tunnel" / "mars-tunnel" due to the docker dns auto-magic for custom networks.

Backups

Backup of DB Zabbix - configuration data only, no item history/trends

docker exec \
    -ti zabbix-db \
    /zabbix-backup/zabbix-mariadb-dump -u zabbix -p <somepw> -o /backups

Full backup of Zabbix DB

docker exec \
    -ti zabbix-db \
    bash -c "\
    mysqldump -u zabbix -p<somepw> zabbix | \
    bzip2 -cq9 > /backups/zabbix_db_dump_$(date +%Y-%m-%d-%H.%M.%S).sql.bz2"

Files are in the /tmp folder of the docker host.

Restore Zabbix DB:

Remove Zabbix server container (stopp it before):

docker rm -f zabbix

Restore data from dump (all current data will be dropped!!!), backup files needs to be located in the /tmp folder of the docker host.

docker exec -i zabbix-db sh -c 'bunzip2 -dc /backups/zabbix_db_dump_*.sql.bz2 | mysql -uzabbix -p --password=<somepw> zabbix'

Start Zabbix container again.

Zabbix agents

To monitor the "standard" facts of a server, install the Zabbix agent and get some basic measures out of the box.

On debian we use the jessie backport for at least the 3.0 version of the agent.

Add the backport repo to the sources.list and install the agent

sudo sh -c 'echo "deb http://ftp.de.debian.org/debian/ jessie-backports main" >> /etc/apt/sources.list'
sudo aptitude update
sudo aptitude -t jessie-backports install zabbix-agent

Add the Zabbix server to the list of allowed servers (this is sometimes tricky to find out with the docker NATting):

sudo vi /etc/zabbix/zabbix_agentd.conf

Server=127.0.0.1,10.10.20.66

Docker commands

Access the zabbix container to inspect files:

docker exec -ti zabbix /bin/bash

See who os connected to a network:

docker network inspect zabbix_nw

systemd files

As a reference, here are the systemd service files (as of 22/11/2016):

$ cat docker-zabbix.service
[Unit]
Description=Zabbix
After=docker-zabbix-db.service
Requires=docker-zabbix-db.service

[Service]
TimeoutStartSec=0
Restart=always
ExecStartPre=/usr/bin/docker pull monitoringartist/zabbix-xxl:latest
ExecStartPre=-/usr/bin/docker kill zabbix
ExecStartPre=-/usr/bin/docker rm zabbix
ExecStart=/usr/bin/docker run -e VIRTUAL_HOST=zabbix.intern.munichmakerlab.de,zefix.intern.munichmakerlab.de --name zabbix -p 10051:10051 -v /etc/localtime:/etc/localtime:ro -v /srv/zabbix-scripts/externalscripts/:/usr/local/share/zabbix/externalscripts/ -v /srv/zabbix-scripts/alertscripts/:/usr/local/share/zabbix/alertscripts/ -e ZS_Timeout=10 -e ZS_DBHost=zabbix-db -e ZS_DBUser=zabbix -e ZS_DBPassword=<somepw> -e XXL_zapix=true -e XXL_grapher=true -e XXL_apiuser=Admin -e XXL_apipass=zabbix -e PHP_date_timezone=Europe/Berlin monitoringartist/zabbix-xxl:latest
ExecStartPost=-/bin/sleep 10
ExecStartPost=-/usr/bin/docker network connect zabbix_nw zabbix
ExecStop=/usr/bin/docker stop -t 20 zabbix
#ExecStopPost=/usr/bin/docker rm zabbix

[Install]
WantedBy=multi-user.target


$ cat docker-zabbix-db.service
[Unit]
Description=Zabbix DB
After=docker.service
Requires=docker.service

[Service]
TimeoutStartSec=0
Restart=always
ExecStartPre=/usr/bin/docker pull monitoringartist/zabbix-db-mariadb
ExecStartPre=-/usr/bin/docker kill zabbix-db
ExecStartPre=-/usr/bin/docker rm zabbix-db
ExecStart=/usr/bin/docker run --network=zabbix_nw --name zabbix-db -v /tmp:/backups -v /etc/localtime:/etc/localtime:ro -v zabbix-db-storage:/var/lib/mysql --env MARIADB_USER=zabbix --env MARIADB_PASS=<somepw> --env DB_innodb_buffer_pool_size=512M monitoringartist/zabbix-db-mariadb
ExecStop=/usr/bin/docker stop -t 20 zabbix-db
#ExecStopPost=/usr/bin/docker rm zabbix-db

[Install]
WantedBy=multi-user.target



$ cat docker-jupiter-tunnel.service
[Unit]
Description=Zabbix Autossh to Jupiter
After=docker.service
Requires=docker.service

[Service]
TimeoutStartSec=0
Restart=always
#ExecStartPre=/usr/bin/docker pull siedi/zabbix-autossh
ExecStartPre=-/usr/bin/docker kill jupiter-tunnel
ExecStartPre=-/usr/bin/docker rm jupiter-tunnel
ExecStart=/usr/bin/docker run --network=zabbix_nw --name jupiter-tunnel siedi/zabbix-autossh zabbixagent@jupiter.munichmakerlab.de
ExecStop=/usr/bin/docker stop -t 2 jupiter-tunnel
#ExecStopPost=/usr/bin/docker rm jupiter-tunnel

[Install]
WantedBy=multi-user.target