Archive:Zabbix

Zabbix is used to monitor our infrastructure of external and internal servers, websites and things.

Installation

We are using the docker images from monitoringartist.

For a persistent database we are using a separate docker volume.

docker volume create --name zabbix-db-storage

The storage is available on the host in

sudo ls -al /var/lib/docker/volumes/zabbix-db-storage/_data

We are using a dedicated bridge network (in opposite to the default one). Docker provides an internal DNS for custom bridge networks which makes it easier to use hostnames between the docker containers.

docker network create --driver bridge zabbix_nw

Now let's start the database instance:

docker run \
    -d \
    --name zabbix-db \
    -v /tmp:/backups \
    -v /etc/localtime:/etc/localtime:ro \
    -v zabbix-db-storage:/var/lib/mysql \
    --network=zabbix_nw \
    --env="MARIADB_USER=zabbix" \
    --env="MARIADB_PASS=test123" \
    --env="DB_innodb_buffer_pool_size=512M" \
    monitoringartist/zabbix-db-mariadb

And finally we can start Zabbix itself:

docker run \
    -d \
    --name zabbix \
    -p 8080:80 \
    -p 10051:10051 \
    -v /etc/localtime:/etc/localtime:ro \
    -v /srv/zabbix-scripts/alertscripts/:/usr/local/share/zabbix/alertscripts/ \
    -v /srv/zabbix-scripts/externalscripts/:/usr/local/share/zabbix/externalscripts/ \
    --env="VIRTUAL_HOST=zabbix.intern.munichmakerlab.de,zefix.intern.munichmakerlab.de" \
    --env="ZS_Timeout=10" \
    --env="ZS_DBHost=zabbix-db" \
    --env="ZS_DBUser=zabbix" \
    --env="ZS_DBPassword=test123" \
    --env="XXL_zapix=true" \
    --env="XXL_grapher=true" \
    --env="XXL_apiuser=Admin" \
    --env="XXL_apipass=zabbix" \
    --env="PHP_date_timezone=Europe/Berlin" \
    monitoringartist/zabbix-xxl:latest

Important: We have to attach this docker container to both networks (the default and the custom one). The reason we are running it in both is, that the nginx-proxy (we are using it as there are couple of web-services running on the docker host) can access the default one only).

/usr/bin/docker network connect zabbix_nw zabbix

SSH tunnel to external servers

We do not want to expose the Zabbix agent on our external servers. Therefore we use an ssh tunnel. We created a custom docker container, which starts an autossh. Run this container per external server:

git clone https://github.com/siedi/zabbix-autossh.git

Generate the keys (how to deploy them on the servers, see the readme in the git repo):

ssh-keygen -t rsa -b 4096 -f ./id_rsa -C "zabbixagent"

Build the docker image, which includes the new pub key

docker build -t siedi/zabbix-autossh .

And run it for our two servers:

docker run -d --network=zabbix_nw --name jupiter-tunnel -t -i siedi/zabbix-autossh zabbixagent@jupiter.munichmakerlab.de
docker run -d --network=zabbix_nw --name mars-tunnel -t -i siedi/zabbix-autossh zabbixagent@mars.munichmakerlab.de

In Zabbix you can connect to the agents on these servers by using the dns name "jupiter-tunnel" / "mars-tunnel" due to the docker dns auto-magic for custom networks.

Backups

Backup of DB Zabbix - configuration data only, no item history/trends

docker exec \
    -ti zabbix-db \
    /zabbix-backup/zabbix-mariadb-dump -u zabbix -p test123 -o /backups

Full backup of Zabbix DB

docker exec \
    -ti zabbix-db \
    bash -c "\
    mysqldump -u zabbix -ptest123 zabbix | \
    bzip2 -cq9 > /backups/zabbix_db_dump_$(date +%Y-%m-%d-%H.%M.%S).sql.bz2"

Files are in the /tmp folder of the docker host.

Restore Zabbix DB:

Remove Zabbix server container (stopp it before):

docker rm -f zabbix

Restore data from dump (all current data will be dropped!!!), backup files needs to be located in the /tmp folder of the docker host.

docker exec -i zabbix-db sh -c 'bunzip2 -dc /backups/zabbix_db_dump_*.sql.bz2 | mysql -uzabbix -p --password=test123 zabbix'

Start Zabbix container again.

Zabbix agents

To monitor the "standard" facts of a server, install the Zabbix agent and get some basic measures out of the box.

On debian we use the jessie backport for at least the 3.0 version of the agent.

Add the backport repo to the sources.list and install the agent

sudo sh -c 'echo "deb http://ftp.de.debian.org/debian/ jessie-backports main" >> /etc/apt/sources.list'
sudo aptitude update
sudo aptitude -t jessie-backports install zabbix-agent

Add the Zabbix server to the list of allowed servers (this is sometimes tricky to find out with the docker NATting):

sudo vi /etc/zabbix/zabbix_agentd.conf

Server=127.0.0.1,10.10.20.66

Docker commands

Access the zabbix container to inspect files:

docker exec -ti zabbix /bin/bash

See who os connected to a network:

docker network inspect zabbix_nw