Swarm Stack local-proxy-prom-elk

Before using these make sure you had clone the repository by:

git submodule update --init --recursive

That example of local deployment is nearly the same to the previews one except we will add the reverse proxy Traefik to permit to us to have many more service even if they need the same port! | Even if in surface that look like same, the deployment file are all modified! | Now let’s go..

(*) All Open Sources

Services	Software
GUI Control	Portainer
Central Monitoring	Promotheus + Grafana
Central Logging	Elastic ELK
Layer 7 Proxy	Traefik
Storage	Local File System
Networking	Docker Swarm Overlay
Orchestration	Docker Swarm
Runtime	Docker CE
Machine and OS	Docker Machine + VirtualBox

You have 2 way to deploy it:

- The fast way by launch the ansible workbook, just type:

    ansible-playbook -i ansible/swarm/local-inventory ansible/swarm/local-proxy-prom-elk.yml
    # NOTE: that will take around 5 minutes

- Or the long way but where you can learn every step ...

Deployment step by step

1/ Create the Machine:

./utils2devops/bin/docker-machine-cluster.sh -c 5

You can go to see the doc of this tools here Create your local cluster of docker-machine

Here we will create a swarm with 3 machines

2/ Enable monitoring (optional):

./utils2devops/bin/enable-monitoring.sh -p ./utils2devops/docker/ -n 5

3/ Create the Docker Swarm:

./utils2devops/bin/swarm.sh -c -m 3 -w 2

You can go to see the doc of this tools here Add local nodes in docker-machine (DEPRECATED in version >= 0.2)

Here we will create a swarm with 3 manager and 2 worker

4/ To launch docker command in the Master with ssh it:

eval "$(docker-machine env node-1)"

5/ Deploy Traefik:

export TRAEFIK_HOST=traefik.yourdomain
default value: traefik.example.com
export TRAEFIK_PUBLIC_TAG=my-traefik-public
default value: traefik-public
docker stack deploy -c ./utils2devops/docker/local-with-prom-elk/traefik.yml traefik

After this step we will have a proxy Dashboard at:

http://traefik.example.com:8080/dashboard/

7/ Deploy Ops Stacks Graphics UI (optional):

export PORTAINER_HOST=portainer.yourdomain
default value: portainer.example.com
docker stack deploy -c ./utils2devops/docker/local-with-prom-elk/portainer.yml portainer

After these steps we will have:

Portainer at:
    http://portainer.example.com/#/init/admin
    http://portainer.example.com/#/dashboard
    http://portainer.example.com/#/containers
    http://portainer.example.com/#/swarm/visualizer
and so many other... have a look here https://www.portainer.io/overview/

8/ Deploy Ops Stacks:

export ADMIN_USER=admin
default value: admin
export ADMIN_PASSWORD=adminadmin
default value: adminadmin
export HASHED_PASSWORD=$(openssl passwd -apr1 -salt pepper $ADMIN_PASSWORD)

You can check the contents with:

echo $HASHED_PASSWORD

it will look like:

$apr1$TsqS2JR3$oGG0NFZsU1VdKn03MAyjh.

Create and export an environment variable DOMAIN, e.g.::

export DOMAIN=example.com

and make sure that the following sub-domains point to your Docker Swarm cluster IPs:

grafana.example.com
alertmanager.example.com
unsee.example.com
prometheus.example.com

Note: You can also use a subdomain, like swarmprom.example.com. Just make sure that the subdomains point to (at least one of) your cluster IPs. Or set up a wildcard subdomain (*).

Set and export an environment variable with the tag used by Traefik public to filter services (by default, it’s traefik-public):

export TRAEFIK_PUBLIC_TAG=traefik-public

If you are using Slack and want to integrate it, set the following environment variables:

export SLACK_URL=https://hooks.slack.com/services/TOKEN
default value: https://hooks.slack.com/services/TOKEN
export SLACK_CHANNEL=utils2devops-tests
default value: general
export SLACK_USER=alertmanager
default value: alertmanager

Then we continue to deploy with swarmprom:

docker stack deploy -c ./utils2devops/docker/local-with-prom-elk/swarmprom.yml prom

After these steps we will have

Grafana login at:
    https://grafana.example.com/login
Grafana Swarm nodes at:
    https://grafana.example.com/d/BPlb-Sgik/docker-swarm-nodes?refresh=30s&orgId=1
Grafana Swarm Services at:
    https://grafana.example.com/d/zr_baSRmk/docker-swarm-services?refresh=30s&orgId=1
Promotheus Stat at:
    http://grafana.example.com/d/mGFfYSRiz/prometheus-2-0-stats?refresh=1m&orgId=1
Promotheus Query at::
    https://prometheus.example.com/graph
Alert manager at:
    https://alertmanager.example.com/#/alerts
Alert Dashboard at:
    https://unsee.example.com/?q=

In promotheus try:

sum(irate(container_cpu_usage_seconds_total{image!=""}[1m])) without (cpu)
container_memory_usage_bytes{image!=""}
sum(rate(container_network_transmit_bytes_total{image!=""}[1m])) without (interface)
sum(rate(container_fs_reads_bytes_total{image!=""}[1m])) without (device)
sum(rate(container_fs_writes_bytes_total{image!=""}[1m])) without (device)

Then we finish to deploy with elk:

export ELASTICSEARCH_USER=admin
default value: admin
export ELASTICSEARCH_PASSWORD=adminadmin
default value: admin
export ELASTICSEARCH_HASHED_PASSWORD=$(openssl passwd -apr1 -salt pepper $ELASTICSEARCH_PASSWORD)
export KIBANA_USER=admin
default value: admin
export KIBANA_PASSWORD=adminadmin
default value: admin
export KIBANA_HASHED_PASSWORD=$(openssl passwd -apr1 $KIBANA_PASSWORD)
docker stack deploy -c ./utils2devops/docker/local-with-prom-elk/elk.yml elk

After these steps we will have:

Elasticsearch at:
    http://elasticsearch.example.com/
kibana at:
    http://kibana.example.com/app/kibana#/home?_g=()
and much more have a look at https://github.com/stefanprodan/swarmprom

Note: | To use elasticsearch you will have to increase the max virtual memory or you will get: | WARN: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144] | It can be fixed by running on the node: | sysctl -w vm.max_map_count=262144 in terminal (Linux/Ubuntu). | Or permanently by create a file name 60-elasticsearch.conf and place it in /etc/sysctl.d/ with the following content: | vm.max_map_count=262144

Now it’s ready to deploy your apps and test them:

docker stack deploy my_company/my_services my_service

When you have finish to use it, Destroy it by:

./utils2devops/bin/docker-machine-cluster.sh -d 5