I’ve seen a bunch of posts lately about how to set up Docker swarm monitoring with tools like cAdvisor and node_exporter which advise running them like this:
docker service create --mode global -p 9100:9100 ...
That will indeed run one container on each swarm node, but it has a subtle problem. When you connect to host:9100, the ingress network routing will connect you to a random instance each connection (for swarm routing values of random). You will indeed get some metrics returned, but they will be for whichever host you happened to be routed to this time.
You can demonstrate this problem quite simply with something like:
docker service create --name hello --mode global -p 8080:80 dockercloud/hello-world
Then just reload the page and you’ll see that the hostname (container ID) changes sometimes. You might see it more readily with curl than a browser.
A simple solution is to just run these containers on each node as normal docker containers outside swarm management. In the setup I’m working on just now, we used the same Puppet automation that provisions the swarm to start the containers.
docker run -d -p 9100:9100 ...