Rotating Docker Logs — Keeping your overlay folder small
August 14, 2018
Learn from my mistakes. If you are using the default json-file driver, configure
max-size options to have Docker automatically rotate and remove old logs.
4:30 pm. Time to go home, figure something out for dinner, relax and — — THE BUILDS ARE FAILING. Classic.
Can’t push or pull to our self-hosted registry. The container is running, so why would it not be able to read/write to the disk unless…the disk is full?
After some quick digging, turns out the Docker overlay folder is big. Like almost 200 GB big. But a quick glance shows that we only have 4 images and a few small volumes, so why would that overlay folder be so big?
The answer: logs. a LOT of logs.
By default, Docker uses the json-file log driver and saves those logs in the overlay folder. Docker doesn’t limit the size of the files or how many log files there can be for a single container. If you continue to run that container, the log file will grow as long as it has drive space to do so. Thankfully, it’s pretty easy to set these limits.
If you are running from the CLI, your option looks like:
docker run --log-opt max-size=10m --log-opt max-file=5 my-app:latest
This sets a maximum of five log files with a max size of 10 Mb each. So at most 50 Mb of logs for that container. Tune those numbers as you see fit.
If you are using docker-compose, your options should look like:
my-app: image: my-app:latest logging: driver: "json-file" options: max-file: 5 max-size: 10m
You can also configure this as a default in your daemon settings https://docs.docker.com/config/containers/logging/configure/#configure-the-logging-driver-for-a-container
Written by James Quigley, a DevOps Engineer, full-stack dev, and general tech nerd. Follow him on Twitter!