Setup nofile for Docker

Problem

Yesterday I met a weird problem. The mosquitto container failed to start due to the error below: The error message indicates that process failed to allocate memory for creating 1073741816 file descriptors.

1567622954: mosquitto version 1.6.4 starting
1567622954: Config loaded from /mosquitto/config/mosquitto.conf.
1567622954: Opening websockets listen socket on port 9001.
1567622954: Error: Unable to create websockets listener on port 9001.
1567623383: mosquitto version 1.6.4 starting
1567623383: Config loaded from /mosquitto/config/mosquitto.conf.
1567623383: Opening websockets listen socket on port 9001.
1567623383: libuv support not compiled in
1567623383: OOM allocating 1073741816 fds
1567623383: ZERO RANDOM FD
1567623383: Error: Unable to create websockets listener on port 9001.

Then I traced into the mosquitto dependency called libwebsocket which raised this error. Telling from its source code, the number 1073741816 actually is the max nofile of the container. But the host systems max nofile is less than the fd value:

1 2	cat /proc/sys/fs/file-nr 992 0 1589038

Theoretically, each container uses host’s max nofile by default. How come does the big value come from?

Solution

Check the max nofile value got in container. It means that the default max nofile set in container is 1073741816.
1
2
sudo docker run --rm debian sh -c "ulimit -n"
1073741816

Investigate where the value comes from. There are a couple of places to set it up.

vim /usr/lib/systemd/system/docker.service

 [Service]
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity

vim /etc/docker/daemon.json, the settings here will ovewrite the setting in docker daemon

 {
        "default-ulimits": {
                "nofile": {
                        "Name": "nofile",
                        "Hard": 20000,
                        "Soft": 20000
                }
        }
}

Set in docker run command like this: The configuration in command line will overwrite all the settings before.
1
docker run --ulimit nofile=1024:1024 --rm debian sh -c "ulimit -n"

The root cause is that the max nofile has been set in file /usr/lib/systemd/system/docker.service as infinity. In order to overwrite the daemon config, I set the nofile in docker-comopse.yml file. And then the error was gone.
1
2
3
4
ulimits:
nofile:
soft: 200000
hard: 200000

Problem

Solution

Reference