docker-stacks/docs/using/troubleshooting.md

Troubleshooting Common Problems

When troubleshooting, you may see unexpected behaviors or receive an error message. This section provides links for identifying the cause of the problem and how to resolve it.

Permission Issues

Permission denied when mounting volumes

If you are running a Docker container while mounting a local volume or host directory using the -v flag like so:

docker run -it --rm \
    -p 8888:8888 \
    -v <my-vol>:<container-dir> \
    jupyter/minimal-notebook:latest

you might face permissions issues when trying to access the mounted volume:

# assuming we mounted the volume in /home/jovyan/stagingarea
$ ls -ld ~/stagingarea/
drwxr-xr-x 2 root root 4096 Feb  1 12:55 stagingarea/

$ touch stagingarea/kale.txt
touch: cannot touch 'stagingarea/kale.txt': Permission denied

Some things to try:

1. Change ownership of the volume mount

   You can change the ownership of the volume mount using the chown command. For example, to change the ownership of the volume mount to the jovyan user:

   docker run -d \
       -v <my-vol>:<container-dir> \
       -p 8888:8888 \
       --user root \
       -e CHOWN_EXTRA="<container-dir>" \
       -e CHOWN_EXTRA_OPTS="-R" \
       jupyter/minimal-notebook
   

   where:

   • CHOWN_EXTRA=<some-dir>: will change the owner and group of the specified container directory (non recursive by default). You need to provide full paths starting with /.
   • CHOWN_EXTRA_OPTS=R: will recursively the owner and group of of the directory specified in CHOWN_EXTRA.
   • --user root: must run the container with the root user to perform the change of ownership.

   so now accessing the mount should work as expected:

   # assuming we mounted the volume in /home/jovyan/stagingarea
   $ ls -ld ~/stagingarea
   drwxr-xr-x 2 jovyan users 4096 Feb  1 12:55 stagingarea/
   
   $ touch stagingarea/kale.txt
   $ ls -la ~/stagingarea/
   -rw-r--r-- 1 jovyan users    0 Feb  1 14:41 kale.txt
   

   Note: If you are mounting your volume inside the /home/ directory you can use the -e CHOWN_HOME=yes and CHOWN_HOME_OPTS="-R" flags instead of the -e CHOWN_EXTRA and -e CHOWN_EXTRA_OPTS in the example above.

2. Matching the container's UID/GID with the host's

   Docker handles mounting host directories differently to mounting volumes, even though the syntax is essentially the same (i.e. -v).

   When you initialize a Docker container using the flag -v the host directories are bind mounted directly into the container. Therefore, the permissions and ownership are copied over and will be exactly the same as the ones in your local host (including user ids) which may result in permissions errors like the one displayed above.

   Suppose your local user has a UID and GID of 1234. To fix the UID discrepancies between your local directories and the container's directoriess, you need to set NB_UID and NB_GID to match the that of the local user:

   $ docker run -it --rm \
       --user root \
       -p 8888:8888 \
       -e NB_UID=1234 \
       -e NB_GID=1234 \
       -v $(PWD)/test:/home/jovyan/work \
       jupyter/minimal-notebook:latest
   
   # you should see an output similar to this
   Update jovyan's UID:GID to 1234:1234
   Running as jovyan: bash
   

   Some important things to notice:

   • You must use --user root to ensure that the UID is updated at runtime

   • The caveat with this approach is that since these changes are applied at runtime, you will need to to re-run the same command with the appropriate flags and environment variables if you need to recreate the container

     Note that this approach only updates the UID and GID of the existing jovyan user:

   $ id
   uid=1234(jovyan) gid=1234(jovyan) groups=1234(jovyan),100(users)
   

If on top of changing the UID and GID you also need to create a new user (and thus ensure this user has a home directory) you can use the following command:

 docker run -it --rm \
 --user root \
 -e NB_UID=1234 \
 -e NB_GID=1234 \
 -e NB_USER=trallard \
 -e CHOWN_HOME=yes \
 -e CHOWN_HOME_OPTS="-R" \
 -w "/home/${NB_USER}" \
 -v $(PWD)/test:/home/trallard/work \
  jupyter/minimal-notebook start.sh

 # expected output
 Updated the jovyan user:
 - username: jovyan       -> trallard
 - home dir: /home/jovyan -> /home/trallard
 Update trallard's UID:GID to 1234:1234
 Attempting to copy /home/jovyan to /home/trallard...
 Success!
 Ensuring /home/trallard is owned by 1234:1234
 Running as trallard: bash

Additional tips and troubleshooting commands:

• Use absolute paths when using the -v flag:

  -v $(PWD)/<my-vol>:/home/jovyan/work
  

  in this example, we use the syntax $(PWD) which will be replaced with the full path to the current directory. The destination path should also be an absolute path starting with a / such as home/jovyan/work.

• You might want to consider using the Docker native --user <UID> and --group-add users flags instead of -e NB_GID and -e NB_UID:

  # note this will use the same UID from
  # the user calling the command thus matching the local host
  
  docker run -it --rm \
      --user "$(id -u)" --group-add users \ # $(id -u) prints the UID of the user that called docker run
      -v <my-vol>:/home/jovyan/work jupyter/datascience-notebook
  

  this command will not only launch the container with a specific user UID, but also add the that user to the users group so that it can modify the files in the default /home and /opt/conda directories. Further avoiding issues when trying to conda install additional packages.

• Use docker inspect <container_id> and look for the Mounts section to verify that the volume was created and mounted accordingly:

  # for example for a my-vol volume created with docker volume create <my-vol>
  
  "Mounts": [
        {
            "Type": "volume",
            "Name": "my-vol",
            "Source": "/var/lib/docker/volumes/stagingarea/_data",
            "Destination": "/home/jovyan/stagingarea",
            "Driver": "local",
            "Mode": "z",
            "RW": true,
            "Propagation": ""
        }
    ],