docker-stacks/docs/using/troubleshooting.md

Troubleshooting Common Problems

When troubleshooting, you may see unexpected behaviors or receive an error message. This section provides advice on how to identify and mitigate the cause of the problem and how to resolve it (for the most commlonly encountered issues).

Most of the docker run flags used in this document are explained in detail in the Common Features, Docker Options section of the documentation.

Permission denied when mounting volumes

If you are running a Docker container while mounting a local volume or host directory using the -v flag like so:

docker run -it --rm \
    -p 8888:8888 \
    -v <my-vol>:<container-dir> \
    jupyter/minimal-notebook:latest

you might face permissions issues when trying to access the mounted volume:

# assuming we mounted the volume in /home/jovyan/stagingarea
# root is the owner of the mounted volume
$ ls -ld ~/stagingarea/
drwxr-xr-x 2 root root 4096 Feb  1 12:55 stagingarea/

$ touch stagingarea/kale.txt
touch: cannot touch 'stagingarea/kale.txt': Permission denied

In this case the user of the container (jovyan) and the owner of the mounted volume (root) have different permission levels and ownership over the container's directories and mounts.

Some things to try:

1. Change ownership of the volume mount

   You can change the ownership of the volume mount using the chown command. In the case of the docker-stacks images, you can set the CHOWN_EXTRA and CHOWN_EXTRA_OPTS environment variables.

   For example, to change the ownership of the volume mount to the jovyan user (non-privileged default user in the Docker images):

   # running in detached mode - can also be run in interactive mode
   docker run -d \
       -v <my-vol>:<container-dir> \
       -p 8888:8888 \
       --user root \
       -e CHOWN_EXTRA="<container-dir>" \
       -e CHOWN_EXTRA_OPTS="-R" \
       jupyter/minimal-notebook
   

   where:

   • CHOWN_EXTRA=<some-dir>: will change the ownership and group of the specified container directory (non-recursive by default). You need to provide full paths starting with /.
   • CHOWN_EXTRA_OPTS="-R": will recursively change the ownership and group of the directory specified in CHOWN_EXTRA.
   • --user root: you must run the container with the root user to change ownership at runtime.

   now accessing the mount should work as expected:

   # assuming we mounted the volume in /home/jovyan/stagingarea
   $ ls -ld ~/stagingarea
   drwxr-xr-x 2 jovyan users 4096 Feb  1 12:55 stagingarea/
   
   $ touch stagingarea/kale.txt
   # jovyan is now the owner of /home/jovyan/stagingarea
   $ ls -la ~/stagingarea/
   -rw-r--r-- 1 jovyan users    0 Feb  1 14:41 kale.txt
   

   Additional notes:

   • If you are mounting your volume inside the /home/ directory, you can use the -e CHOWN_HOME=yes and CHOWN_HOME_OPTS="-R" flags instead of the -e CHOWN_EXTRA and -e CHOWN_EXTRA_OPTS in the example above.
   • This solution should work in most cases where you have created a docker volume (i.e. using the docker volume create --name <my-volume> command) and mounted it using the-v flag in docker run.

2. Matching the container's UID/GID with the host's

   Docker handles mounting host directories differently from mounting volumes, even though the syntax is essentially the same (i.e. -v).

   When you initialize a Docker container using the flag -v, the host directories are bind-mounted directly into the container. Therefore, the permissions and ownership are copied over and will be the same as the ones in your local host (including user ids) which may result in permissions errors like the one displayed above.

   Suppose your local user has a UID and GID of 1234. To fix the UID discrepancies between your local directories and the container's directories, you can run the container with an explicit NB_UID and NB_GID to match the that of the local user:

   $ docker run -it --rm \
       --user root \
       -p 8888:8888 \
       -e NB_UID=1234 \
       -e NB_GID=1234 \
       -v $(PWD)/test:/home/jovyan/work \
       jupyter/minimal-notebook:latest
   
   # you should see an output similar to this
   Update jovyan's UID:GID to 1234:1234
   Running as jovyan: bash
   

   where:

   • NB_IUD and NB_GID should match the local user's UID and GID.
   • You must use --user root to ensure that the UID and GID are updated at runtime.

   Additional notes:

   • The caveat with this approach is that since these changes are applied at runtime, you will need to re-run the same command with the appropriate flags and environment variables if you need to recreate the container (i.e. after removing/destroying it).

   • This approach only updates the UID and GID of the existing jovyan user instead of creating a new user, so from the above example you'd get the following:

     $ id
     uid=1234(jovyan) gid=1234(jovyan) groups=1234(jovyan),100(users)
     

   • If you pass a numeric UID, it must be in the range of 0-2147483647

     If you need to specify a different user check the following section in this page.

Permission issues after changing the UID/GIU and USER in the container

If on top of changing the UID and GID you also need to create a new user, you might be experiencing any of the following issues:

• root is the owner of /home or a mounted volume
• when starting the container, you get an error such as Failed to change ownership of the home directory.
• getting permission denied when trying to conda install packages

Some things to try:

1. Ensure the new user has ownership of /home and volume mounts

   For example, say you want to create a user callisto with a GID and UID of 1234, you will have to add the following flags to the docker run command:

    docker run -it --rm \
        --user root \
        -e NB_USER=callisto \
        -e NB_UID=1234 \
        -e NB_GID=1234 \
        -e CHOWN_HOME=yes \
        -e CHOWN_HOME_OPTS="-R" \
        -w "/home/${NB_USER}" \
        -v $(PWD)/test:/home/callisto/work \
        jupyter/minimal-notebook
   
    # expected output
    Updated the jovyan user:
    - username: jovyan       -> callisto
    - home dir: /home/jovyan -> /home/callisto
    Update callisto's UID:GID to 1234:1234
    Attempting to copy /home/jovyan to /home/callisto...
    Success!
    Ensuring /home/callisto is owned by 1234:1234
    Running as callisto: bash
   

   where:

   • -e NB_USER=callisto: will create a new user callisto and automatically add it to the users group (does not delete jovyan)
   • -e NB_UID=1234 and -e NB_GID=1234: will set the UID and GID of the new user (callisto) to 1234
   • -e CHOWN_HOME_OPTS="-R" and -e CHOWN_HOME=yes: ensure that the new user is the owner of the /home directory and subdirectories (setting CHOWN_HOME_OPTS="-R will ensure this change is applied recursively)
   • -w "/home/${NB_USER}" sets the working directory to be the new user's home
   • mounting volumes: in this case, the -v flag is used to mount the local volume onto the /home directory; if you, however, are mounting these volumes elsewhere, you also need to use the -e CHOWN_EXTRA=<some-dir> flag to avoid any permission issues (see section above)

2. Dynamically assign the user ID and GID

   The above case ensures that the /home directory is owned by the a newly created user with an specific UID and GID, but if you want to assign the UID and GID of the new user dynamically you can make the following adjustments:

   docker run -it --rm \
       --user root \
       -e NB_USER=callisto \
       -e NB_UID="$(id -u)" \
       -e NB_GID="$(id -g)"  \
       -e CHOWN_HOME=yes \
       -e CHOWN_HOME_OPTS="-R" \
       -w "/home/${NB_USER}" \
       -v $(PWD)/test:/home/callisto/work \
       jupyter/minimal-notebook
   

   where:

   • "$(id -u)" and "$(id -g)" will dynamically assign the UID and GID of the new user (callisto) to that of the local user executing the run command

Additional tips and troubleshooting commands

• Use absolute paths when using the -v flag:

  -v $(PWD)/<my-vol>:/home/jovyan/work
  

  In this example, we use the syntax $(PWD), which is replaced with the full path to the current directory. The destination path should also be an absolute path starting with a / such as home/jovyan/work.

• You might want to consider using the Docker native --user <UID> and --group-add users flags instead of -e NB_GID and -e NB_UID:

  # note this will use the same UID from
  # the user calling the command, thus matching the local host
  
  docker run -it --rm \
      --user "$(id -u)" --group-add users \
      -v <my-vol>:/home/jovyan/work jupyter/datascience-notebook
  

  This command will launch the container with a specific user UID and add that user to the users group so that it can modify the files in the default /home and /opt/conda directories. Further avoiding issues when trying to conda install additional packages.

• Use docker inspect <container_id> and look for the Mounts section to verify that the volume was created and mounted accordingly:

  # for example, for a my-vol volume created with
  # docker volume create --name <my-vol>
  
  "Mounts": [
        {
            "Type": "volume",
            "Name": "my-vol",
            "Source": "/var/lib/docker/volumes/stagingarea/_data",
            "Destination": "/home/jovyan/stagingarea",
            "Driver": "local",
            "Mode": "z",
            "RW": true,
            "Propagation": ""
        }
    ],