hazza/docker-stacks
1
0
Fork 0
mirror of https://github.com/jupyter/docker-stacks.git synced 2025-10-18 15:32:56 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'master' into asalikhov/strict_python_version

Browse Source
This commit is contained in:
Ayaz Salikhov
2021-06-07 23:03:33 +03:00
parent b58d834716 396024a4dd
commit cbdc85ac55
29 changed files with 572 additions and 501 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
.github/workflows/docker.yml vendored
View File

@@ -65,7 +65,7 @@ jobs:
uses: docker/login-action@v1 uses: docker/login-action@v1
with: with:
username: ${{secrets.DOCKERHUB_USERNAME}} username: ${{secrets.DOCKERHUB_USERNAME}}
password: ${{secrets.DOCKERHUB_PASSWORD}} password: ${{secrets.DOCKERHUB_TOKEN}}
- name: Push Images to DockerHub - name: Push Images to DockerHub
if: github.ref == 'refs/heads/master' if: github.ref == 'refs/heads/master'
run: make -C main push-all run: make -C main push-all

66
README.md
View File

@@ -10,38 +10,6 @@
Jupyter Docker Stacks are a set of ready-to-run [Docker images](https://hub.docker.com/u/jupyter) Jupyter Docker Stacks are a set of ready-to-run [Docker images](https://hub.docker.com/u/jupyter)
containing Jupyter applications and interactive computing tools. containing Jupyter applications and interactive computing tools.
## Maintainer Help Wanted
We value all positive contributions to the Docker stacks project, from
[bug reports](https://jupyter-docker-stacks.readthedocs.io/en/latest/contributing/issues.html) to
[pull requests](https://jupyter-docker-stacks.readthedocs.io/en/latest/contributing/packages.html)
to
[translations](https://jupyter-docker-stacks.readthedocs.io/en/latest/contributing/translations.html)
to help answering questions. We'd also like to invite members of the community to help with two
maintainer activities:
- Issue triage: Reading and providing a first response to issues, labeling issues appropriately,
redirecting cross-project questions to Jupyter Discourse
- Pull request reviews: Reading proposed documentation and code changes, working with the submitter
to improve the contribution, deciding if the contribution should take another form (e.g., a recipe
instead of a permanent change to the images)
Anyone in the community can jump in and help with these activities at any time. We will happily
grant additional permissions (e.g., ability to merge PRs) to anyone who shows an on-going interest
in working on the project.
## Jupyter Notebook Deprecation Notice
Following [Jupyter Notebook notice](https://github.com/jupyter/notebook#notice), we encourage users to transition to JupyterLab.
This can be done by passing the environment variable `JUPYTER_ENABLE_LAB=yes` at container startup,
more information is available in the [documentation](https://jupyter-docker-stacks.readthedocs.io/en/latest/using/common.html#docker-options).
At some point, JupyterLab will become the default for all of the Jupyter Docker stack images, however a new environment variable will be introduced to switch back to Jupyter Notebook if needed.
After the change of default, and according to the Jupyter Notebook project status and its compatibility with JupyterLab, these Docker images may remove the classic Jupyter Notebook interface altogether in favor of another _classic-like_ UI built atop JupyterLab.
This change is tracked in the issue [#1217](https://github.com/jupyter/docker-stacks/issues/1217), please check its content for more information.
## Quick Start ## Quick Start
You can try a You can try a
@@ -81,7 +49,7 @@ JupyterLab, where `hostname` is the name of the computer running docker and `tok
token printed in the console. Docker destroys the container after notebook server exit, but any token printed in the console. Docker destroys the container after notebook server exit, but any
files written to `~/work` in the container remain intact on the host. files written to `~/work` in the container remain intact on the host.
docker run --rm -p 10000:8888 -e JUPYTER_ENABLE_LAB=yes -v "$PWD":/home/jovyan/work jupyter/datascience-notebook:33add21fab64 docker run --rm -p 10000:8888 -e JUPYTER_ENABLE_LAB=yes -v "${PWD}":/home/jovyan/work jupyter/datascience-notebook:33add21fab64
## Contributing ## Contributing
@@ -89,6 +57,38 @@ Please see the [Contributor Guide on ReadTheDocs](https://jupyter-docker-stacks.
information about how to contribute package updates, recipes, features, tests, and community information about how to contribute package updates, recipes, features, tests, and community
maintained stacks. maintained stacks.
## Maintainer Help Wanted
We value all positive contributions to the Docker stacks project, from
[bug reports](https://jupyter-docker-stacks.readthedocs.io/en/latest/contributing/issues.html) to
[pull requests](https://jupyter-docker-stacks.readthedocs.io/en/latest/contributing/packages.html)
to
[translations](https://jupyter-docker-stacks.readthedocs.io/en/latest/contributing/translations.html)
to help answering questions. We'd also like to invite members of the community to help with two
maintainer activities:
- Issue triage: Reading and providing a first response to issues, labeling issues appropriately,
redirecting cross-project questions to Jupyter Discourse
- Pull request reviews: Reading proposed documentation and code changes, working with the submitter
to improve the contribution, deciding if the contribution should take another form (e.g., a recipe
instead of a permanent change to the images)
Anyone in the community can jump in and help with these activities at any time. We will happily
grant additional permissions (e.g., ability to merge PRs) to anyone who shows an on-going interest
in working on the project.
## Jupyter Notebook Deprecation Notice
Following [Jupyter Notebook notice](https://github.com/jupyter/notebook#notice), we encourage users to transition to JupyterLab.
This can be done by passing the environment variable `JUPYTER_ENABLE_LAB=yes` at container startup,
more information is available in the [documentation](https://jupyter-docker-stacks.readthedocs.io/en/latest/using/common.html#docker-options).
At some point, JupyterLab will become the default for all of the Jupyter Docker stack images, however a new environment variable will be introduced to switch back to Jupyter Notebook if needed.
After the change of default, and according to the Jupyter Notebook project status and its compatibility with JupyterLab, these Docker images may remove the classic Jupyter Notebook interface altogether in favor of another _classic-like_ UI built atop JupyterLab.
This change is tracked in the issue [#1217](https://github.com/jupyter/docker-stacks/issues/1217), please check its content for more information.
## Alternatives ## Alternatives
- [jupyter/repo2docker](https://github.com/jupyter/repo2docker) - Turn git repositories into - [jupyter/repo2docker](https://github.com/jupyter/repo2docker) - Turn git repositories into

2
all-spark-notebook/Dockerfile
View File

@@ -19,7 +19,7 @@ RUN apt-get update --yes && \
gcc && \ gcc && \
apt-get clean && rm -rf /var/lib/apt/lists/* apt-get clean && rm -rf /var/lib/apt/lists/*
USER $NB_UID USER ${NB_UID}
# R packages including IRKernel which gets installed globally. # R packages including IRKernel which gets installed globally.
RUN conda install --quiet --yes \ RUN conda install --quiet --yes \

36
base-notebook/Dockerfile
View File

@@ -60,14 +60,14 @@ RUN apt-get update --yes && \
# Configure environment # Configure environment
ENV CONDA_DIR=/opt/conda \ ENV CONDA_DIR=/opt/conda \
SHELL=/bin/bash \ SHELL=/bin/bash \
NB_USER=$NB_USER \ NB_USER="${NB_USER}" \
NB_UID=$NB_UID \ NB_UID=${NB_UID} \
NB_GID=$NB_GID \ NB_GID=${NB_GID} \
LC_ALL=en_US.UTF-8 \ LC_ALL=en_US.UTF-8 \
LANG=en_US.UTF-8 \ LANG=en_US.UTF-8 \
LANGUAGE=en_US.UTF-8 LANGUAGE=en_US.UTF-8
ENV PATH=$CONDA_DIR/bin:$PATH \ ENV PATH="${CONDA_DIR}/bin:${PATH}" \
HOME=/home/$NB_USER \ HOME="/home/${NB_USER}" \
CONDA_VERSION="${conda_version}" \ CONDA_VERSION="${conda_version}" \
MINIFORGE_VERSION="${miniforge_version}" MINIFORGE_VERSION="${miniforge_version}"
@@ -86,18 +86,18 @@ RUN sed -i 's/^#force_color_prompt=yes/force_color_prompt=yes/' /etc/skel/.bashr
RUN echo "auth requisite pam_deny.so" >> /etc/pam.d/su && \ RUN echo "auth requisite pam_deny.so" >> /etc/pam.d/su && \
sed -i.bak -e 's/^%admin/#%admin/' /etc/sudoers && \ sed -i.bak -e 's/^%admin/#%admin/' /etc/sudoers && \
sed -i.bak -e 's/^%sudo/#%sudo/' /etc/sudoers && \ sed -i.bak -e 's/^%sudo/#%sudo/' /etc/sudoers && \
useradd -l -m -s /bin/bash -N -u $NB_UID $NB_USER && \ useradd -l -m -s /bin/bash -N -u "${NB_UID}" "${NB_USER}" && \
mkdir -p $CONDA_DIR && \ mkdir -p "${CONDA_DIR}" && \
chown $NB_USER:$NB_GID $CONDA_DIR && \ chown "${NB_USER}:${NB_GID}" "${CONDA_DIR}" && \
chmod g+w /etc/passwd && \ chmod g+w /etc/passwd && \
fix-permissions "${HOME}" && \ fix-permissions "${HOME}" && \
fix-permissions "${CONDA_DIR}" fix-permissions "${CONDA_DIR}"
USER $NB_UID USER ${NB_UID}
ARG PYTHON_VERSION=default ARG PYTHON_VERSION=default
# Setup work directory for backward-compatibility # Setup work directory for backward-compatibility
RUN mkdir "/home/$NB_USER/work" && \ RUN mkdir "/home/${NB_USER}/work" && \
fix-permissions "/home/${NB_USER}" fix-permissions "/home/${NB_USER}"
# Install conda as jovyan and check the sha256 sum provided on the download site # Install conda as jovyan and check the sha256 sum provided on the download site
@@ -106,20 +106,20 @@ WORKDIR /tmp
# Prerequisites installation: conda, mamba, pip, tini # Prerequisites installation: conda, mamba, pip, tini
RUN wget --quiet "https://github.com/conda-forge/miniforge/releases/download/${miniforge_version}/${miniforge_installer}" && \ RUN wget --quiet "https://github.com/conda-forge/miniforge/releases/download/${miniforge_version}/${miniforge_installer}" && \
echo "${miniforge_checksum} *${miniforge_installer}" | sha256sum --check && \ echo "${miniforge_checksum} *${miniforge_installer}" | sha256sum --check && \
/bin/bash "${miniforge_installer}" -f -b -p $CONDA_DIR && \ /bin/bash "${miniforge_installer}" -f -b -p "${CONDA_DIR}" && \
rm "${miniforge_installer}" && \ rm "${miniforge_installer}" && \
# Conda configuration see https://conda.io/projects/conda/en/latest/configuration.html # Conda configuration see https://conda.io/projects/conda/en/latest/configuration.html
echo "conda ${CONDA_VERSION}" >> $CONDA_DIR/conda-meta/pinned && \ echo "conda ${CONDA_VERSION}" >> "${CONDA_DIR}/conda-meta/pinned" && \
conda config --system --set auto_update_conda false && \ conda config --system --set auto_update_conda false && \
conda config --system --set show_channel_urls true && \ conda config --system --set show_channel_urls true && \
if [ ! $PYTHON_VERSION = 'default' ]; then conda install --yes python=$PYTHON_VERSION; fi && \ if [[ "${PYTHON_VERSION}" != "default" ]]; then conda install --yes python="${PYTHON_VERSION}"; fi && \
conda list python | grep '^python ' | tr -s ' ' | cut -d ' ' -f 1,2 >> $CONDA_DIR/conda-meta/pinned && \ conda list python | grep '^python ' | tr -s ' ' | cut -d ' ' -f 1,2 >> "${CONDA_DIR}/conda-meta/pinned" && \
conda install --quiet --yes \ conda install --quiet --yes \
"conda=${CONDA_VERSION}" \ "conda=${CONDA_VERSION}" \
'pip' && \ 'pip' && \
conda update --all --quiet --yes && \ conda update --all --quiet --yes && \
conda clean --all -f -y && \ conda clean --all -f -y && \
rm -rf /home/$NB_USER/.cache/yarn && \ rm -rf "/home/${NB_USER}/.cache/yarn" && \
fix-permissions "${CONDA_DIR}" && \ fix-permissions "${CONDA_DIR}" && \
fix-permissions "/home/${NB_USER}" fix-permissions "/home/${NB_USER}"
@@ -137,7 +137,7 @@ RUN conda install --quiet --yes \
npm cache clean --force && \ npm cache clean --force && \
jupyter notebook --generate-config && \ jupyter notebook --generate-config && \
jupyter lab clean && \ jupyter lab clean && \
rm -rf /home/$NB_USER/.cache/yarn && \ rm -rf "/home/${NB_USER}/.cache/yarn" && \
fix-permissions "${CONDA_DIR}" && \ fix-permissions "${CONDA_DIR}" && \
fix-permissions "/home/${NB_USER}" fix-permissions "/home/${NB_USER}"
@@ -161,6 +161,6 @@ RUN sed -re "s/c.NotebookApp/c.ServerApp/g" \
fix-permissions /etc/jupyter/ fix-permissions /etc/jupyter/
# Switch back to jovyan to avoid accidental container runs as root # Switch back to jovyan to avoid accidental container runs as root
USER $NB_UID USER ${NB_UID}
WORKDIR $HOME WORKDIR "${HOME}"

8
base-notebook/fix-permissions
View File

@@ -2,7 +2,7 @@
# set permissions on a directory # set permissions on a directory
# after any installation, if a directory needs to be (human) user-writable, # after any installation, if a directory needs to be (human) user-writable,
# run this script on it. # run this script on it.
# It will make everything in the directory owned by the group $NB_GID # It will make everything in the directory owned by the group ${NB_GID}
# and writable by that group. # and writable by that group.
# Deployments that want to set a specific user id can preserve permissions # Deployments that want to set a specific user id can preserve permissions
# by adding the `--group-add users` line to `docker run`. # by adding the `--group-add users` line to `docker run`.
@@ -11,14 +11,14 @@
# which would cause massive image explosion # which would cause massive image explosion
# right permissions are: # right permissions are:
# group=$NB_GID # group=${NB_GID}
# AND permissions include group rwX (directory-execute) # AND permissions include group rwX (directory-execute)
# AND directories have setuid,setgid bits set # AND directories have setuid,setgid bits set
set -e set -e
for d in "$@"; do for d in "$@"; do
find "$d" \ find "${d}" \
! \( \ ! \( \
-group "${NB_GID}" \ -group "${NB_GID}" \
-a -perm -g+rwX \ -a -perm -g+rwX \
@@ -26,7 +26,7 @@ for d in "$@"; do
-exec chgrp "${NB_GID}" {} \; \ -exec chgrp "${NB_GID}" {} \; \
-exec chmod g+rwX {} \; -exec chmod g+rwX {} \;
# setuid, setgid *on directories only* # setuid, setgid *on directories only*
find "$d" \ find "${d}" \
\( \ \( \
-type d \ -type d \
-a ! -perm -6000 \ -a ! -perm -6000 \

4
base-notebook/start-notebook.sh
View File

@@ -14,9 +14,9 @@ if [[ -n "${JUPYTERHUB_API_TOKEN}" ]]; then
exec /usr/local/bin/start-singleuser.sh "$@" exec /usr/local/bin/start-singleuser.sh "$@"
elif [[ -n "${JUPYTER_ENABLE_LAB}" ]]; then elif [[ -n "${JUPYTER_ENABLE_LAB}" ]]; then
# shellcheck disable=SC1091 # shellcheck disable=SC1091
. /usr/local/bin/start.sh $wrapper jupyter lab "$@" . /usr/local/bin/start.sh ${wrapper} jupyter lab "$@"
else else
echo "WARN: Jupyter Notebook deprecation notice https://github.com/jupyter/docker-stacks#jupyter-notebook-deprecation-notice." echo "WARN: Jupyter Notebook deprecation notice https://github.com/jupyter/docker-stacks#jupyter-notebook-deprecation-notice."
# shellcheck disable=SC1091 # shellcheck disable=SC1091
. /usr/local/bin/start.sh $wrapper jupyter notebook "$@" . /usr/local/bin/start.sh ${wrapper} jupyter notebook "$@"
fi fi

34
base-notebook/start-singleuser.sh
View File

@@ -5,37 +5,37 @@
set -e set -e
# set default ip to 0.0.0.0 # set default ip to 0.0.0.0
if [[ "$NOTEBOOK_ARGS $*" != *"--ip="* ]]; then if [[ "${NOTEBOOK_ARGS} $*" != *"--ip="* ]]; then
NOTEBOOK_ARGS="--ip=0.0.0.0 $NOTEBOOK_ARGS" NOTEBOOK_ARGS="--ip=0.0.0.0 ${NOTEBOOK_ARGS}"
fi fi
# handle some deprecated environment variables # handle some deprecated environment variables
# from DockerSpawner < 0.8. # from DockerSpawner < 0.8.
# These won't be passed from DockerSpawner 0.9, # These won't be passed from DockerSpawner 0.9,
# so avoid specifying --arg=empty-string # so avoid specifying --arg=empty-string
if [ -n "$NOTEBOOK_DIR" ]; then if [ -n "${NOTEBOOK_DIR}" ]; then
# shellcheck disable=SC2089 # shellcheck disable=SC2089
NOTEBOOK_ARGS="--notebook-dir='$NOTEBOOK_DIR' $NOTEBOOK_ARGS" NOTEBOOK_ARGS="--notebook-dir='${NOTEBOOK_DIR}' ${NOTEBOOK_ARGS}"
fi fi
if [ -n "$JPY_PORT" ]; then if [ -n "${JPY_PORT}" ]; then
NOTEBOOK_ARGS="--port=$JPY_PORT $NOTEBOOK_ARGS" NOTEBOOK_ARGS="--port=${JPY_PORT} ${NOTEBOOK_ARGS}"
fi fi
if [ -n "$JPY_USER" ]; then if [ -n "${JPY_USER}" ]; then
NOTEBOOK_ARGS="--user=$JPY_USER $NOTEBOOK_ARGS" NOTEBOOK_ARGS="--user=${JPY_USER} ${NOTEBOOK_ARGS}"
fi fi
if [ -n "$JPY_COOKIE_NAME" ]; then if [ -n "${JPY_COOKIE_NAME}" ]; then
NOTEBOOK_ARGS="--cookie-name=$JPY_COOKIE_NAME $NOTEBOOK_ARGS" NOTEBOOK_ARGS="--cookie-name=${JPY_COOKIE_NAME} ${NOTEBOOK_ARGS}"
fi fi
if [ -n "$JPY_BASE_URL" ]; then if [ -n "${JPY_BASE_URL}" ]; then
NOTEBOOK_ARGS="--base-url=$JPY_BASE_URL $NOTEBOOK_ARGS" NOTEBOOK_ARGS="--base-url=${JPY_BASE_URL} ${NOTEBOOK_ARGS}"
fi fi
if [ -n "$JPY_HUB_PREFIX" ]; then if [ -n "${JPY_HUB_PREFIX}" ]; then
NOTEBOOK_ARGS="--hub-prefix=$JPY_HUB_PREFIX $NOTEBOOK_ARGS" NOTEBOOK_ARGS="--hub-prefix=${JPY_HUB_PREFIX} ${NOTEBOOK_ARGS}"
fi fi
if [ -n "$JPY_HUB_API_URL" ]; then if [ -n "${JPY_HUB_API_URL}" ]; then
NOTEBOOK_ARGS="--hub-api-url=$JPY_HUB_API_URL $NOTEBOOK_ARGS" NOTEBOOK_ARGS="--hub-api-url=${JPY_HUB_API_URL} ${NOTEBOOK_ARGS}"
fi fi
NOTEBOOK_BIN="jupyterhub-singleuser" NOTEBOOK_BIN="jupyterhub-singleuser"
# shellcheck disable=SC1091,SC2086,SC2090 # shellcheck disable=SC1091,SC2086,SC2090
. /usr/local/bin/start.sh "$NOTEBOOK_BIN" $NOTEBOOK_ARGS "$@" . /usr/local/bin/start.sh "${NOTEBOOK_BIN}" ${NOTEBOOK_ARGS} "$@"

98
base-notebook/start.sh
View File

@@ -13,28 +13,28 @@ fi
run-hooks () { run-hooks () {
# Source scripts or run executable files in a directory # Source scripts or run executable files in a directory
if [[ ! -d "$1" ]] ; then if [[ ! -d "${1}" ]] ; then
return return
fi fi
echo "$0: running hooks in $1" echo "${0}: running hooks in ${1}"
for f in "$1/"*; do for f in "${1}/"*; do
case "$f" in case "${f}" in
*.sh) *.sh)
echo "$0: running $f" echo "${0}: running ${f}"
# shellcheck disable=SC1090 # shellcheck disable=SC1090
source "$f" source "${f}"
;; ;;
*) *)
if [[ -x "$f" ]] ; then if [[ -x "${f}" ]] ; then
echo "$0: running $f" echo "${0}: running ${f}"
"$f" "${f}"
else else
echo "$0: ignoring $f" echo "${0}: ignoring ${f}"
fi fi
;; ;;
esac esac
done done
echo "$0: done running hooks in $1" echo "${0}: done running hooks in ${1}"
} }
run-hooks /usr/local/bin/start-notebook.d run-hooks /usr/local/bin/start-notebook.d
@@ -44,73 +44,73 @@ if [ "$(id -u)" == 0 ] ; then
# Only attempt to change the jovyan username if it exists # Only attempt to change the jovyan username if it exists
if id jovyan &> /dev/null ; then if id jovyan &> /dev/null ; then
echo "Set username to: $NB_USER" echo "Set username to: ${NB_USER}"
usermod -d "/home/$NB_USER" -l "$NB_USER" jovyan usermod -d "/home/${NB_USER}" -l "${NB_USER}" jovyan
fi fi
# handle home and working directory if the username changed # handle home and working directory if the username changed
if [[ "$NB_USER" != "jovyan" ]]; then if [[ "${NB_USER}" != "jovyan" ]]; then
# changing username, make sure homedir exists # changing username, make sure homedir exists
# (it could be mounted, and we shouldn't create it if it already exists) # (it could be mounted, and we shouldn't create it if it already exists)
if [[ ! -e "/home/$NB_USER" ]]; then if [[ ! -e "/home/${NB_USER}" ]]; then
echo "Relocating home dir to /home/$NB_USER" echo "Relocating home dir to /home/${NB_USER}"
mv /home/jovyan "/home/$NB_USER" || ln -s /home/jovyan "/home/$NB_USER" mv /home/jovyan "/home/${NB_USER}" || ln -s /home/jovyan "/home/${NB_USER}"
fi fi
# if workdir is in /home/jovyan, cd to /home/$NB_USER # if workdir is in /home/jovyan, cd to /home/${NB_USER}
if [[ "$PWD/" == "/home/jovyan/"* ]]; then if [[ "${PWD}/" == "/home/jovyan/"* ]]; then
newcwd="/home/$NB_USER/${PWD:13}" newcwd="/home/${NB_USER}/${PWD:13}"
echo "Setting CWD to $newcwd" echo "Setting CWD to ${newcwd}"
cd "$newcwd" cd "${newcwd}"
fi fi
fi fi
# Handle case where provisioned storage does not have the correct permissions by default # Handle case where provisioned storage does not have the correct permissions by default
# Ex: default NFS/EFS (no auto-uid/gid) # Ex: default NFS/EFS (no auto-uid/gid)
if [[ "$CHOWN_HOME" == "1" || "$CHOWN_HOME" == 'yes' ]]; then if [[ "${CHOWN_HOME}" == "1" || "${CHOWN_HOME}" == 'yes' ]]; then
echo "Changing ownership of /home/$NB_USER to $NB_UID:$NB_GID with options '${CHOWN_HOME_OPTS}'" echo "Changing ownership of /home/${NB_USER} to ${NB_UID}:${NB_GID} with options '${CHOWN_HOME_OPTS}'"
# shellcheck disable=SC2086 # shellcheck disable=SC2086
chown $CHOWN_HOME_OPTS "$NB_UID:$NB_GID" "/home/$NB_USER" chown ${CHOWN_HOME_OPTS} "${NB_UID}:${NB_GID}" "/home/${NB_USER}"
fi fi
if [ -n "$CHOWN_EXTRA" ]; then if [ -n "${CHOWN_EXTRA}" ]; then
for extra_dir in $(echo "$CHOWN_EXTRA" | tr ',' ' '); do for extra_dir in $(echo "${CHOWN_EXTRA}" | tr ',' ' '); do
echo "Changing ownership of ${extra_dir} to $NB_UID:$NB_GID with options '${CHOWN_EXTRA_OPTS}'" echo "Changing ownership of ${extra_dir} to ${NB_UID}:${NB_GID} with options '${CHOWN_EXTRA_OPTS}'"
# shellcheck disable=SC2086 # shellcheck disable=SC2086
chown $CHOWN_EXTRA_OPTS "$NB_UID:$NB_GID" "$extra_dir" chown ${CHOWN_EXTRA_OPTS} "${NB_UID}:${NB_GID}" "${extra_dir}"
done done
fi fi
# Change UID:GID of NB_USER to NB_UID:NB_GID if it does not match # Change UID:GID of NB_USER to NB_UID:NB_GID if it does not match
if [ "$NB_UID" != "$(id -u "$NB_USER")" ] || [ "$NB_GID" != "$(id -g "$NB_USER")" ]; then if [ "${NB_UID}" != "$(id -u "${NB_USER}")" ] || [ "${NB_GID}" != "$(id -g "${NB_USER}")" ]; then
echo "Set user $NB_USER UID:GID to: $NB_UID:$NB_GID" echo "Set user ${NB_USER} UID:GID to: ${NB_UID}:${NB_GID}"
if [ "$NB_GID" != "$(id -g "$NB_USER")" ]; then if [ "${NB_GID}" != "$(id -g "${NB_USER}")" ]; then
groupadd -f -g "$NB_GID" -o "${NB_GROUP:-${NB_USER}}" groupadd -f -g "${NB_GID}" -o "${NB_GROUP:-${NB_USER}}"
fi fi
userdel "$NB_USER" userdel "${NB_USER}"
useradd --home "/home/$NB_USER" -u "$NB_UID" -g "$NB_GID" -G 100 -l "$NB_USER" useradd --home "/home/${NB_USER}" -u "${NB_UID}" -g "${NB_GID}" -G 100 -l "${NB_USER}"
fi fi
# Enable sudo if requested # Enable sudo if requested
if [[ "$GRANT_SUDO" == "1" || "$GRANT_SUDO" == 'yes' ]]; then if [[ "${GRANT_SUDO}" == "1" || "${GRANT_SUDO}" == 'yes' ]]; then
echo "Granting $NB_USER sudo access and appending $CONDA_DIR/bin to sudo PATH" echo "Granting ${NB_USER} sudo access and appending ${CONDA_DIR}/bin to sudo PATH"
echo "$NB_USER ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/notebook echo "${NB_USER} ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/notebook
fi fi
# Add $CONDA_DIR/bin to sudo secure_path # Add ${CONDA_DIR}/bin to sudo secure_path
sed -r "s#Defaults\s+secure_path\s*=\s*\"?([^\"]+)\"?#Defaults secure_path=\"\1:$CONDA_DIR/bin\"#" /etc/sudoers | grep secure_path > /etc/sudoers.d/path sed -r "s#Defaults\s+secure_path\s*=\s*\"?([^\"]+)\"?#Defaults secure_path=\"\1:${CONDA_DIR}/bin\"#" /etc/sudoers | grep secure_path > /etc/sudoers.d/path
# Exec the command as NB_USER with the PATH and the rest of # Exec the command as NB_USER with the PATH and the rest of
# the environment preserved # the environment preserved
run-hooks /usr/local/bin/before-notebook.d run-hooks /usr/local/bin/before-notebook.d
echo "Executing the command:" "${cmd[@]}" echo "Executing the command:" "${cmd[@]}"
exec sudo -E -H -u "$NB_USER" PATH="$PATH" XDG_CACHE_HOME="/home/$NB_USER/.cache" PYTHONPATH="${PYTHONPATH:-}" "${cmd[@]}" exec sudo -E -H -u "${NB_USER}" PATH="${PATH}" XDG_CACHE_HOME="/home/${NB_USER}/.cache" PYTHONPATH="${PYTHONPATH:-}" "${cmd[@]}"
else else
if [[ "$NB_UID" == "$(id -u jovyan 2>/dev/null)" && "$NB_GID" == "$(id -g jovyan 2>/dev/null)" ]]; then if [[ "${NB_UID}" == "$(id -u jovyan 2>/dev/null)" && "${NB_GID}" == "$(id -g jovyan 2>/dev/null)" ]]; then
# User is not attempting to override user/group via environment # User is not attempting to override user/group via environment
# variables, but they could still have overridden the uid/gid that # variables, but they could still have overridden the uid/gid that
# container runs as. Check that the user has an entry in the passwd # container runs as. Check that the user has an entry in the passwd
# file and if not add an entry. # file and if not add an entry.
STATUS=0 && whoami &> /dev/null || STATUS=$? && true STATUS=0 && whoami &> /dev/null || STATUS=$? && true
if [[ "$STATUS" != "0" ]]; then if [[ "${STATUS}" != "0" ]]; then
if [[ -w /etc/passwd ]]; then if [[ -w /etc/passwd ]]; then
echo "Adding passwd file entry for $(id -u)" echo "Adding passwd file entry for $(id -u)"
sed -e "s/^jovyan:/nayvoj:/" /etc/passwd > /tmp/passwd sed -e "s/^jovyan:/nayvoj:/" /etc/passwd > /tmp/passwd
@@ -122,24 +122,24 @@ else
fi fi
fi fi
# Warn if the user isn't going to be able to write files to $HOME. # Warn if the user isn't going to be able to write files to ${HOME}.
if [[ ! -w /home/jovyan ]]; then if [[ ! -w /home/jovyan ]]; then
echo 'Container must be run with group "users" to update files' echo 'Container must be run with group "users" to update files'
fi fi
else else
# Warn if looks like user want to override uid/gid but hasn't # Warn if looks like user want to override uid/gid but hasn't
# run the container as root. # run the container as root.
if [[ -n "$NB_UID" && "$NB_UID" != "$(id -u)" ]]; then if [[ -n "${NB_UID}" && "${NB_UID}" != "$(id -u)" ]]; then
echo "Container must be run as root to set NB_UID to $NB_UID" echo "Container must be run as root to set NB_UID to ${NB_UID}"
fi fi
if [[ -n "$NB_GID" && "$NB_GID" != "$(id -g)" ]]; then if [[ -n "${NB_GID}" && "${NB_GID}" != "$(id -g)" ]]; then
echo "Container must be run as root to set NB_GID to $NB_GID" echo "Container must be run as root to set NB_GID to ${NB_GID}"
fi fi
fi fi
# Warn if looks like user want to run in sudo mode but hasn't run # Warn if looks like user want to run in sudo mode but hasn't run
# the container as root. # the container as root.
if [[ "$GRANT_SUDO" == "1" || "$GRANT_SUDO" == 'yes' ]]; then if [[ "${GRANT_SUDO}" == "1" || "${GRANT_SUDO}" == 'yes' ]]; then
echo 'Container must be run as root to grant sudo permissions' echo 'Container must be run as root to grant sudo permissions'
fi fi

6
binder/Dockerfile
View File

@@ -8,7 +8,7 @@ FROM $BASE_CONTAINER
LABEL maintainer="Jupyter Project <jupyter@googlegroups.com>" LABEL maintainer="Jupyter Project <jupyter@googlegroups.com>"
ENV TAG="33add21fab64" ENV TAG="33add21fab64"
WORKDIR $HOME WORKDIR "${HOME}"
COPY binder/README.ipynb . COPY binder/README.ipynb .
# Fix permissions on README.ipynb as root # Fix permissions on README.ipynb as root
@@ -17,6 +17,6 @@ USER root
RUN fix-permissions README.ipynb RUN fix-permissions README.ipynb
# Switch back to jovyan to avoid accidental container runs as root # Switch back to jovyan to avoid accidental container runs as root
USER $NB_UID USER ${NB_UID}
WORKDIR $HOME WORKDIR "${HOME}"

8
datascience-notebook/Dockerfile
View File

@@ -27,7 +27,7 @@ RUN apt-get update --yes && \
apt-get clean && rm -rf /var/lib/apt/lists/* apt-get clean && rm -rf /var/lib/apt/lists/*
# Julia dependencies # Julia dependencies
# install Julia packages in /opt/julia instead of $HOME # install Julia packages in /opt/julia instead of ${HOME}
ENV JULIA_DEPOT_PATH=/opt/julia \ ENV JULIA_DEPOT_PATH=/opt/julia \
JULIA_PKGDIR=/opt/julia \ JULIA_PKGDIR=/opt/julia \
JULIA_VERSION="${julia_version}" JULIA_VERSION="${julia_version}"
@@ -44,13 +44,13 @@ RUN mkdir "/opt/julia-${JULIA_VERSION}" && \
# Show Julia where conda libraries are \ # Show Julia where conda libraries are \
RUN mkdir /etc/julia && \ RUN mkdir /etc/julia && \
echo "push!(Libdl.DL_LOAD_PATH, \"$CONDA_DIR/lib\")" >> /etc/julia/juliarc.jl && \ echo "push!(Libdl.DL_LOAD_PATH, \"${CONDA_DIR}/lib\")" >> /etc/julia/juliarc.jl && \
# Create JULIA_PKGDIR \ # Create JULIA_PKGDIR \
mkdir "${JULIA_PKGDIR}" && \ mkdir "${JULIA_PKGDIR}" && \
chown "${NB_USER}" "${JULIA_PKGDIR}" && \ chown "${NB_USER}" "${JULIA_PKGDIR}" && \
fix-permissions "${JULIA_PKGDIR}" fix-permissions "${JULIA_PKGDIR}"
USER $NB_UID USER ${NB_UID}
# R packages including IRKernel which gets installed globally. # R packages including IRKernel which gets installed globally.
RUN conda install --quiet --yes \ RUN conda install --quiet --yes \
@@ -91,4 +91,4 @@ RUN julia -e 'import Pkg; Pkg.update()' && \
rm -rf "${HOME}/.local" && \ rm -rf "${HOME}/.local" && \
fix-permissions "${JULIA_PKGDIR}" "${CONDA_DIR}/share/jupyter" fix-permissions "${JULIA_PKGDIR}" "${CONDA_DIR}/share/jupyter"
WORKDIR $HOME WORKDIR "${HOME}"

2
docs/index.rst
View File

@@ -24,7 +24,7 @@ The other pages in this documentation describe additional uses and features in d
**Example 3:** This command pulls the ``jupyter/datascience-notebook`` image tagged ``33add21fab64`` from Docker Hub if it is not already present on the local host. It then starts an *ephemeral* container running a Jupyter Notebook server and exposes the server on host port 10000. The command mounts the current working directory on the host as ``/home/jovyan/work`` in the container. The server logs appear in the terminal. Visiting ``http://<hostname>:10000/lab?token=<token>`` in a browser loads JupyterLab, where ``hostname`` is the name of the computer running docker and ``token`` is the secret token printed in the console. Docker destroys the container after notebook server exit, but any files written to ``~/work`` in the container remain intact on the host.:: **Example 3:** This command pulls the ``jupyter/datascience-notebook`` image tagged ``33add21fab64`` from Docker Hub if it is not already present on the local host. It then starts an *ephemeral* container running a Jupyter Notebook server and exposes the server on host port 10000. The command mounts the current working directory on the host as ``/home/jovyan/work`` in the container. The server logs appear in the terminal. Visiting ``http://<hostname>:10000/lab?token=<token>`` in a browser loads JupyterLab, where ``hostname`` is the name of the computer running docker and ``token`` is the secret token printed in the console. Docker destroys the container after notebook server exit, but any files written to ``~/work`` in the container remain intact on the host.::
docker run --rm -p 10000:8888 -e JUPYTER_ENABLE_LAB=yes -v "$PWD":/home/jovyan/work jupyter/datascience-notebook:33add21fab64 docker run --rm -p 10000:8888 -e JUPYTER_ENABLE_LAB=yes -v "${PWD}":/home/jovyan/work jupyter/datascience-notebook:33add21fab64
Table of Contents Table of Contents
----------------- -----------------

600
docs/locale/en/LC_MESSAGES/using.po
View File

File diff suppressed because it is too large Load Diff

18
docs/using/common.md
View File

@@ -23,19 +23,19 @@ docker run -d -p 8888:8888 jupyter/base-notebook start-notebook.sh --NotebookApp
You may instruct the `start-notebook.sh` script to customize the container environment before launching You may instruct the `start-notebook.sh` script to customize the container environment before launching
the notebook server. You do so by passing arguments to the `docker run` command. the notebook server. You do so by passing arguments to the `docker run` command.
- `-e NB_USER=jovyan` - Instructs the startup script to change the default container username from `jovyan` to the provided value. Causes the script to rename the `jovyan` user home folder. For this option to take effect, you must run the container with `--user root`, set the working directory `-w /home/$NB_USER` and set the environment variable `-e CHOWN_HOME=yes` (see below for detail). This feature is useful when mounting host volumes with specific home folder. - `-e NB_USER=jovyan` - Instructs the startup script to change the default container username from `jovyan` to the provided value. Causes the script to rename the `jovyan` user home folder. For this option to take effect, you must run the container with `--user root`, set the working directory `-w /home/${NB_USER}` and set the environment variable `-e CHOWN_HOME=yes` (see below for detail). This feature is useful when mounting host volumes with specific home folder.
- `-e NB_UID=1000` - Instructs the startup script to switch the numeric user ID of `$NB_USER` to the given value. This feature is useful when mounting host volumes with specific owner permissions. For this option to take effect, you must run the container with `--user root`. (The startup script will `su $NB_USER` after adjusting the user ID.) You might consider using modern Docker options `--user` and `--group-add` instead. See the last bullet below for details. - `-e NB_UID=1000` - Instructs the startup script to switch the numeric user ID of `${NB_USER}` to the given value. This feature is useful when mounting host volumes with specific owner permissions. For this option to take effect, you must run the container with `--user root`. (The startup script will `su ${NB_USER}` after adjusting the user ID.) You might consider using modern Docker options `--user` and `--group-add` instead. See the last bullet below for details.
- `-e NB_GID=100` - Instructs the startup script to change the primary group of`$NB_USER` to `$NB_GID` (the new group is added with a name of `$NB_GROUP` if it is defined, otherwise the group is named `$NB_USER`). This feature is useful when mounting host volumes with specific group permissions. For this option to take effect, you must run the container with `--user root`. (The startup script will `su $NB_USER` after adjusting the group ID.) You might consider using modern Docker options `--user` and `--group-add` instead. See the last bullet below for details. The user is added to supplemental group `users` (gid 100) in order to allow write access to the home directory and `/opt/conda`. If you override the user/group logic, ensure the user stays in group `users` if you want them to be able to modify files in the image. - `-e NB_GID=100` - Instructs the startup script to change the primary group of`${NB_USER}` to `${NB_GID}` (the new group is added with a name of `${NB_GROUP}` if it is defined, otherwise the group is named `${NB_USER}`). This feature is useful when mounting host volumes with specific group permissions. For this option to take effect, you must run the container with `--user root`. (The startup script will `su ${NB_USER}` after adjusting the group ID.) You might consider using modern Docker options `--user` and `--group-add` instead. See the last bullet below for details. The user is added to supplemental group `users` (gid 100) in order to allow write access to the home directory and `/opt/conda`. If you override the user/group logic, ensure the user stays in group `users` if you want them to be able to modify files in the image.
- `-e NB_GROUP=<name>` - The name used for `$NB_GID`, which defaults to `$NB_USER`. This is only used if `$NB_GID` is specified and completely optional: there is only cosmetic effect. - `-e NB_GROUP=<name>` - The name used for `${NB_GID}`, which defaults to `${NB_USER}`. This is only used if `${NB_GID}` is specified and completely optional: there is only cosmetic effect.
- `-e NB_UMASK=<umask>` - Configures Jupyter to use a different umask value from default, i.e. `022`. For example, if setting umask to `002`, new files will be readable and writable by group members instead of just writable by the owner. Wikipedia has a good article about [umask](https://en.wikipedia.org/wiki/Umask). Feel free to read it in order to choose the value that better fits your needs. Default value should fit most situations. Note that `NB_UMASK` when set only applies to the Jupyter process itself - you cannot use it to set a umask for additional files created during run-hooks e.g. via `pip` or `conda` - if you need to set a umask for these you must set `umask` for each command. - `-e NB_UMASK=<umask>` - Configures Jupyter to use a different umask value from default, i.e. `022`. For example, if setting umask to `002`, new files will be readable and writable by group members instead of just writable by the owner. Wikipedia has a good article about [umask](https://en.wikipedia.org/wiki/Umask). Feel free to read it in order to choose the value that better fits your needs. Default value should fit most situations. Note that `NB_UMASK` when set only applies to the Jupyter process itself - you cannot use it to set a umask for additional files created during run-hooks e.g. via `pip` or `conda` - if you need to set a umask for these you must set `umask` for each command.
- `-e CHOWN_HOME=yes` - Instructs the startup script to change the `$NB_USER` home directory owner and group to the current value of `$NB_UID` and `$NB_GID`. This change will take effect even if the user home directory is mounted from the host using `-v` as described below. The change is **not** applied recursively by default. You can change modify the `chown` behavior by setting `CHOWN_HOME_OPTS` (e.g., `-e CHOWN_HOME_OPTS='-R'`). - `-e CHOWN_HOME=yes` - Instructs the startup script to change the `${NB_USER}` home directory owner and group to the current value of `${NB_UID}` and `${NB_GID}`. This change will take effect even if the user home directory is mounted from the host using `-v` as described below. The change is **not** applied recursively by default. You can change modify the `chown` behavior by setting `CHOWN_HOME_OPTS` (e.g., `-e CHOWN_HOME_OPTS='-R'`).
- `-e CHOWN_EXTRA="<some dir>,<some other dir>"` - Instructs the startup script to change the owner and group of each comma-separated container directory to the current value of `$NB_UID` and `$NB_GID`. The change is **not** applied recursively by default. You can change modify the `chown` behavior by setting `CHOWN_EXTRA_OPTS` (e.g., `-e CHOWN_EXTRA_OPTS='-R'`). - `-e CHOWN_EXTRA="<some dir>,<some other dir>"` - Instructs the startup script to change the owner and group of each comma-separated container directory to the current value of `${NB_UID}` and `${NB_GID}`. The change is **not** applied recursively by default. You can change modify the `chown` behavior by setting `CHOWN_EXTRA_OPTS` (e.g., `-e CHOWN_EXTRA_OPTS='-R'`).
- `-e GRANT_SUDO=yes` - Instructs the startup script to grant the `NB_USER` user passwordless `sudo` capability. You do **not** need this option to allow the user to `conda` or `pip` install additional packages. This option is useful, however, when you wish to give `$NB_USER` the ability to install OS packages with `apt` or modify other root-owned files in the container. For this option to take effect, you must run the container with `--user root`. (The `start-notebook.sh` script will `su $NB_USER` after adding `$NB_USER` to sudoers.) **You should only enable `sudo` if you trust the user or if the container is running on an isolated host.** - `-e GRANT_SUDO=yes` - Instructs the startup script to grant the `NB_USER` user passwordless `sudo` capability. You do **not** need this option to allow the user to `conda` or `pip` install additional packages. This option is useful, however, when you wish to give `${NB_USER}` the ability to install OS packages with `apt` or modify other root-owned files in the container. For this option to take effect, you must run the container with `--user root`. (The `start-notebook.sh` script will `su ${NB_USER}` after adding `${NB_USER}` to sudoers.) **You should only enable `sudo` if you trust the user or if the container is running on an isolated host.**
- `-e GEN_CERT=yes` - Instructs the startup script to generates a self-signed SSL certificate and configure Jupyter Notebook to use it to accept encrypted HTTPS connections. - `-e GEN_CERT=yes` - Instructs the startup script to generates a self-signed SSL certificate and configure Jupyter Notebook to use it to accept encrypted HTTPS connections.
- `-e JUPYTER_ENABLE_LAB=yes` - Instructs the startup script to run `jupyter lab` instead of the default `jupyter notebook` command. Useful in container orchestration environments where setting environment variables is easier than change command line parameters. - `-e JUPYTER_ENABLE_LAB=yes` - Instructs the startup script to run `jupyter lab` instead of the default `jupyter notebook` command. Useful in container orchestration environments where setting environment variables is easier than change command line parameters.
- `-e RESTARTABLE=yes` - Runs Jupyter in a loop so that quitting Jupyter does not cause the container to exit. This may be useful when you need to install extensions that require restarting Jupyter. - `-e RESTARTABLE=yes` - Runs Jupyter in a loop so that quitting Jupyter does not cause the container to exit. This may be useful when you need to install extensions that require restarting Jupyter.
- `-v /some/host/folder/for/work:/home/jovyan/work` - Mounts a host machine directory as folder in the container. Useful when you want to preserve notebooks and other work even after the container is destroyed. **You must grant the within-container notebook user or group (`NB_UID` or `NB_GID`) write access to the host directory (e.g., `sudo chown 1000 /some/host/folder/for/work`).** - `-v /some/host/folder/for/work:/home/jovyan/work` - Mounts a host machine directory as folder in the container. Useful when you want to preserve notebooks and other work even after the container is destroyed. **You must grant the within-container notebook user or group (`NB_UID` or `NB_GID`) write access to the host directory (e.g., `sudo chown 1000 /some/host/folder/for/work`).**
- `--user 5000 --group-add users` - Launches the container with a specific user ID and adds that user to the `users` group so that it can modify files in the default home directory and `/opt/conda`. You can use these arguments as alternatives to setting `$NB_UID` and `$NB_GID`. - `--user 5000 --group-add users` - Launches the container with a specific user ID and adds that user to the `users` group so that it can modify files in the default home directory and `/opt/conda`. You can use these arguments as alternatives to setting `${NB_UID}` and `${NB_GID}`.
## Startup Hooks ## Startup Hooks
@@ -103,7 +103,7 @@ You can bypass the provided scripts and specify an arbitrary start command. If y
## Conda Environments ## Conda Environments
The default Python 3.x [Conda environment](https://conda.io/projects/conda/en/latest/user-guide/concepts/environments.html) resides in `/opt/conda`. The `/opt/conda/bin` directory is part of the default `jovyan` user's `$PATH`. That directory is also whitelisted for use in `sudo` commands by the `start.sh` script. The default Python 3.x [Conda environment](https://conda.io/projects/conda/en/latest/user-guide/concepts/environments.html) resides in `/opt/conda`. The `/opt/conda/bin` directory is part of the default `jovyan` user's `${PATH}`. That directory is also whitelisted for use in `sudo` commands by the `start.sh` script.
The `jovyan` user has full read/write access to the `/opt/conda` directory. You can use either `pip`, `conda` or `mamba` to install new packages without any additional permissions. The `jovyan` user has full read/write access to the `/opt/conda` directory. You can use either `pip`, `conda` or `mamba` to install new packages without any additional permissions.

44
docs/using/recipes.md
View File

@@ -84,17 +84,17 @@ FROM jupyter/scipy-notebook:latest
# Create a Python 2.x environment using conda including at least the ipython kernel # Create a Python 2.x environment using conda including at least the ipython kernel
# and the kernda utility. Add any additional packages you want available for use # and the kernda utility. Add any additional packages you want available for use
# in a Python 2 notebook to the first line here (e.g., pandas, matplotlib, etc.) # in a Python 2 notebook to the first line here (e.g., pandas, matplotlib, etc.)
RUN conda create --quiet --yes -p $CONDA_DIR/envs/python2 python=2.7 ipython ipykernel kernda && \ RUN conda create --quiet --yes -p "${CONDA_DIR}/envs/python2" python=2.7 ipython ipykernel kernda && \
conda clean --all -f -y conda clean --all -f -y
USER root USER root
# Create a global kernelspec in the image and modify it so that it properly activates # Create a global kernelspec in the image and modify it so that it properly activates
# the python2 conda environment. # the python2 conda environment.
RUN $CONDA_DIR/envs/python2/bin/python -m ipykernel install && \ RUN "${CONDA_DIR}/envs/python2/bin/python" -m ipykernel install && \
$CONDA_DIR/envs/python2/bin/kernda -o -y /usr/local/share/jupyter/kernels/python2/kernel.json "${CONDA_DIR}/envs/python2/bin/kernda" -o -y /usr/local/share/jupyter/kernels/python2/kernel.json
USER $NB_USER USER ${NB_UID}
``` ```
Ref: <https://github.com/jupyter/docker-stacks/issues/440> Ref: <https://github.com/jupyter/docker-stacks/issues/440>
@@ -113,28 +113,28 @@ ARG conda_env=python36
ARG py_ver=3.6 ARG py_ver=3.6
# you can add additional libraries you want conda to install by listing them below the first line and ending with "&& \" # you can add additional libraries you want conda to install by listing them below the first line and ending with "&& \"
RUN conda create --quiet --yes -p $CONDA_DIR/envs/$conda_env python=$py_ver ipython ipykernel && \ RUN conda create --quiet --yes -p "${CONDA_DIR}/envs/${conda_env}" python=${py_ver} ipython ipykernel && \
conda clean --all -f -y conda clean --all -f -y
# alternatively, you can comment out the lines above and uncomment those below # alternatively, you can comment out the lines above and uncomment those below
# if you'd prefer to use a YAML file present in the docker build context # if you'd prefer to use a YAML file present in the docker build context
# COPY --chown=${NB_UID}:${NB_GID} environment.yml /home/$NB_USER/tmp/ # COPY --chown=${NB_UID}:${NB_GID} environment.yml "/home/${NB_USER}/tmp/"
# RUN cd /home/$NB_USER/tmp/ && \ # RUN cd "/home/${NB_USER}/tmp/" && \
# conda env create -p $CONDA_DIR/envs/$conda_env -f environment.yml && \ # conda env create -p "${CONDA_DIR}/envs/${conda_env}" -f environment.yml && \
# conda clean --all -f -y # conda clean --all -f -y
# create Python 3.x environment and link it to jupyter # create Python 3.x environment and link it to jupyter
RUN $CONDA_DIR/envs/${conda_env}/bin/python -m ipykernel install --user --name=${conda_env} && \ RUN "${CONDA_DIR}/envs/${conda_env}/bin/python" -m ipykernel install --user --name="${conda_env}" && \
fix-permissions "${CONDA_DIR}" && \ fix-permissions "${CONDA_DIR}" && \
fix-permissions "/home/${NB_USER}" fix-permissions "/home/${NB_USER}"
# any additional pip installs can be added by uncommenting the following line # any additional pip installs can be added by uncommenting the following line
# RUN $CONDA_DIR/envs/${conda_env}/bin/pip install # RUN "${CONDA_DIR}/envs/${conda_env}/bin/pip" install
# prepend conda environment to path # prepend conda environment to path
ENV PATH $CONDA_DIR/envs/${conda_env}/bin:$PATH ENV PATH "${CONDA_DIR}/envs/${conda_env}/bin:${PATH}"
# if you want this environment to be the default one, uncomment the following line: # if you want this environment to be the default one, uncomment the following line:
# ENV CONDA_DEFAULT_ENV ${conda_env} # ENV CONDA_DEFAULT_ENV ${conda_env}
@@ -266,7 +266,7 @@ RUN rm /etc/dpkg/dpkg.cfg.d/excludes && \
dpkg -l | grep ^ii | cut -d' ' -f3 | xargs apt-get install --yes --no-install-recommends --reinstall man && \ dpkg -l | grep ^ii | cut -d' ' -f3 | xargs apt-get install --yes --no-install-recommends --reinstall man && \
apt-get clean && rm -rf /var/lib/apt/lists/* apt-get clean && rm -rf /var/lib/apt/lists/*
USER $NB_UID USER ${NB_UID}
``` ```
Adding the documentation on top of an existing singleuser image wastes a lot of space and requires Adding the documentation on top of an existing singleuser image wastes a lot of space and requires
@@ -428,7 +428,7 @@ RUN echo 'deb https://cdn-fastly.deb.debian.org/debian jessie-backports main' >
# Add hadoop binaries # Add hadoop binaries
wget https://mirrors.ukfast.co.uk/sites/ftp.apache.org/hadoop/common/hadoop-2.7.3/hadoop-2.7.3.tar.gz && \ wget https://mirrors.ukfast.co.uk/sites/ftp.apache.org/hadoop/common/hadoop-2.7.3/hadoop-2.7.3.tar.gz && \
tar -xvf hadoop-2.7.3.tar.gz -C /usr/local && \ tar -xvf hadoop-2.7.3.tar.gz -C /usr/local && \
chown -R $NB_USER:users /usr/local/hadoop-2.7.3 && \ chown -R "${NB_USER}:users" /usr/local/hadoop-2.7.3 && \
rm -f hadoop-2.7.3.tar.gz && \ rm -f hadoop-2.7.3.tar.gz && \
# Install os dependencies required for pydoop, pyhive # Install os dependencies required for pydoop, pyhive
apt-get update --yes && \ apt-get update --yes && \
@@ -447,13 +447,13 @@ RUN echo "spark.driver.extraJavaOptions -Dhdp.version=2.5.3.0-37" >> /usr/local/
echo "spark.yarn.am.extraJavaOptions -Dhdp.version=2.5.3.0-37" >> /usr/local/spark/conf/spark-defaults.conf && \ echo "spark.yarn.am.extraJavaOptions -Dhdp.version=2.5.3.0-37" >> /usr/local/spark/conf/spark-defaults.conf && \
echo "spark.master=yarn" >> /usr/local/spark/conf/spark-defaults.conf && \ echo "spark.master=yarn" >> /usr/local/spark/conf/spark-defaults.conf && \
echo "spark.hadoop.yarn.timeline-service.enabled=false" >> /usr/local/spark/conf/spark-defaults.conf && \ echo "spark.hadoop.yarn.timeline-service.enabled=false" >> /usr/local/spark/conf/spark-defaults.conf && \
chown -R $NB_USER:users /usr/local/spark/conf/spark-defaults.conf && \ chown -R "${NB_USER}:users" /usr/local/spark/conf/spark-defaults.conf && \
# Create an alternative HADOOP_CONF_HOME so we can mount as a volume and repoint # Create an alternative HADOOP_CONF_HOME so we can mount as a volume and repoint
# using ENV var if needed # using ENV var if needed
mkdir -p /etc/hadoop/conf/ && \ mkdir -p /etc/hadoop/conf/ && \
chown $NB_USER:users /etc/hadoop/conf/ chown "${NB_USER}":users /etc/hadoop/conf/
USER $NB_USER USER ${NB_UID}
# Install useful jupyter extensions and python libraries like : # Install useful jupyter extensions and python libraries like :
# - Dashboards # - Dashboards
@@ -468,7 +468,7 @@ RUN pip install --quiet --no-cache-dir jupyter_dashboards faker && \
USER root USER root
# Ensure we overwrite the kernel config so that toree connects to cluster # Ensure we overwrite the kernel config so that toree connects to cluster
RUN jupyter toree install --sys-prefix --spark_opts="--master yarn --deploy-mode client --driver-memory 512m --executor-memory 512m --executor-cores 1 --driver-java-options -Dhdp.version=2.5.3.0-37 --conf spark.hadoop.yarn.timeline-service.enabled=false" RUN jupyter toree install --sys-prefix --spark_opts="--master yarn --deploy-mode client --driver-memory 512m --executor-memory 512m --executor-cores 1 --driver-java-options -Dhdp.version=2.5.3.0-37 --conf spark.hadoop.yarn.timeline-service.enabled=false"
USER $NB_USER USER ${NB_UID}
``` ```
Credit: [britishbadger](https://github.com/britishbadger) from Credit: [britishbadger](https://github.com/britishbadger) from
@@ -503,7 +503,7 @@ NB: this works for classic notebooks only
# Update with your base image of choice # Update with your base image of choice
FROM jupyter/minimal-notebook:latest FROM jupyter/minimal-notebook:latest
USER $NB_USER USER ${NB_UID}
RUN pip install --quiet --no-cache-dir jupyter_contrib_nbextensions && \ RUN pip install --quiet --no-cache-dir jupyter_contrib_nbextensions && \
jupyter contrib nbextension install --user && \ jupyter contrib nbextension install --user && \
@@ -530,7 +530,7 @@ RUN apt-get update --yes && \
apt-get install --yes --no-install-recommends swig && \ apt-get install --yes --no-install-recommends swig && \
apt-get clean && rm -rf /var/lib/apt/lists/* apt-get clean && rm -rf /var/lib/apt/lists/*
USER $NB_UID USER ${NB_UID}
RUN pip install --quiet --no-cache-dir auto-sklearn && \ RUN pip install --quiet --no-cache-dir auto-sklearn && \
fix-permissions "${CONDA_DIR}" && \ fix-permissions "${CONDA_DIR}" && \
@@ -551,10 +551,10 @@ RUN pip install --quiet --no-cache-dir delta-spark==${DELTA_CORE_VERSION} && \
USER root USER root
RUN echo 'spark.sql.extensions io.delta.sql.DeltaSparkSessionExtension' >> $SPARK_HOME/conf/spark-defaults.conf && \ RUN echo 'spark.sql.extensions io.delta.sql.DeltaSparkSessionExtension' >> "${SPARK_HOME}/conf/spark-defaults.conf" && \
echo 'spark.sql.catalog.spark_catalog org.apache.spark.sql.delta.catalog.DeltaCatalog' >> $SPARK_HOME/conf/spark-defaults.conf echo 'spark.sql.catalog.spark_catalog org.apache.spark.sql.delta.catalog.DeltaCatalog' >> "${SPARK_HOME}/conf/spark-defaults.conf"
USER $NB_UID USER ${NB_UID}
# Trigger download of delta lake files # Trigger download of delta lake files
RUN echo "from pyspark.sql import SparkSession" > /tmp/init-delta.py && \ RUN echo "from pyspark.sql import SparkSession" > /tmp/init-delta.py && \

2
docs/using/running.md
View File

@@ -55,7 +55,7 @@ d67fe77f1a84
**Example 2** This command pulls the `jupyter/r-notebook` image tagged `33add21fab64` from Docker Hub if it is not already present on the local host. It then starts a container running a Jupyter Notebook server and exposes the server on host port 10000. The server logs appear in the terminal and include a URL to the notebook server, but with the internal container port (8888) instead of the the correct host port (10000). **Example 2** This command pulls the `jupyter/r-notebook` image tagged `33add21fab64` from Docker Hub if it is not already present on the local host. It then starts a container running a Jupyter Notebook server and exposes the server on host port 10000. The server logs appear in the terminal and include a URL to the notebook server, but with the internal container port (8888) instead of the the correct host port (10000).
```bash ```bash
$ docker run --rm -p 10000:8888 -v "$PWD":/home/jovyan/work jupyter/r-notebook:33add21fab64 $ docker run --rm -p 10000:8888 -v "${PWD}":/home/jovyan/work jupyter/r-notebook:33add21fab64
Executing the command: jupyter notebook Executing the command: jupyter notebook
[I 19:31:09.573 NotebookApp] Writing notebook server cookie secret to /home/jovyan/.local/share/jupyter/runtime/notebook_cookie_secret [I 19:31:09.573 NotebookApp] Writing notebook server cookie secret to /home/jovyan/.local/share/jupyter/runtime/notebook_cookie_secret

33
docs/using/specifics.md
View File

@@ -212,6 +212,39 @@ rdd.sum()
// 5050 // 5050
``` ```
### Define Spark Dependencies
Spark dependencies can be declared thanks to the `spark.jars.packages` property
(see [Spark Configuration](https://spark.apache.org/docs/latest/configuration.html#runtime-environment) for more information).
They can be defined as a comma-separated list of Maven coordinates at the creation of the Spark session.
```python
from pyspark.sql import SparkSession
spark = (
SparkSession.builder.appName("elasticsearch")
.config(
"spark.jars.packages",
"org.elasticsearch:elasticsearch-spark-30_2.12:7.13.0"
)
.getOrCreate()
)
```
Dependencies can also be defined in the `spark-defaults.conf`.
However, it has to be done by `root` so it should only be considered to build custom images.
```dockerfile
USER root
RUN echo "spark.jars.packages org.elasticsearch:elasticsearch-spark-30_2.12:7.13.0" >> "${SPARK_HOME}/conf/spark-defaults.conf"
USER ${NB_UID}
```
Jars will be downloaded dynamically at the creation of the Spark session and stored by default in `${HOME}/.ivy2/jars` (can be changed by setting `spark.jars.ivy`).
_Note: This example is given for [Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/hadoop/current/install.html)._
## Tensorflow ## Tensorflow
The `jupyter/tensorflow-notebook` image supports the use of The `jupyter/tensorflow-notebook` image supports the use of

18
examples/docker-compose/bin/letsencrypt.sh
View File

@@ -8,11 +8,11 @@
set -e set -e
# Get domain and email from environment # Get domain and email from environment
[ -z "$FQDN" ] && \ [ -z "${FQDN}" ] && \
echo "ERROR: Must set FQDN environment varable" && \ echo "ERROR: Must set FQDN environment varable" && \
exit 1 exit 1
[ -z "$EMAIL" ] && \ [ -z "${EMAIL}" ] && \
echo "ERROR: Must set EMAIL environment varable" && \ echo "ERROR: Must set EMAIL environment varable" && \
exit 1 exit 1
@@ -22,11 +22,11 @@ set -e
# Create Docker volume to contain the cert # Create Docker volume to contain the cert
: "${SECRETS_VOLUME:=my-notebook-secrets}" : "${SECRETS_VOLUME:=my-notebook-secrets}"
docker volume create --name $SECRETS_VOLUME 1>/dev/null docker volume create --name "${SECRETS_VOLUME}" 1>/dev/null
# Generate the cert and save it to the Docker volume # Generate the cert and save it to the Docker volume
docker run --rm -it \ docker run --rm -it \
-p 80:80 \ -p 80:80 \
-v $SECRETS_VOLUME:/etc/letsencrypt \ -v "${SECRETS_VOLUME}":/etc/letsencrypt \
quay.io/letsencrypt/letsencrypt:latest \ quay.io/letsencrypt/letsencrypt:latest \
certonly \ certonly \
--non-interactive \ --non-interactive \
@@ -34,15 +34,15 @@ docker run --rm -it \
--standalone \ --standalone \
--standalone-supported-challenges http-01 \ --standalone-supported-challenges http-01 \
--agree-tos \ --agree-tos \
--domain "$FQDN" \ --domain "${FQDN}" \
--email "$EMAIL" \ --email "${EMAIL}" \
$CERT_SERVER "${CERT_SERVER}"
# Set permissions so nobody can read the cert and key. # Set permissions so nobody can read the cert and key.
# Also symlink the certs into the root of the /etc/letsencrypt # Also symlink the certs into the root of the /etc/letsencrypt
# directory so that the FQDN doesn't have to be known later. # directory so that the FQDN doesn't have to be known later.
docker run --rm -it \ docker run --rm -it \
-v $SECRETS_VOLUME:/etc/letsencrypt \ -v "${SECRETS_VOLUME}":/etc/letsencrypt \
ubuntu:20.04 \ ubuntu:20.04 \
bash -c "ln -s /etc/letsencrypt/live/$FQDN/* /etc/letsencrypt/ && \ bash -c "ln -s /etc/letsencrypt/live/${FQDN}/* /etc/letsencrypt/ && \
find /etc/letsencrypt -type d -exec chmod 755 {} +" find /etc/letsencrypt -type d -exec chmod 755 {} +"

12
examples/docker-compose/bin/sl-dns.sh
View File

@@ -7,17 +7,17 @@ set -e
# User must have slcli installed # User must have slcli installed
which slcli > /dev/null || (echo "SoftLayer cli not found (pip install softlayer)"; exit 1) which slcli > /dev/null || (echo "SoftLayer cli not found (pip install softlayer)"; exit 1)
USAGE="Usage: $(basename "$0") machine_name [domain]" USAGE="Usage: $(basename "${0}") machine_name [domain]"
E_BADARGS=85 E_BADARGS=85
# Machine name is first command line arg # Machine name is first command line arg
MACHINE_NAME=$1 && [ -z "$MACHINE_NAME" ] && echo "$USAGE" && exit $E_BADARGS MACHINE_NAME="${1}" && [ -z "${MACHINE_NAME}" ] && echo "${USAGE}" && exit ${E_BADARGS}
# Use SOFTLAYER_DOMAIN env var if domain name not set as second arg # Use SOFTLAYER_DOMAIN env var if domain name not set as second arg
DOMAIN="${2:-$SOFTLAYER_DOMAIN}" && [ -z "$DOMAIN" ] && \ DOMAIN="${2:-$SOFTLAYER_DOMAIN}" && [ -z "${DOMAIN}" ] && \
echo "Must specify domain or set SOFTLAYER_DOMAIN environment varable" && \ echo "Must specify domain or set SOFTLAYER_DOMAIN environment varable" && \
echo "$USAGE" && exit $E_BADARGS echo "${USAGE}" && exit ${E_BADARGS}
IP=$(docker-machine ip "$MACHINE_NAME") IP=$(docker-machine ip "${MACHINE_NAME}")
slcli dns record-add "$DOMAIN" "$MACHINE_NAME" A "$IP" slcli dns record-add "${DOMAIN}" "${MACHINE_NAME}" A "${IP}"

2
examples/docker-compose/notebook/Dockerfile
View File

@@ -9,7 +9,7 @@ USER root
# Add permanent apt-get installs and other root commands here # Add permanent apt-get installs and other root commands here
# e.g., RUN apt-get install --yes --no-install-recommends npm nodejs # e.g., RUN apt-get install --yes --no-install-recommends npm nodejs
USER $NB_UID USER ${NB_UID}
# Switch back to jovyan to avoid accidental container runs as root # Switch back to jovyan to avoid accidental container runs as root
# Add permanent pip/conda installs, data files, other user libs here # Add permanent pip/conda installs, data files, other user libs here

4
examples/docker-compose/notebook/build.sh
View File

@@ -6,7 +6,7 @@ DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
# Setup environment # Setup environment
# shellcheck disable=SC1091 # shellcheck disable=SC1091
source "$DIR/env.sh" source "${DIR}/env.sh"
# Build the notebook image # Build the notebook image
docker-compose -f "$DIR/notebook.yml" build docker-compose -f "${DIR}/notebook.yml" build

4
examples/docker-compose/notebook/down.sh
View File

@@ -6,7 +6,7 @@ DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
# Setup environment # Setup environment
# shellcheck disable=SC1091 # shellcheck disable=SC1091
source "$DIR/env.sh" source "${DIR}/env.sh"
# Bring down the notebook container, using container name as project name # Bring down the notebook container, using container name as project name
docker-compose -f "$DIR/notebook.yml" -p "$NAME" down docker-compose -f "${DIR}/notebook.yml" -p "${NAME}" down

4
examples/docker-compose/notebook/env.sh
View File

@@ -14,9 +14,9 @@ export NAME
export PORT export PORT
# Container work volume name # Container work volume name
: "${WORK_VOLUME:=$NAME-work}" : "${WORK_VOLUME:=${NAME}-work}"
export WORK_VOLUME export WORK_VOLUME
# Container secrets volume name # Container secrets volume name
: "${SECRETS_VOLUME:=$NAME-secrets}" : "${SECRETS_VOLUME:=${NAME}-secrets}"
export SECRETS_VOLUME export SECRETS_VOLUME

28
examples/docker-compose/notebook/up.sh
View File

@@ -6,14 +6,14 @@ set -e
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
USAGE="Usage: $(basename "$0") [--secure | --letsencrypt] [--password PASSWORD] [--secrets SECRETS_VOLUME]" USAGE="Usage: $(basename "${0}") [--secure | --letsencrypt] [--password PASSWORD] [--secrets SECRETS_VOLUME]"
# Parse args to determine security settings # Parse args to determine security settings
SECURE=${SECURE:=no} SECURE=${SECURE:=no}
LETSENCRYPT=${LETSENCRYPT:=no} LETSENCRYPT=${LETSENCRYPT:=no}
while [[ $# -gt 0 ]]; do while [[ $# -gt 0 ]]; do
key="$1" key="${1}"
case $key in case "${key}" in
--secure) --secure)
SECURE=yes SECURE=yes
;; ;;
@@ -21,11 +21,11 @@ case $key in
LETSENCRYPT=yes LETSENCRYPT=yes
;; ;;
--secrets) --secrets)
SECRETS_VOLUME="$2" SECRETS_VOLUME="${2}"
shift # past argument shift # past argument
;; ;;
--password) --password)
PASSWORD="$2" PASSWORD="${2}"
export PASSWORD export PASSWORD
shift # past argument shift # past argument
;; ;;
@@ -35,17 +35,17 @@ esac
shift # past argument or value shift # past argument or value
done done
if [[ "$LETSENCRYPT" == yes || "$SECURE" == yes ]]; then if [[ "${LETSENCRYPT}" == yes || "${SECURE}" == yes ]]; then
if [ -z "${PASSWORD:+x}" ]; then if [ -z "${PASSWORD:+x}" ]; then
echo "ERROR: Must set PASSWORD if running in secure mode" echo "ERROR: Must set PASSWORD if running in secure mode"
echo "$USAGE" echo "${USAGE}"
exit 1 exit 1
fi fi
if [ "$LETSENCRYPT" == yes ]; then if [ "${LETSENCRYPT}" == yes ]; then
CONFIG=letsencrypt-notebook.yml CONFIG=letsencrypt-notebook.yml
if [ -z "${SECRETS_VOLUME:+x}" ]; then if [ -z "${SECRETS_VOLUME:+x}" ]; then
echo "ERROR: Must set SECRETS_VOLUME if running in letsencrypt mode" echo "ERROR: Must set SECRETS_VOLUME if running in letsencrypt mode"
echo "$USAGE" echo "${USAGE}"
exit 1 exit 1
fi fi
else else
@@ -59,14 +59,14 @@ fi
# Setup environment # Setup environment
# shellcheck disable=SC1091 # shellcheck disable=SC1091
source "$DIR/env.sh" source "${DIR}/env.sh"
# Create a Docker volume to store notebooks # Create a Docker volume to store notebooks
docker volume create --name "$WORK_VOLUME" docker volume create --name "${WORK_VOLUME}"
# Bring up a notebook container, using container name as project name # Bring up a notebook container, using container name as project name
echo "Bringing up notebook '$NAME'" echo "Bringing up notebook '${NAME}'"
docker-compose -f "$DIR/$CONFIG" -p "$NAME" up -d docker-compose -f "${DIR}/${CONFIG}" -p "${NAME}" up -d
IP=$(docker-machine ip "$(docker-machine active)") IP=$(docker-machine ip "$(docker-machine active)")
echo "Notebook $NAME listening on $IP:$PORT" echo "Notebook ${NAME} listening on ${IP}:${PORT}"

2
examples/make-deploy/Dockerfile
View File

@@ -9,7 +9,7 @@ USER root
# Add permanent apt-get installs and other root commands here # Add permanent apt-get installs and other root commands here
# e.g., RUN apt-get install --yes --no-install-recommends npm nodejs # e.g., RUN apt-get install --yes --no-install-recommends npm nodejs
USER $NB_UID USER ${NB_UID}
# Switch back to jovyan to avoid accidental container runs as root # Switch back to jovyan to avoid accidental container runs as root
# Add permanent pip/conda installs, data files, other user libs here # Add permanent pip/conda installs, data files, other user libs here

10
examples/source-to-image/README.md
View File

@@ -77,7 +77,7 @@ The supplied `assemble` script performs a few key steps.
The first steps copy files into the location they need to be when the image is run, from the directory where they are initially placed by the `s2i` command. The first steps copy files into the location they need to be when the image is run, from the directory where they are initially placed by the `s2i` command.
```bash ```bash
cp -Rf /tmp/src/. /home/$NB_USER cp -Rf /tmp/src/. "/home/${NB_USER}"
rm -rf /tmp/src rm -rf /tmp/src
``` ```
@@ -85,12 +85,12 @@ rm -rf /tmp/src
The next steps are: The next steps are:
```bash ```bash
if [ -f /home/$NB_USER/environment.yml ]; then if [ -f "/home/${NB_USER}/environment.yml" ]; then
conda env update --name root --file /home/$NB_USER/environment.yml conda env update --name root --file "/home/${NB_USER}/environment.yml"
conda clean --all -f -y conda clean --all -f -y
else else
if [ -f /home/$NB_USER/requirements.txt ]; then if [ -f "/home/${NB_USER}/requirements.txt" ]; then
pip --no-cache-dir install -r /home/$NB_USER/requirements.txt pip --no-cache-dir install -r "/home/${NB_USER}/requirements.txt"
fi fi
fi fi
``` ```

2
minimal-notebook/Dockerfile
View File

@@ -33,4 +33,4 @@ RUN apt-get update --yes && \
RUN update-alternatives --install /usr/bin/nano nano /bin/nano-tiny 10 RUN update-alternatives --install /usr/bin/nano nano /bin/nano-tiny 10
# Switch back to jovyan to avoid accidental container runs as root # Switch back to jovyan to avoid accidental container runs as root
USER $NB_UID USER ${NB_UID}

16
pyspark-notebook/Dockerfile
View File

@@ -13,9 +13,9 @@ USER root
# Spark dependencies # Spark dependencies
# Default values can be overridden at build time # Default values can be overridden at build time
# (ARGS are in lower case to distinguish them from ENV) # (ARGS are in lower case to distinguish them from ENV)
ARG spark_version="3.1.1" ARG spark_version="3.1.2"
ARG hadoop_version="3.2" ARG hadoop_version="3.2"
ARG spark_checksum="E90B31E58F6D95A42900BA4D288261D71F6C19FA39C1CB71862B792D1B5564941A320227F6AB0E09D946F16B8C1969ED2DEA2A369EC8F9D2D7099189234DE1BE" ARG spark_checksum="2385CB772F21B014CE2ABD6B8F5E815721580D6E8BC42A26D70BBCDDA8D303D886A6F12B36D40F6971B5547B70FAE62B5A96146F0421CB93D4E51491308EF5D5"
ARG openjdk_version="11" ARG openjdk_version="11"
ENV APACHE_SPARK_VERSION="${spark_version}" \ ENV APACHE_SPARK_VERSION="${spark_version}" \
@@ -39,7 +39,7 @@ WORKDIR /usr/local
# Configure Spark # Configure Spark
ENV SPARK_HOME=/usr/local/spark ENV SPARK_HOME=/usr/local/spark
ENV SPARK_OPTS="--driver-java-options=-Xms1024M --driver-java-options=-Xmx4096M --driver-java-options=-Dlog4j.logLevel=info" \ ENV SPARK_OPTS="--driver-java-options=-Xms1024M --driver-java-options=-Xmx4096M --driver-java-options=-Dlog4j.logLevel=info" \
PATH=$PATH:$SPARK_HOME/bin PATH="${PATH}:${SPARK_HOME}/bin"
RUN ln -s "spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}" spark && \ RUN ln -s "spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}" spark && \
# Add a link in the before_notebook hook in order to source automatically PYTHONPATH # Add a link in the before_notebook hook in order to source automatically PYTHONPATH
@@ -48,11 +48,11 @@ RUN ln -s "spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}" spark && \
# Fix Spark installation for Java 11 and Apache Arrow library # Fix Spark installation for Java 11 and Apache Arrow library
# see: https://github.com/apache/spark/pull/27356, https://spark.apache.org/docs/latest/#downloading # see: https://github.com/apache/spark/pull/27356, https://spark.apache.org/docs/latest/#downloading
RUN cp -p "$SPARK_HOME/conf/spark-defaults.conf.template" "$SPARK_HOME/conf/spark-defaults.conf" && \ RUN cp -p "${SPARK_HOME}/conf/spark-defaults.conf.template" "${SPARK_HOME}/conf/spark-defaults.conf" && \
echo 'spark.driver.extraJavaOptions -Dio.netty.tryReflectionSetAccessible=true' >> $SPARK_HOME/conf/spark-defaults.conf && \ echo 'spark.driver.extraJavaOptions -Dio.netty.tryReflectionSetAccessible=true' >> "${SPARK_HOME}/conf/spark-defaults.conf" && \
echo 'spark.executor.extraJavaOptions -Dio.netty.tryReflectionSetAccessible=true' >> $SPARK_HOME/conf/spark-defaults.conf echo 'spark.executor.extraJavaOptions -Dio.netty.tryReflectionSetAccessible=true' >> "${SPARK_HOME}/conf/spark-defaults.conf"
USER $NB_UID USER ${NB_UID}
# Install pyarrow # Install pyarrow
RUN conda install --quiet --yes --satisfied-skip-solve \ RUN conda install --quiet --yes --satisfied-skip-solve \
@@ -61,4 +61,4 @@ RUN conda install --quiet --yes --satisfied-skip-solve \
fix-permissions "${CONDA_DIR}" && \ fix-permissions "${CONDA_DIR}" && \
fix-permissions "/home/${NB_USER}" fix-permissions "/home/${NB_USER}"
WORKDIR $HOME WORKDIR "${HOME}"

2
r-notebook/Dockerfile
View File

@@ -21,7 +21,7 @@ RUN apt-get update --yes && \
# Fix for devtools https://github.com/conda-forge/r-devtools-feedstock/issues/4 # Fix for devtools https://github.com/conda-forge/r-devtools-feedstock/issues/4
RUN ln -s /bin/tar /bin/gtar RUN ln -s /bin/tar /bin/gtar
USER $NB_UID USER ${NB_UID}
# R packages including IRKernel which gets installed globally. # R packages including IRKernel which gets installed globally.
RUN conda install --quiet --yes \ RUN conda install --quiet --yes \

6
scipy-notebook/Dockerfile
View File

@@ -12,7 +12,7 @@ RUN apt-get update --yes && \
apt-get install --yes --no-install-recommends ffmpeg dvipng cm-super && \ apt-get install --yes --no-install-recommends ffmpeg dvipng cm-super && \
apt-get clean && rm -rf /var/lib/apt/lists/* apt-get clean && rm -rf /var/lib/apt/lists/*
USER $NB_UID USER ${NB_UID}
# Install Python 3 packages # Install Python 3 packages
RUN conda install --quiet --yes \ RUN conda install --quiet --yes \
@@ -62,6 +62,6 @@ ENV XDG_CACHE_HOME="/home/${NB_USER}/.cache/"
RUN MPLBACKEND=Agg python -c "import matplotlib.pyplot" && \ RUN MPLBACKEND=Agg python -c "import matplotlib.pyplot" && \
fix-permissions "/home/${NB_USER}" fix-permissions "/home/${NB_USER}"
USER $NB_UID USER ${NB_UID}
WORKDIR $HOME WORKDIR "${HOME}"