mirror of
https://github.com/jupyter/docker-stacks.git
synced 2025-10-17 15:02:57 +00:00
Merge branch 'master' into asalikhov/strict_python_version
This commit is contained in:
Ayaz Salikhov
2
.github/workflows/docker.yml
vendored
2
.github/workflows/docker.yml
vendored
@@ -65,7 +65,7 @@ jobs:
|
||||
uses: docker/login-action@v1
|
||||
with:
|
||||
username: ${{secrets.DOCKERHUB_USERNAME}}
|
||||
password: ${{secrets.DOCKERHUB_PASSWORD}}
|
||||
password: ${{secrets.DOCKERHUB_TOKEN}}
|
||||
- name: Push Images to DockerHub
|
||||
if: github.ref == 'refs/heads/master'
|
||||
run: make -C main push-all
|
||||
|
||||
66
README.md
66
README.md
@@ -10,38 +10,6 @@
|
||||
Jupyter Docker Stacks are a set of ready-to-run [Docker images](https://hub.docker.com/u/jupyter)
|
||||
containing Jupyter applications and interactive computing tools.
|
||||
|
||||
## Maintainer Help Wanted
|
||||
|
||||
We value all positive contributions to the Docker stacks project, from
|
||||
[bug reports](https://jupyter-docker-stacks.readthedocs.io/en/latest/contributing/issues.html) to
|
||||
[pull requests](https://jupyter-docker-stacks.readthedocs.io/en/latest/contributing/packages.html)
|
||||
to
|
||||
[translations](https://jupyter-docker-stacks.readthedocs.io/en/latest/contributing/translations.html)
|
||||
to help answering questions. We'd also like to invite members of the community to help with two
|
||||
maintainer activities:
|
||||
|
||||
- Issue triage: Reading and providing a first response to issues, labeling issues appropriately,
|
||||
redirecting cross-project questions to Jupyter Discourse
|
||||
- Pull request reviews: Reading proposed documentation and code changes, working with the submitter
|
||||
to improve the contribution, deciding if the contribution should take another form (e.g., a recipe
|
||||
instead of a permanent change to the images)
|
||||
|
||||
Anyone in the community can jump in and help with these activities at any time. We will happily
|
||||
grant additional permissions (e.g., ability to merge PRs) to anyone who shows an on-going interest
|
||||
in working on the project.
|
||||
|
||||
## Jupyter Notebook Deprecation Notice
|
||||
|
||||
Following [Jupyter Notebook notice](https://github.com/jupyter/notebook#notice), we encourage users to transition to JupyterLab.
|
||||
This can be done by passing the environment variable `JUPYTER_ENABLE_LAB=yes` at container startup,
|
||||
more information is available in the [documentation](https://jupyter-docker-stacks.readthedocs.io/en/latest/using/common.html#docker-options).
|
||||
|
||||
At some point, JupyterLab will become the default for all of the Jupyter Docker stack images, however a new environment variable will be introduced to switch back to Jupyter Notebook if needed.
|
||||
|
||||
After the change of default, and according to the Jupyter Notebook project status and its compatibility with JupyterLab, these Docker images may remove the classic Jupyter Notebook interface altogether in favor of another _classic-like_ UI built atop JupyterLab.
|
||||
|
||||
This change is tracked in the issue [#1217](https://github.com/jupyter/docker-stacks/issues/1217), please check its content for more information.
|
||||
|
||||
## Quick Start
|
||||
|
||||
You can try a
|
||||
@@ -81,7 +49,7 @@ JupyterLab, where `hostname` is the name of the computer running docker and `tok
|
||||
token printed in the console. Docker destroys the container after notebook server exit, but any
|
||||
files written to `~/work` in the container remain intact on the host.
|
||||
|
||||
docker run --rm -p 10000:8888 -e JUPYTER_ENABLE_LAB=yes -v "$PWD":/home/jovyan/work jupyter/datascience-notebook:33add21fab64
|
||||
docker run --rm -p 10000:8888 -e JUPYTER_ENABLE_LAB=yes -v "${PWD}":/home/jovyan/work jupyter/datascience-notebook:33add21fab64
|
||||
|
||||
## Contributing
|
||||
|
||||
@@ -89,6 +57,38 @@ Please see the [Contributor Guide on ReadTheDocs](https://jupyter-docker-stacks.
|
||||
information about how to contribute package updates, recipes, features, tests, and community
|
||||
maintained stacks.
|
||||
|
||||
## Maintainer Help Wanted
|
||||
|
||||
We value all positive contributions to the Docker stacks project, from
|
||||
[bug reports](https://jupyter-docker-stacks.readthedocs.io/en/latest/contributing/issues.html) to
|
||||
[pull requests](https://jupyter-docker-stacks.readthedocs.io/en/latest/contributing/packages.html)
|
||||
to
|
||||
[translations](https://jupyter-docker-stacks.readthedocs.io/en/latest/contributing/translations.html)
|
||||
to help answering questions. We'd also like to invite members of the community to help with two
|
||||
maintainer activities:
|
||||
|
||||
- Issue triage: Reading and providing a first response to issues, labeling issues appropriately,
|
||||
redirecting cross-project questions to Jupyter Discourse
|
||||
- Pull request reviews: Reading proposed documentation and code changes, working with the submitter
|
||||
to improve the contribution, deciding if the contribution should take another form (e.g., a recipe
|
||||
instead of a permanent change to the images)
|
||||
|
||||
Anyone in the community can jump in and help with these activities at any time. We will happily
|
||||
grant additional permissions (e.g., ability to merge PRs) to anyone who shows an on-going interest
|
||||
in working on the project.
|
||||
|
||||
## Jupyter Notebook Deprecation Notice
|
||||
|
||||
Following [Jupyter Notebook notice](https://github.com/jupyter/notebook#notice), we encourage users to transition to JupyterLab.
|
||||
This can be done by passing the environment variable `JUPYTER_ENABLE_LAB=yes` at container startup,
|
||||
more information is available in the [documentation](https://jupyter-docker-stacks.readthedocs.io/en/latest/using/common.html#docker-options).
|
||||
|
||||
At some point, JupyterLab will become the default for all of the Jupyter Docker stack images, however a new environment variable will be introduced to switch back to Jupyter Notebook if needed.
|
||||
|
||||
After the change of default, and according to the Jupyter Notebook project status and its compatibility with JupyterLab, these Docker images may remove the classic Jupyter Notebook interface altogether in favor of another _classic-like_ UI built atop JupyterLab.
|
||||
|
||||
This change is tracked in the issue [#1217](https://github.com/jupyter/docker-stacks/issues/1217), please check its content for more information.
|
||||
|
||||
## Alternatives
|
||||
|
||||
- [jupyter/repo2docker](https://github.com/jupyter/repo2docker) - Turn git repositories into
|
||||
|
||||
@@ -19,7 +19,7 @@ RUN apt-get update --yes && \
|
||||
gcc && \
|
||||
apt-get clean && rm -rf /var/lib/apt/lists/*
|
||||
|
||||
USER $NB_UID
|
||||
USER ${NB_UID}
|
||||
|
||||
# R packages including IRKernel which gets installed globally.
|
||||
RUN conda install --quiet --yes \
|
||||
|
||||
@@ -60,14 +60,14 @@ RUN apt-get update --yes && \
|
||||
# Configure environment
|
||||
ENV CONDA_DIR=/opt/conda \
|
||||
SHELL=/bin/bash \
|
||||
NB_USER=$NB_USER \
|
||||
NB_UID=$NB_UID \
|
||||
NB_GID=$NB_GID \
|
||||
NB_USER="${NB_USER}" \
|
||||
NB_UID=${NB_UID} \
|
||||
NB_GID=${NB_GID} \
|
||||
LC_ALL=en_US.UTF-8 \
|
||||
LANG=en_US.UTF-8 \
|
||||
LANGUAGE=en_US.UTF-8
|
||||
ENV PATH=$CONDA_DIR/bin:$PATH \
|
||||
HOME=/home/$NB_USER \
|
||||
ENV PATH="${CONDA_DIR}/bin:${PATH}" \
|
||||
HOME="/home/${NB_USER}" \
|
||||
CONDA_VERSION="${conda_version}" \
|
||||
MINIFORGE_VERSION="${miniforge_version}"
|
||||
|
||||
@@ -86,18 +86,18 @@ RUN sed -i 's/^#force_color_prompt=yes/force_color_prompt=yes/' /etc/skel/.bashr
|
||||
RUN echo "auth requisite pam_deny.so" >> /etc/pam.d/su && \
|
||||
sed -i.bak -e 's/^%admin/#%admin/' /etc/sudoers && \
|
||||
sed -i.bak -e 's/^%sudo/#%sudo/' /etc/sudoers && \
|
||||
useradd -l -m -s /bin/bash -N -u $NB_UID $NB_USER && \
|
||||
mkdir -p $CONDA_DIR && \
|
||||
chown $NB_USER:$NB_GID $CONDA_DIR && \
|
||||
useradd -l -m -s /bin/bash -N -u "${NB_UID}" "${NB_USER}" && \
|
||||
mkdir -p "${CONDA_DIR}" && \
|
||||
chown "${NB_USER}:${NB_GID}" "${CONDA_DIR}" && \
|
||||
chmod g+w /etc/passwd && \
|
||||
fix-permissions "${HOME}" && \
|
||||
fix-permissions "${CONDA_DIR}"
|
||||
|
||||
USER $NB_UID
|
||||
USER ${NB_UID}
|
||||
ARG PYTHON_VERSION=default
|
||||
|
||||
# Setup work directory for backward-compatibility
|
||||
RUN mkdir "/home/$NB_USER/work" && \
|
||||
RUN mkdir "/home/${NB_USER}/work" && \
|
||||
fix-permissions "/home/${NB_USER}"
|
||||
|
||||
# Install conda as jovyan and check the sha256 sum provided on the download site
|
||||
@@ -106,20 +106,20 @@ WORKDIR /tmp
|
||||
# Prerequisites installation: conda, mamba, pip, tini
|
||||
RUN wget --quiet "https://github.com/conda-forge/miniforge/releases/download/${miniforge_version}/${miniforge_installer}" && \
|
||||
echo "${miniforge_checksum} *${miniforge_installer}" | sha256sum --check && \
|
||||
/bin/bash "${miniforge_installer}" -f -b -p $CONDA_DIR && \
|
||||
/bin/bash "${miniforge_installer}" -f -b -p "${CONDA_DIR}" && \
|
||||
rm "${miniforge_installer}" && \
|
||||
# Conda configuration see https://conda.io/projects/conda/en/latest/configuration.html
|
||||
echo "conda ${CONDA_VERSION}" >> $CONDA_DIR/conda-meta/pinned && \
|
||||
echo "conda ${CONDA_VERSION}" >> "${CONDA_DIR}/conda-meta/pinned" && \
|
||||
conda config --system --set auto_update_conda false && \
|
||||
conda config --system --set show_channel_urls true && \
|
||||
if [ ! $PYTHON_VERSION = 'default' ]; then conda install --yes python=$PYTHON_VERSION; fi && \
|
||||
conda list python | grep '^python ' | tr -s ' ' | cut -d ' ' -f 1,2 >> $CONDA_DIR/conda-meta/pinned && \
|
||||
if [[ "${PYTHON_VERSION}" != "default" ]]; then conda install --yes python="${PYTHON_VERSION}"; fi && \
|
||||
conda list python | grep '^python ' | tr -s ' ' | cut -d ' ' -f 1,2 >> "${CONDA_DIR}/conda-meta/pinned" && \
|
||||
conda install --quiet --yes \
|
||||
"conda=${CONDA_VERSION}" \
|
||||
'pip' && \
|
||||
conda update --all --quiet --yes && \
|
||||
conda clean --all -f -y && \
|
||||
rm -rf /home/$NB_USER/.cache/yarn && \
|
||||
rm -rf "/home/${NB_USER}/.cache/yarn" && \
|
||||
fix-permissions "${CONDA_DIR}" && \
|
||||
fix-permissions "/home/${NB_USER}"
|
||||
|
||||
@@ -137,7 +137,7 @@ RUN conda install --quiet --yes \
|
||||
npm cache clean --force && \
|
||||
jupyter notebook --generate-config && \
|
||||
jupyter lab clean && \
|
||||
rm -rf /home/$NB_USER/.cache/yarn && \
|
||||
rm -rf "/home/${NB_USER}/.cache/yarn" && \
|
||||
fix-permissions "${CONDA_DIR}" && \
|
||||
fix-permissions "/home/${NB_USER}"
|
||||
|
||||
@@ -161,6 +161,6 @@ RUN sed -re "s/c.NotebookApp/c.ServerApp/g" \
|
||||
fix-permissions /etc/jupyter/
|
||||
|
||||
# Switch back to jovyan to avoid accidental container runs as root
|
||||
USER $NB_UID
|
||||
USER ${NB_UID}
|
||||
|
||||
WORKDIR $HOME
|
||||
WORKDIR "${HOME}"
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
# set permissions on a directory
|
||||
# after any installation, if a directory needs to be (human) user-writable,
|
||||
# run this script on it.
|
||||
# It will make everything in the directory owned by the group $NB_GID
|
||||
# It will make everything in the directory owned by the group ${NB_GID}
|
||||
# and writable by that group.
|
||||
# Deployments that want to set a specific user id can preserve permissions
|
||||
# by adding the `--group-add users` line to `docker run`.
|
||||
@@ -11,14 +11,14 @@
|
||||
# which would cause massive image explosion
|
||||
|
||||
# right permissions are:
|
||||
# group=$NB_GID
|
||||
# group=${NB_GID}
|
||||
# AND permissions include group rwX (directory-execute)
|
||||
# AND directories have setuid,setgid bits set
|
||||
|
||||
set -e
|
||||
|
||||
for d in "$@"; do
|
||||
find "$d" \
|
||||
find "${d}" \
|
||||
! \( \
|
||||
-group "${NB_GID}" \
|
||||
-a -perm -g+rwX \
|
||||
@@ -26,7 +26,7 @@ for d in "$@"; do
|
||||
-exec chgrp "${NB_GID}" {} \; \
|
||||
-exec chmod g+rwX {} \;
|
||||
# setuid, setgid *on directories only*
|
||||
find "$d" \
|
||||
find "${d}" \
|
||||
\( \
|
||||
-type d \
|
||||
-a ! -perm -6000 \
|
||||
|
||||
@@ -14,9 +14,9 @@ if [[ -n "${JUPYTERHUB_API_TOKEN}" ]]; then
|
||||
exec /usr/local/bin/start-singleuser.sh "$@"
|
||||
elif [[ -n "${JUPYTER_ENABLE_LAB}" ]]; then
|
||||
# shellcheck disable=SC1091
|
||||
. /usr/local/bin/start.sh $wrapper jupyter lab "$@"
|
||||
. /usr/local/bin/start.sh ${wrapper} jupyter lab "$@"
|
||||
else
|
||||
echo "WARN: Jupyter Notebook deprecation notice https://github.com/jupyter/docker-stacks#jupyter-notebook-deprecation-notice."
|
||||
# shellcheck disable=SC1091
|
||||
. /usr/local/bin/start.sh $wrapper jupyter notebook "$@"
|
||||
. /usr/local/bin/start.sh ${wrapper} jupyter notebook "$@"
|
||||
fi
|
||||
|
||||
@@ -5,37 +5,37 @@
|
||||
set -e
|
||||
|
||||
# set default ip to 0.0.0.0
|
||||
if [[ "$NOTEBOOK_ARGS $*" != *"--ip="* ]]; then
|
||||
NOTEBOOK_ARGS="--ip=0.0.0.0 $NOTEBOOK_ARGS"
|
||||
if [[ "${NOTEBOOK_ARGS} $*" != *"--ip="* ]]; then
|
||||
NOTEBOOK_ARGS="--ip=0.0.0.0 ${NOTEBOOK_ARGS}"
|
||||
fi
|
||||
|
||||
# handle some deprecated environment variables
|
||||
# from DockerSpawner < 0.8.
|
||||
# These won't be passed from DockerSpawner 0.9,
|
||||
# so avoid specifying --arg=empty-string
|
||||
if [ -n "$NOTEBOOK_DIR" ]; then
|
||||
if [ -n "${NOTEBOOK_DIR}" ]; then
|
||||
# shellcheck disable=SC2089
|
||||
NOTEBOOK_ARGS="--notebook-dir='$NOTEBOOK_DIR' $NOTEBOOK_ARGS"
|
||||
NOTEBOOK_ARGS="--notebook-dir='${NOTEBOOK_DIR}' ${NOTEBOOK_ARGS}"
|
||||
fi
|
||||
if [ -n "$JPY_PORT" ]; then
|
||||
NOTEBOOK_ARGS="--port=$JPY_PORT $NOTEBOOK_ARGS"
|
||||
if [ -n "${JPY_PORT}" ]; then
|
||||
NOTEBOOK_ARGS="--port=${JPY_PORT} ${NOTEBOOK_ARGS}"
|
||||
fi
|
||||
if [ -n "$JPY_USER" ]; then
|
||||
NOTEBOOK_ARGS="--user=$JPY_USER $NOTEBOOK_ARGS"
|
||||
if [ -n "${JPY_USER}" ]; then
|
||||
NOTEBOOK_ARGS="--user=${JPY_USER} ${NOTEBOOK_ARGS}"
|
||||
fi
|
||||
if [ -n "$JPY_COOKIE_NAME" ]; then
|
||||
NOTEBOOK_ARGS="--cookie-name=$JPY_COOKIE_NAME $NOTEBOOK_ARGS"
|
||||
if [ -n "${JPY_COOKIE_NAME}" ]; then
|
||||
NOTEBOOK_ARGS="--cookie-name=${JPY_COOKIE_NAME} ${NOTEBOOK_ARGS}"
|
||||
fi
|
||||
if [ -n "$JPY_BASE_URL" ]; then
|
||||
NOTEBOOK_ARGS="--base-url=$JPY_BASE_URL $NOTEBOOK_ARGS"
|
||||
if [ -n "${JPY_BASE_URL}" ]; then
|
||||
NOTEBOOK_ARGS="--base-url=${JPY_BASE_URL} ${NOTEBOOK_ARGS}"
|
||||
fi
|
||||
if [ -n "$JPY_HUB_PREFIX" ]; then
|
||||
NOTEBOOK_ARGS="--hub-prefix=$JPY_HUB_PREFIX $NOTEBOOK_ARGS"
|
||||
if [ -n "${JPY_HUB_PREFIX}" ]; then
|
||||
NOTEBOOK_ARGS="--hub-prefix=${JPY_HUB_PREFIX} ${NOTEBOOK_ARGS}"
|
||||
fi
|
||||
if [ -n "$JPY_HUB_API_URL" ]; then
|
||||
NOTEBOOK_ARGS="--hub-api-url=$JPY_HUB_API_URL $NOTEBOOK_ARGS"
|
||||
if [ -n "${JPY_HUB_API_URL}" ]; then
|
||||
NOTEBOOK_ARGS="--hub-api-url=${JPY_HUB_API_URL} ${NOTEBOOK_ARGS}"
|
||||
fi
|
||||
NOTEBOOK_BIN="jupyterhub-singleuser"
|
||||
|
||||
# shellcheck disable=SC1091,SC2086,SC2090
|
||||
. /usr/local/bin/start.sh "$NOTEBOOK_BIN" $NOTEBOOK_ARGS "$@"
|
||||
. /usr/local/bin/start.sh "${NOTEBOOK_BIN}" ${NOTEBOOK_ARGS} "$@"
|
||||
|
||||
@@ -13,28 +13,28 @@ fi
|
||||
|
||||
run-hooks () {
|
||||
# Source scripts or run executable files in a directory
|
||||
if [[ ! -d "$1" ]] ; then
|
||||
if [[ ! -d "${1}" ]] ; then
|
||||
return
|
||||
fi
|
||||
echo "$0: running hooks in $1"
|
||||
for f in "$1/"*; do
|
||||
case "$f" in
|
||||
echo "${0}: running hooks in ${1}"
|
||||
for f in "${1}/"*; do
|
||||
case "${f}" in
|
||||
*.sh)
|
||||
echo "$0: running $f"
|
||||
echo "${0}: running ${f}"
|
||||
# shellcheck disable=SC1090
|
||||
source "$f"
|
||||
source "${f}"
|
||||
;;
|
||||
*)
|
||||
if [[ -x "$f" ]] ; then
|
||||
echo "$0: running $f"
|
||||
"$f"
|
||||
if [[ -x "${f}" ]] ; then
|
||||
echo "${0}: running ${f}"
|
||||
"${f}"
|
||||
else
|
||||
echo "$0: ignoring $f"
|
||||
echo "${0}: ignoring ${f}"
|
||||
fi
|
||||
;;
|
||||
esac
|
||||
done
|
||||
echo "$0: done running hooks in $1"
|
||||
echo "${0}: done running hooks in ${1}"
|
||||
}
|
||||
|
||||
run-hooks /usr/local/bin/start-notebook.d
|
||||
@@ -44,73 +44,73 @@ if [ "$(id -u)" == 0 ] ; then
|
||||
|
||||
# Only attempt to change the jovyan username if it exists
|
||||
if id jovyan &> /dev/null ; then
|
||||
echo "Set username to: $NB_USER"
|
||||
usermod -d "/home/$NB_USER" -l "$NB_USER" jovyan
|
||||
echo "Set username to: ${NB_USER}"
|
||||
usermod -d "/home/${NB_USER}" -l "${NB_USER}" jovyan
|
||||
fi
|
||||
|
||||
# handle home and working directory if the username changed
|
||||
if [[ "$NB_USER" != "jovyan" ]]; then
|
||||
if [[ "${NB_USER}" != "jovyan" ]]; then
|
||||
# changing username, make sure homedir exists
|
||||
# (it could be mounted, and we shouldn't create it if it already exists)
|
||||
if [[ ! -e "/home/$NB_USER" ]]; then
|
||||
echo "Relocating home dir to /home/$NB_USER"
|
||||
mv /home/jovyan "/home/$NB_USER" || ln -s /home/jovyan "/home/$NB_USER"
|
||||
if [[ ! -e "/home/${NB_USER}" ]]; then
|
||||
echo "Relocating home dir to /home/${NB_USER}"
|
||||
mv /home/jovyan "/home/${NB_USER}" || ln -s /home/jovyan "/home/${NB_USER}"
|
||||
fi
|
||||
# if workdir is in /home/jovyan, cd to /home/$NB_USER
|
||||
if [[ "$PWD/" == "/home/jovyan/"* ]]; then
|
||||
newcwd="/home/$NB_USER/${PWD:13}"
|
||||
echo "Setting CWD to $newcwd"
|
||||
cd "$newcwd"
|
||||
# if workdir is in /home/jovyan, cd to /home/${NB_USER}
|
||||
if [[ "${PWD}/" == "/home/jovyan/"* ]]; then
|
||||
newcwd="/home/${NB_USER}/${PWD:13}"
|
||||
echo "Setting CWD to ${newcwd}"
|
||||
cd "${newcwd}"
|
||||
fi
|
||||
fi
|
||||
|
||||
# Handle case where provisioned storage does not have the correct permissions by default
|
||||
# Ex: default NFS/EFS (no auto-uid/gid)
|
||||
if [[ "$CHOWN_HOME" == "1" || "$CHOWN_HOME" == 'yes' ]]; then
|
||||
echo "Changing ownership of /home/$NB_USER to $NB_UID:$NB_GID with options '${CHOWN_HOME_OPTS}'"
|
||||
if [[ "${CHOWN_HOME}" == "1" || "${CHOWN_HOME}" == 'yes' ]]; then
|
||||
echo "Changing ownership of /home/${NB_USER} to ${NB_UID}:${NB_GID} with options '${CHOWN_HOME_OPTS}'"
|
||||
# shellcheck disable=SC2086
|
||||
chown $CHOWN_HOME_OPTS "$NB_UID:$NB_GID" "/home/$NB_USER"
|
||||
chown ${CHOWN_HOME_OPTS} "${NB_UID}:${NB_GID}" "/home/${NB_USER}"
|
||||
fi
|
||||
if [ -n "$CHOWN_EXTRA" ]; then
|
||||
for extra_dir in $(echo "$CHOWN_EXTRA" | tr ',' ' '); do
|
||||
echo "Changing ownership of ${extra_dir} to $NB_UID:$NB_GID with options '${CHOWN_EXTRA_OPTS}'"
|
||||
if [ -n "${CHOWN_EXTRA}" ]; then
|
||||
for extra_dir in $(echo "${CHOWN_EXTRA}" | tr ',' ' '); do
|
||||
echo "Changing ownership of ${extra_dir} to ${NB_UID}:${NB_GID} with options '${CHOWN_EXTRA_OPTS}'"
|
||||
# shellcheck disable=SC2086
|
||||
chown $CHOWN_EXTRA_OPTS "$NB_UID:$NB_GID" "$extra_dir"
|
||||
chown ${CHOWN_EXTRA_OPTS} "${NB_UID}:${NB_GID}" "${extra_dir}"
|
||||
done
|
||||
fi
|
||||
|
||||
# Change UID:GID of NB_USER to NB_UID:NB_GID if it does not match
|
||||
if [ "$NB_UID" != "$(id -u "$NB_USER")" ] || [ "$NB_GID" != "$(id -g "$NB_USER")" ]; then
|
||||
echo "Set user $NB_USER UID:GID to: $NB_UID:$NB_GID"
|
||||
if [ "$NB_GID" != "$(id -g "$NB_USER")" ]; then
|
||||
groupadd -f -g "$NB_GID" -o "${NB_GROUP:-${NB_USER}}"
|
||||
if [ "${NB_UID}" != "$(id -u "${NB_USER}")" ] || [ "${NB_GID}" != "$(id -g "${NB_USER}")" ]; then
|
||||
echo "Set user ${NB_USER} UID:GID to: ${NB_UID}:${NB_GID}"
|
||||
if [ "${NB_GID}" != "$(id -g "${NB_USER}")" ]; then
|
||||
groupadd -f -g "${NB_GID}" -o "${NB_GROUP:-${NB_USER}}"
|
||||
fi
|
||||
userdel "$NB_USER"
|
||||
useradd --home "/home/$NB_USER" -u "$NB_UID" -g "$NB_GID" -G 100 -l "$NB_USER"
|
||||
userdel "${NB_USER}"
|
||||
useradd --home "/home/${NB_USER}" -u "${NB_UID}" -g "${NB_GID}" -G 100 -l "${NB_USER}"
|
||||
fi
|
||||
|
||||
# Enable sudo if requested
|
||||
if [[ "$GRANT_SUDO" == "1" || "$GRANT_SUDO" == 'yes' ]]; then
|
||||
echo "Granting $NB_USER sudo access and appending $CONDA_DIR/bin to sudo PATH"
|
||||
echo "$NB_USER ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/notebook
|
||||
if [[ "${GRANT_SUDO}" == "1" || "${GRANT_SUDO}" == 'yes' ]]; then
|
||||
echo "Granting ${NB_USER} sudo access and appending ${CONDA_DIR}/bin to sudo PATH"
|
||||
echo "${NB_USER} ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/notebook
|
||||
fi
|
||||
|
||||
# Add $CONDA_DIR/bin to sudo secure_path
|
||||
sed -r "s#Defaults\s+secure_path\s*=\s*\"?([^\"]+)\"?#Defaults secure_path=\"\1:$CONDA_DIR/bin\"#" /etc/sudoers | grep secure_path > /etc/sudoers.d/path
|
||||
# Add ${CONDA_DIR}/bin to sudo secure_path
|
||||
sed -r "s#Defaults\s+secure_path\s*=\s*\"?([^\"]+)\"?#Defaults secure_path=\"\1:${CONDA_DIR}/bin\"#" /etc/sudoers | grep secure_path > /etc/sudoers.d/path
|
||||
|
||||
# Exec the command as NB_USER with the PATH and the rest of
|
||||
# the environment preserved
|
||||
run-hooks /usr/local/bin/before-notebook.d
|
||||
echo "Executing the command:" "${cmd[@]}"
|
||||
exec sudo -E -H -u "$NB_USER" PATH="$PATH" XDG_CACHE_HOME="/home/$NB_USER/.cache" PYTHONPATH="${PYTHONPATH:-}" "${cmd[@]}"
|
||||
exec sudo -E -H -u "${NB_USER}" PATH="${PATH}" XDG_CACHE_HOME="/home/${NB_USER}/.cache" PYTHONPATH="${PYTHONPATH:-}" "${cmd[@]}"
|
||||
else
|
||||
if [[ "$NB_UID" == "$(id -u jovyan 2>/dev/null)" && "$NB_GID" == "$(id -g jovyan 2>/dev/null)" ]]; then
|
||||
if [[ "${NB_UID}" == "$(id -u jovyan 2>/dev/null)" && "${NB_GID}" == "$(id -g jovyan 2>/dev/null)" ]]; then
|
||||
# User is not attempting to override user/group via environment
|
||||
# variables, but they could still have overridden the uid/gid that
|
||||
# container runs as. Check that the user has an entry in the passwd
|
||||
# file and if not add an entry.
|
||||
STATUS=0 && whoami &> /dev/null || STATUS=$? && true
|
||||
if [[ "$STATUS" != "0" ]]; then
|
||||
if [[ "${STATUS}" != "0" ]]; then
|
||||
if [[ -w /etc/passwd ]]; then
|
||||
echo "Adding passwd file entry for $(id -u)"
|
||||
sed -e "s/^jovyan:/nayvoj:/" /etc/passwd > /tmp/passwd
|
||||
@@ -122,24 +122,24 @@ else
|
||||
fi
|
||||
fi
|
||||
|
||||
# Warn if the user isn't going to be able to write files to $HOME.
|
||||
# Warn if the user isn't going to be able to write files to ${HOME}.
|
||||
if [[ ! -w /home/jovyan ]]; then
|
||||
echo 'Container must be run with group "users" to update files'
|
||||
fi
|
||||
else
|
||||
# Warn if looks like user want to override uid/gid but hasn't
|
||||
# run the container as root.
|
||||
if [[ -n "$NB_UID" && "$NB_UID" != "$(id -u)" ]]; then
|
||||
echo "Container must be run as root to set NB_UID to $NB_UID"
|
||||
if [[ -n "${NB_UID}" && "${NB_UID}" != "$(id -u)" ]]; then
|
||||
echo "Container must be run as root to set NB_UID to ${NB_UID}"
|
||||
fi
|
||||
if [[ -n "$NB_GID" && "$NB_GID" != "$(id -g)" ]]; then
|
||||
echo "Container must be run as root to set NB_GID to $NB_GID"
|
||||
if [[ -n "${NB_GID}" && "${NB_GID}" != "$(id -g)" ]]; then
|
||||
echo "Container must be run as root to set NB_GID to ${NB_GID}"
|
||||
fi
|
||||
fi
|
||||
|
||||
# Warn if looks like user want to run in sudo mode but hasn't run
|
||||
# the container as root.
|
||||
if [[ "$GRANT_SUDO" == "1" || "$GRANT_SUDO" == 'yes' ]]; then
|
||||
if [[ "${GRANT_SUDO}" == "1" || "${GRANT_SUDO}" == 'yes' ]]; then
|
||||
echo 'Container must be run as root to grant sudo permissions'
|
||||
fi
|
||||
|
||||
|
||||
@@ -8,7 +8,7 @@ FROM $BASE_CONTAINER
|
||||
LABEL maintainer="Jupyter Project <jupyter@googlegroups.com>"
|
||||
ENV TAG="33add21fab64"
|
||||
|
||||
WORKDIR $HOME
|
||||
WORKDIR "${HOME}"
|
||||
COPY binder/README.ipynb .
|
||||
|
||||
# Fix permissions on README.ipynb as root
|
||||
@@ -17,6 +17,6 @@ USER root
|
||||
RUN fix-permissions README.ipynb
|
||||
|
||||
# Switch back to jovyan to avoid accidental container runs as root
|
||||
USER $NB_UID
|
||||
USER ${NB_UID}
|
||||
|
||||
WORKDIR $HOME
|
||||
WORKDIR "${HOME}"
|
||||
|
||||
@@ -27,7 +27,7 @@ RUN apt-get update --yes && \
|
||||
apt-get clean && rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Julia dependencies
|
||||
# install Julia packages in /opt/julia instead of $HOME
|
||||
# install Julia packages in /opt/julia instead of ${HOME}
|
||||
ENV JULIA_DEPOT_PATH=/opt/julia \
|
||||
JULIA_PKGDIR=/opt/julia \
|
||||
JULIA_VERSION="${julia_version}"
|
||||
@@ -44,13 +44,13 @@ RUN mkdir "/opt/julia-${JULIA_VERSION}" && \
|
||||
|
||||
# Show Julia where conda libraries are \
|
||||
RUN mkdir /etc/julia && \
|
||||
echo "push!(Libdl.DL_LOAD_PATH, \"$CONDA_DIR/lib\")" >> /etc/julia/juliarc.jl && \
|
||||
echo "push!(Libdl.DL_LOAD_PATH, \"${CONDA_DIR}/lib\")" >> /etc/julia/juliarc.jl && \
|
||||
# Create JULIA_PKGDIR \
|
||||
mkdir "${JULIA_PKGDIR}" && \
|
||||
chown "${NB_USER}" "${JULIA_PKGDIR}" && \
|
||||
fix-permissions "${JULIA_PKGDIR}"
|
||||
|
||||
USER $NB_UID
|
||||
USER ${NB_UID}
|
||||
|
||||
# R packages including IRKernel which gets installed globally.
|
||||
RUN conda install --quiet --yes \
|
||||
@@ -91,4 +91,4 @@ RUN julia -e 'import Pkg; Pkg.update()' && \
|
||||
rm -rf "${HOME}/.local" && \
|
||||
fix-permissions "${JULIA_PKGDIR}" "${CONDA_DIR}/share/jupyter"
|
||||
|
||||
WORKDIR $HOME
|
||||
WORKDIR "${HOME}"
|
||||
|
||||
@@ -24,7 +24,7 @@ The other pages in this documentation describe additional uses and features in d
|
||||
|
||||
**Example 3:** This command pulls the ``jupyter/datascience-notebook`` image tagged ``33add21fab64`` from Docker Hub if it is not already present on the local host. It then starts an *ephemeral* container running a Jupyter Notebook server and exposes the server on host port 10000. The command mounts the current working directory on the host as ``/home/jovyan/work`` in the container. The server logs appear in the terminal. Visiting ``http://<hostname>:10000/lab?token=<token>`` in a browser loads JupyterLab, where ``hostname`` is the name of the computer running docker and ``token`` is the secret token printed in the console. Docker destroys the container after notebook server exit, but any files written to ``~/work`` in the container remain intact on the host.::
|
||||
|
||||
docker run --rm -p 10000:8888 -e JUPYTER_ENABLE_LAB=yes -v "$PWD":/home/jovyan/work jupyter/datascience-notebook:33add21fab64
|
||||
docker run --rm -p 10000:8888 -e JUPYTER_ENABLE_LAB=yes -v "${PWD}":/home/jovyan/work jupyter/datascience-notebook:33add21fab64
|
||||
|
||||
Table of Contents
|
||||
-----------------
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -23,19 +23,19 @@ docker run -d -p 8888:8888 jupyter/base-notebook start-notebook.sh --NotebookApp
|
||||
You may instruct the `start-notebook.sh` script to customize the container environment before launching
|
||||
the notebook server. You do so by passing arguments to the `docker run` command.
|
||||
|
||||
- `-e NB_USER=jovyan` - Instructs the startup script to change the default container username from `jovyan` to the provided value. Causes the script to rename the `jovyan` user home folder. For this option to take effect, you must run the container with `--user root`, set the working directory `-w /home/$NB_USER` and set the environment variable `-e CHOWN_HOME=yes` (see below for detail). This feature is useful when mounting host volumes with specific home folder.
|
||||
- `-e NB_UID=1000` - Instructs the startup script to switch the numeric user ID of `$NB_USER` to the given value. This feature is useful when mounting host volumes with specific owner permissions. For this option to take effect, you must run the container with `--user root`. (The startup script will `su $NB_USER` after adjusting the user ID.) You might consider using modern Docker options `--user` and `--group-add` instead. See the last bullet below for details.
|
||||
- `-e NB_GID=100` - Instructs the startup script to change the primary group of`$NB_USER` to `$NB_GID` (the new group is added with a name of `$NB_GROUP` if it is defined, otherwise the group is named `$NB_USER`). This feature is useful when mounting host volumes with specific group permissions. For this option to take effect, you must run the container with `--user root`. (The startup script will `su $NB_USER` after adjusting the group ID.) You might consider using modern Docker options `--user` and `--group-add` instead. See the last bullet below for details. The user is added to supplemental group `users` (gid 100) in order to allow write access to the home directory and `/opt/conda`. If you override the user/group logic, ensure the user stays in group `users` if you want them to be able to modify files in the image.
|
||||
- `-e NB_GROUP=<name>` - The name used for `$NB_GID`, which defaults to `$NB_USER`. This is only used if `$NB_GID` is specified and completely optional: there is only cosmetic effect.
|
||||
- `-e NB_USER=jovyan` - Instructs the startup script to change the default container username from `jovyan` to the provided value. Causes the script to rename the `jovyan` user home folder. For this option to take effect, you must run the container with `--user root`, set the working directory `-w /home/${NB_USER}` and set the environment variable `-e CHOWN_HOME=yes` (see below for detail). This feature is useful when mounting host volumes with specific home folder.
|
||||
- `-e NB_UID=1000` - Instructs the startup script to switch the numeric user ID of `${NB_USER}` to the given value. This feature is useful when mounting host volumes with specific owner permissions. For this option to take effect, you must run the container with `--user root`. (The startup script will `su ${NB_USER}` after adjusting the user ID.) You might consider using modern Docker options `--user` and `--group-add` instead. See the last bullet below for details.
|
||||
- `-e NB_GID=100` - Instructs the startup script to change the primary group of`${NB_USER}` to `${NB_GID}` (the new group is added with a name of `${NB_GROUP}` if it is defined, otherwise the group is named `${NB_USER}`). This feature is useful when mounting host volumes with specific group permissions. For this option to take effect, you must run the container with `--user root`. (The startup script will `su ${NB_USER}` after adjusting the group ID.) You might consider using modern Docker options `--user` and `--group-add` instead. See the last bullet below for details. The user is added to supplemental group `users` (gid 100) in order to allow write access to the home directory and `/opt/conda`. If you override the user/group logic, ensure the user stays in group `users` if you want them to be able to modify files in the image.
|
||||
- `-e NB_GROUP=<name>` - The name used for `${NB_GID}`, which defaults to `${NB_USER}`. This is only used if `${NB_GID}` is specified and completely optional: there is only cosmetic effect.
|
||||
- `-e NB_UMASK=<umask>` - Configures Jupyter to use a different umask value from default, i.e. `022`. For example, if setting umask to `002`, new files will be readable and writable by group members instead of just writable by the owner. Wikipedia has a good article about [umask](https://en.wikipedia.org/wiki/Umask). Feel free to read it in order to choose the value that better fits your needs. Default value should fit most situations. Note that `NB_UMASK` when set only applies to the Jupyter process itself - you cannot use it to set a umask for additional files created during run-hooks e.g. via `pip` or `conda` - if you need to set a umask for these you must set `umask` for each command.
|
||||
- `-e CHOWN_HOME=yes` - Instructs the startup script to change the `$NB_USER` home directory owner and group to the current value of `$NB_UID` and `$NB_GID`. This change will take effect even if the user home directory is mounted from the host using `-v` as described below. The change is **not** applied recursively by default. You can change modify the `chown` behavior by setting `CHOWN_HOME_OPTS` (e.g., `-e CHOWN_HOME_OPTS='-R'`).
|
||||
- `-e CHOWN_EXTRA="<some dir>,<some other dir>"` - Instructs the startup script to change the owner and group of each comma-separated container directory to the current value of `$NB_UID` and `$NB_GID`. The change is **not** applied recursively by default. You can change modify the `chown` behavior by setting `CHOWN_EXTRA_OPTS` (e.g., `-e CHOWN_EXTRA_OPTS='-R'`).
|
||||
- `-e GRANT_SUDO=yes` - Instructs the startup script to grant the `NB_USER` user passwordless `sudo` capability. You do **not** need this option to allow the user to `conda` or `pip` install additional packages. This option is useful, however, when you wish to give `$NB_USER` the ability to install OS packages with `apt` or modify other root-owned files in the container. For this option to take effect, you must run the container with `--user root`. (The `start-notebook.sh` script will `su $NB_USER` after adding `$NB_USER` to sudoers.) **You should only enable `sudo` if you trust the user or if the container is running on an isolated host.**
|
||||
- `-e CHOWN_HOME=yes` - Instructs the startup script to change the `${NB_USER}` home directory owner and group to the current value of `${NB_UID}` and `${NB_GID}`. This change will take effect even if the user home directory is mounted from the host using `-v` as described below. The change is **not** applied recursively by default. You can change modify the `chown` behavior by setting `CHOWN_HOME_OPTS` (e.g., `-e CHOWN_HOME_OPTS='-R'`).
|
||||
- `-e CHOWN_EXTRA="<some dir>,<some other dir>"` - Instructs the startup script to change the owner and group of each comma-separated container directory to the current value of `${NB_UID}` and `${NB_GID}`. The change is **not** applied recursively by default. You can change modify the `chown` behavior by setting `CHOWN_EXTRA_OPTS` (e.g., `-e CHOWN_EXTRA_OPTS='-R'`).
|
||||
- `-e GRANT_SUDO=yes` - Instructs the startup script to grant the `NB_USER` user passwordless `sudo` capability. You do **not** need this option to allow the user to `conda` or `pip` install additional packages. This option is useful, however, when you wish to give `${NB_USER}` the ability to install OS packages with `apt` or modify other root-owned files in the container. For this option to take effect, you must run the container with `--user root`. (The `start-notebook.sh` script will `su ${NB_USER}` after adding `${NB_USER}` to sudoers.) **You should only enable `sudo` if you trust the user or if the container is running on an isolated host.**
|
||||
- `-e GEN_CERT=yes` - Instructs the startup script to generates a self-signed SSL certificate and configure Jupyter Notebook to use it to accept encrypted HTTPS connections.
|
||||
- `-e JUPYTER_ENABLE_LAB=yes` - Instructs the startup script to run `jupyter lab` instead of the default `jupyter notebook` command. Useful in container orchestration environments where setting environment variables is easier than change command line parameters.
|
||||
- `-e RESTARTABLE=yes` - Runs Jupyter in a loop so that quitting Jupyter does not cause the container to exit. This may be useful when you need to install extensions that require restarting Jupyter.
|
||||
- `-v /some/host/folder/for/work:/home/jovyan/work` - Mounts a host machine directory as folder in the container. Useful when you want to preserve notebooks and other work even after the container is destroyed. **You must grant the within-container notebook user or group (`NB_UID` or `NB_GID`) write access to the host directory (e.g., `sudo chown 1000 /some/host/folder/for/work`).**
|
||||
- `--user 5000 --group-add users` - Launches the container with a specific user ID and adds that user to the `users` group so that it can modify files in the default home directory and `/opt/conda`. You can use these arguments as alternatives to setting `$NB_UID` and `$NB_GID`.
|
||||
- `--user 5000 --group-add users` - Launches the container with a specific user ID and adds that user to the `users` group so that it can modify files in the default home directory and `/opt/conda`. You can use these arguments as alternatives to setting `${NB_UID}` and `${NB_GID}`.
|
||||
|
||||
## Startup Hooks
|
||||
|
||||
@@ -103,7 +103,7 @@ You can bypass the provided scripts and specify an arbitrary start command. If y
|
||||
|
||||
## Conda Environments
|
||||
|
||||
The default Python 3.x [Conda environment](https://conda.io/projects/conda/en/latest/user-guide/concepts/environments.html) resides in `/opt/conda`. The `/opt/conda/bin` directory is part of the default `jovyan` user's `$PATH`. That directory is also whitelisted for use in `sudo` commands by the `start.sh` script.
|
||||
The default Python 3.x [Conda environment](https://conda.io/projects/conda/en/latest/user-guide/concepts/environments.html) resides in `/opt/conda`. The `/opt/conda/bin` directory is part of the default `jovyan` user's `${PATH}`. That directory is also whitelisted for use in `sudo` commands by the `start.sh` script.
|
||||
|
||||
The `jovyan` user has full read/write access to the `/opt/conda` directory. You can use either `pip`, `conda` or `mamba` to install new packages without any additional permissions.
|
||||
|
||||
|
||||
@@ -84,17 +84,17 @@ FROM jupyter/scipy-notebook:latest
|
||||
# Create a Python 2.x environment using conda including at least the ipython kernel
|
||||
# and the kernda utility. Add any additional packages you want available for use
|
||||
# in a Python 2 notebook to the first line here (e.g., pandas, matplotlib, etc.)
|
||||
RUN conda create --quiet --yes -p $CONDA_DIR/envs/python2 python=2.7 ipython ipykernel kernda && \
|
||||
RUN conda create --quiet --yes -p "${CONDA_DIR}/envs/python2" python=2.7 ipython ipykernel kernda && \
|
||||
conda clean --all -f -y
|
||||
|
||||
USER root
|
||||
|
||||
# Create a global kernelspec in the image and modify it so that it properly activates
|
||||
# the python2 conda environment.
|
||||
RUN $CONDA_DIR/envs/python2/bin/python -m ipykernel install && \
|
||||
$CONDA_DIR/envs/python2/bin/kernda -o -y /usr/local/share/jupyter/kernels/python2/kernel.json
|
||||
RUN "${CONDA_DIR}/envs/python2/bin/python" -m ipykernel install && \
|
||||
"${CONDA_DIR}/envs/python2/bin/kernda" -o -y /usr/local/share/jupyter/kernels/python2/kernel.json
|
||||
|
||||
USER $NB_USER
|
||||
USER ${NB_UID}
|
||||
```
|
||||
|
||||
Ref: <https://github.com/jupyter/docker-stacks/issues/440>
|
||||
@@ -113,28 +113,28 @@ ARG conda_env=python36
|
||||
ARG py_ver=3.6
|
||||
|
||||
# you can add additional libraries you want conda to install by listing them below the first line and ending with "&& \"
|
||||
RUN conda create --quiet --yes -p $CONDA_DIR/envs/$conda_env python=$py_ver ipython ipykernel && \
|
||||
RUN conda create --quiet --yes -p "${CONDA_DIR}/envs/${conda_env}" python=${py_ver} ipython ipykernel && \
|
||||
conda clean --all -f -y
|
||||
|
||||
# alternatively, you can comment out the lines above and uncomment those below
|
||||
# if you'd prefer to use a YAML file present in the docker build context
|
||||
|
||||
# COPY --chown=${NB_UID}:${NB_GID} environment.yml /home/$NB_USER/tmp/
|
||||
# RUN cd /home/$NB_USER/tmp/ && \
|
||||
# conda env create -p $CONDA_DIR/envs/$conda_env -f environment.yml && \
|
||||
# COPY --chown=${NB_UID}:${NB_GID} environment.yml "/home/${NB_USER}/tmp/"
|
||||
# RUN cd "/home/${NB_USER}/tmp/" && \
|
||||
# conda env create -p "${CONDA_DIR}/envs/${conda_env}" -f environment.yml && \
|
||||
# conda clean --all -f -y
|
||||
|
||||
|
||||
# create Python 3.x environment and link it to jupyter
|
||||
RUN $CONDA_DIR/envs/${conda_env}/bin/python -m ipykernel install --user --name=${conda_env} && \
|
||||
RUN "${CONDA_DIR}/envs/${conda_env}/bin/python" -m ipykernel install --user --name="${conda_env}" && \
|
||||
fix-permissions "${CONDA_DIR}" && \
|
||||
fix-permissions "/home/${NB_USER}"
|
||||
|
||||
# any additional pip installs can be added by uncommenting the following line
|
||||
# RUN $CONDA_DIR/envs/${conda_env}/bin/pip install
|
||||
# RUN "${CONDA_DIR}/envs/${conda_env}/bin/pip" install
|
||||
|
||||
# prepend conda environment to path
|
||||
ENV PATH $CONDA_DIR/envs/${conda_env}/bin:$PATH
|
||||
ENV PATH "${CONDA_DIR}/envs/${conda_env}/bin:${PATH}"
|
||||
|
||||
# if you want this environment to be the default one, uncomment the following line:
|
||||
# ENV CONDA_DEFAULT_ENV ${conda_env}
|
||||
@@ -266,7 +266,7 @@ RUN rm /etc/dpkg/dpkg.cfg.d/excludes && \
|
||||
dpkg -l | grep ^ii | cut -d' ' -f3 | xargs apt-get install --yes --no-install-recommends --reinstall man && \
|
||||
apt-get clean && rm -rf /var/lib/apt/lists/*
|
||||
|
||||
USER $NB_UID
|
||||
USER ${NB_UID}
|
||||
```
|
||||
|
||||
Adding the documentation on top of an existing singleuser image wastes a lot of space and requires
|
||||
@@ -428,7 +428,7 @@ RUN echo 'deb https://cdn-fastly.deb.debian.org/debian jessie-backports main' >
|
||||
# Add hadoop binaries
|
||||
wget https://mirrors.ukfast.co.uk/sites/ftp.apache.org/hadoop/common/hadoop-2.7.3/hadoop-2.7.3.tar.gz && \
|
||||
tar -xvf hadoop-2.7.3.tar.gz -C /usr/local && \
|
||||
chown -R $NB_USER:users /usr/local/hadoop-2.7.3 && \
|
||||
chown -R "${NB_USER}:users" /usr/local/hadoop-2.7.3 && \
|
||||
rm -f hadoop-2.7.3.tar.gz && \
|
||||
# Install os dependencies required for pydoop, pyhive
|
||||
apt-get update --yes && \
|
||||
@@ -447,13 +447,13 @@ RUN echo "spark.driver.extraJavaOptions -Dhdp.version=2.5.3.0-37" >> /usr/local/
|
||||
echo "spark.yarn.am.extraJavaOptions -Dhdp.version=2.5.3.0-37" >> /usr/local/spark/conf/spark-defaults.conf && \
|
||||
echo "spark.master=yarn" >> /usr/local/spark/conf/spark-defaults.conf && \
|
||||
echo "spark.hadoop.yarn.timeline-service.enabled=false" >> /usr/local/spark/conf/spark-defaults.conf && \
|
||||
chown -R $NB_USER:users /usr/local/spark/conf/spark-defaults.conf && \
|
||||
chown -R "${NB_USER}:users" /usr/local/spark/conf/spark-defaults.conf && \
|
||||
# Create an alternative HADOOP_CONF_HOME so we can mount as a volume and repoint
|
||||
# using ENV var if needed
|
||||
mkdir -p /etc/hadoop/conf/ && \
|
||||
chown $NB_USER:users /etc/hadoop/conf/
|
||||
chown "${NB_USER}":users /etc/hadoop/conf/
|
||||
|
||||
USER $NB_USER
|
||||
USER ${NB_UID}
|
||||
|
||||
# Install useful jupyter extensions and python libraries like :
|
||||
# - Dashboards
|
||||
@@ -468,7 +468,7 @@ RUN pip install --quiet --no-cache-dir jupyter_dashboards faker && \
|
||||
USER root
|
||||
# Ensure we overwrite the kernel config so that toree connects to cluster
|
||||
RUN jupyter toree install --sys-prefix --spark_opts="--master yarn --deploy-mode client --driver-memory 512m --executor-memory 512m --executor-cores 1 --driver-java-options -Dhdp.version=2.5.3.0-37 --conf spark.hadoop.yarn.timeline-service.enabled=false"
|
||||
USER $NB_USER
|
||||
USER ${NB_UID}
|
||||
```
|
||||
|
||||
Credit: [britishbadger](https://github.com/britishbadger) from
|
||||
@@ -503,7 +503,7 @@ NB: this works for classic notebooks only
|
||||
# Update with your base image of choice
|
||||
FROM jupyter/minimal-notebook:latest
|
||||
|
||||
USER $NB_USER
|
||||
USER ${NB_UID}
|
||||
|
||||
RUN pip install --quiet --no-cache-dir jupyter_contrib_nbextensions && \
|
||||
jupyter contrib nbextension install --user && \
|
||||
@@ -530,7 +530,7 @@ RUN apt-get update --yes && \
|
||||
apt-get install --yes --no-install-recommends swig && \
|
||||
apt-get clean && rm -rf /var/lib/apt/lists/*
|
||||
|
||||
USER $NB_UID
|
||||
USER ${NB_UID}
|
||||
|
||||
RUN pip install --quiet --no-cache-dir auto-sklearn && \
|
||||
fix-permissions "${CONDA_DIR}" && \
|
||||
@@ -551,10 +551,10 @@ RUN pip install --quiet --no-cache-dir delta-spark==${DELTA_CORE_VERSION} && \
|
||||
|
||||
USER root
|
||||
|
||||
RUN echo 'spark.sql.extensions io.delta.sql.DeltaSparkSessionExtension' >> $SPARK_HOME/conf/spark-defaults.conf && \
|
||||
echo 'spark.sql.catalog.spark_catalog org.apache.spark.sql.delta.catalog.DeltaCatalog' >> $SPARK_HOME/conf/spark-defaults.conf
|
||||
RUN echo 'spark.sql.extensions io.delta.sql.DeltaSparkSessionExtension' >> "${SPARK_HOME}/conf/spark-defaults.conf" && \
|
||||
echo 'spark.sql.catalog.spark_catalog org.apache.spark.sql.delta.catalog.DeltaCatalog' >> "${SPARK_HOME}/conf/spark-defaults.conf"
|
||||
|
||||
USER $NB_UID
|
||||
USER ${NB_UID}
|
||||
|
||||
# Trigger download of delta lake files
|
||||
RUN echo "from pyspark.sql import SparkSession" > /tmp/init-delta.py && \
|
||||
|
||||
@@ -55,7 +55,7 @@ d67fe77f1a84
|
||||
**Example 2** This command pulls the `jupyter/r-notebook` image tagged `33add21fab64` from Docker Hub if it is not already present on the local host. It then starts a container running a Jupyter Notebook server and exposes the server on host port 10000. The server logs appear in the terminal and include a URL to the notebook server, but with the internal container port (8888) instead of the the correct host port (10000).
|
||||
|
||||
```bash
|
||||
$ docker run --rm -p 10000:8888 -v "$PWD":/home/jovyan/work jupyter/r-notebook:33add21fab64
|
||||
$ docker run --rm -p 10000:8888 -v "${PWD}":/home/jovyan/work jupyter/r-notebook:33add21fab64
|
||||
|
||||
Executing the command: jupyter notebook
|
||||
[I 19:31:09.573 NotebookApp] Writing notebook server cookie secret to /home/jovyan/.local/share/jupyter/runtime/notebook_cookie_secret
|
||||
|
||||
@@ -212,6 +212,39 @@ rdd.sum()
|
||||
// 5050
|
||||
```
|
||||
|
||||
### Define Spark Dependencies
|
||||
|
||||
Spark dependencies can be declared thanks to the `spark.jars.packages` property
|
||||
(see [Spark Configuration](https://spark.apache.org/docs/latest/configuration.html#runtime-environment) for more information).
|
||||
|
||||
They can be defined as a comma-separated list of Maven coordinates at the creation of the Spark session.
|
||||
|
||||
```python
|
||||
from pyspark.sql import SparkSession
|
||||
|
||||
spark = (
|
||||
SparkSession.builder.appName("elasticsearch")
|
||||
.config(
|
||||
"spark.jars.packages",
|
||||
"org.elasticsearch:elasticsearch-spark-30_2.12:7.13.0"
|
||||
)
|
||||
.getOrCreate()
|
||||
)
|
||||
```
|
||||
|
||||
Dependencies can also be defined in the `spark-defaults.conf`.
|
||||
However, it has to be done by `root` so it should only be considered to build custom images.
|
||||
|
||||
```dockerfile
|
||||
USER root
|
||||
RUN echo "spark.jars.packages org.elasticsearch:elasticsearch-spark-30_2.12:7.13.0" >> "${SPARK_HOME}/conf/spark-defaults.conf"
|
||||
USER ${NB_UID}
|
||||
```
|
||||
|
||||
Jars will be downloaded dynamically at the creation of the Spark session and stored by default in `${HOME}/.ivy2/jars` (can be changed by setting `spark.jars.ivy`).
|
||||
|
||||
_Note: This example is given for [Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/hadoop/current/install.html)._
|
||||
|
||||
## Tensorflow
|
||||
|
||||
The `jupyter/tensorflow-notebook` image supports the use of
|
||||
|
||||
@@ -8,11 +8,11 @@
|
||||
set -e
|
||||
|
||||
# Get domain and email from environment
|
||||
[ -z "$FQDN" ] && \
|
||||
[ -z "${FQDN}" ] && \
|
||||
echo "ERROR: Must set FQDN environment varable" && \
|
||||
exit 1
|
||||
|
||||
[ -z "$EMAIL" ] && \
|
||||
[ -z "${EMAIL}" ] && \
|
||||
echo "ERROR: Must set EMAIL environment varable" && \
|
||||
exit 1
|
||||
|
||||
@@ -22,11 +22,11 @@ set -e
|
||||
|
||||
# Create Docker volume to contain the cert
|
||||
: "${SECRETS_VOLUME:=my-notebook-secrets}"
|
||||
docker volume create --name $SECRETS_VOLUME 1>/dev/null
|
||||
docker volume create --name "${SECRETS_VOLUME}" 1>/dev/null
|
||||
# Generate the cert and save it to the Docker volume
|
||||
docker run --rm -it \
|
||||
-p 80:80 \
|
||||
-v $SECRETS_VOLUME:/etc/letsencrypt \
|
||||
-v "${SECRETS_VOLUME}":/etc/letsencrypt \
|
||||
quay.io/letsencrypt/letsencrypt:latest \
|
||||
certonly \
|
||||
--non-interactive \
|
||||
@@ -34,15 +34,15 @@ docker run --rm -it \
|
||||
--standalone \
|
||||
--standalone-supported-challenges http-01 \
|
||||
--agree-tos \
|
||||
--domain "$FQDN" \
|
||||
--email "$EMAIL" \
|
||||
$CERT_SERVER
|
||||
--domain "${FQDN}" \
|
||||
--email "${EMAIL}" \
|
||||
"${CERT_SERVER}"
|
||||
|
||||
# Set permissions so nobody can read the cert and key.
|
||||
# Also symlink the certs into the root of the /etc/letsencrypt
|
||||
# directory so that the FQDN doesn't have to be known later.
|
||||
docker run --rm -it \
|
||||
-v $SECRETS_VOLUME:/etc/letsencrypt \
|
||||
-v "${SECRETS_VOLUME}":/etc/letsencrypt \
|
||||
ubuntu:20.04 \
|
||||
bash -c "ln -s /etc/letsencrypt/live/$FQDN/* /etc/letsencrypt/ && \
|
||||
bash -c "ln -s /etc/letsencrypt/live/${FQDN}/* /etc/letsencrypt/ && \
|
||||
find /etc/letsencrypt -type d -exec chmod 755 {} +"
|
||||
|
||||
@@ -7,17 +7,17 @@ set -e
|
||||
# User must have slcli installed
|
||||
which slcli > /dev/null || (echo "SoftLayer cli not found (pip install softlayer)"; exit 1)
|
||||
|
||||
USAGE="Usage: $(basename "$0") machine_name [domain]"
|
||||
USAGE="Usage: $(basename "${0}") machine_name [domain]"
|
||||
E_BADARGS=85
|
||||
|
||||
# Machine name is first command line arg
|
||||
MACHINE_NAME=$1 && [ -z "$MACHINE_NAME" ] && echo "$USAGE" && exit $E_BADARGS
|
||||
MACHINE_NAME="${1}" && [ -z "${MACHINE_NAME}" ] && echo "${USAGE}" && exit ${E_BADARGS}
|
||||
|
||||
# Use SOFTLAYER_DOMAIN env var if domain name not set as second arg
|
||||
DOMAIN="${2:-$SOFTLAYER_DOMAIN}" && [ -z "$DOMAIN" ] && \
|
||||
DOMAIN="${2:-$SOFTLAYER_DOMAIN}" && [ -z "${DOMAIN}" ] && \
|
||||
echo "Must specify domain or set SOFTLAYER_DOMAIN environment varable" && \
|
||||
echo "$USAGE" && exit $E_BADARGS
|
||||
echo "${USAGE}" && exit ${E_BADARGS}
|
||||
|
||||
IP=$(docker-machine ip "$MACHINE_NAME")
|
||||
IP=$(docker-machine ip "${MACHINE_NAME}")
|
||||
|
||||
slcli dns record-add "$DOMAIN" "$MACHINE_NAME" A "$IP"
|
||||
slcli dns record-add "${DOMAIN}" "${MACHINE_NAME}" A "${IP}"
|
||||
|
||||
@@ -9,7 +9,7 @@ USER root
|
||||
# Add permanent apt-get installs and other root commands here
|
||||
# e.g., RUN apt-get install --yes --no-install-recommends npm nodejs
|
||||
|
||||
USER $NB_UID
|
||||
USER ${NB_UID}
|
||||
|
||||
# Switch back to jovyan to avoid accidental container runs as root
|
||||
# Add permanent pip/conda installs, data files, other user libs here
|
||||
|
||||
@@ -6,7 +6,7 @@ DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
|
||||
|
||||
# Setup environment
|
||||
# shellcheck disable=SC1091
|
||||
source "$DIR/env.sh"
|
||||
source "${DIR}/env.sh"
|
||||
|
||||
# Build the notebook image
|
||||
docker-compose -f "$DIR/notebook.yml" build
|
||||
docker-compose -f "${DIR}/notebook.yml" build
|
||||
|
||||
@@ -6,7 +6,7 @@ DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
|
||||
|
||||
# Setup environment
|
||||
# shellcheck disable=SC1091
|
||||
source "$DIR/env.sh"
|
||||
source "${DIR}/env.sh"
|
||||
|
||||
# Bring down the notebook container, using container name as project name
|
||||
docker-compose -f "$DIR/notebook.yml" -p "$NAME" down
|
||||
docker-compose -f "${DIR}/notebook.yml" -p "${NAME}" down
|
||||
|
||||
@@ -14,9 +14,9 @@ export NAME
|
||||
export PORT
|
||||
|
||||
# Container work volume name
|
||||
: "${WORK_VOLUME:=$NAME-work}"
|
||||
: "${WORK_VOLUME:=${NAME}-work}"
|
||||
export WORK_VOLUME
|
||||
|
||||
# Container secrets volume name
|
||||
: "${SECRETS_VOLUME:=$NAME-secrets}"
|
||||
: "${SECRETS_VOLUME:=${NAME}-secrets}"
|
||||
export SECRETS_VOLUME
|
||||
|
||||
@@ -6,14 +6,14 @@ set -e
|
||||
|
||||
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
|
||||
|
||||
USAGE="Usage: $(basename "$0") [--secure | --letsencrypt] [--password PASSWORD] [--secrets SECRETS_VOLUME]"
|
||||
USAGE="Usage: $(basename "${0}") [--secure | --letsencrypt] [--password PASSWORD] [--secrets SECRETS_VOLUME]"
|
||||
|
||||
# Parse args to determine security settings
|
||||
SECURE=${SECURE:=no}
|
||||
LETSENCRYPT=${LETSENCRYPT:=no}
|
||||
while [[ $# -gt 0 ]]; do
|
||||
key="$1"
|
||||
case $key in
|
||||
key="${1}"
|
||||
case "${key}" in
|
||||
--secure)
|
||||
SECURE=yes
|
||||
;;
|
||||
@@ -21,11 +21,11 @@ case $key in
|
||||
LETSENCRYPT=yes
|
||||
;;
|
||||
--secrets)
|
||||
SECRETS_VOLUME="$2"
|
||||
SECRETS_VOLUME="${2}"
|
||||
shift # past argument
|
||||
;;
|
||||
--password)
|
||||
PASSWORD="$2"
|
||||
PASSWORD="${2}"
|
||||
export PASSWORD
|
||||
shift # past argument
|
||||
;;
|
||||
@@ -35,17 +35,17 @@ esac
|
||||
shift # past argument or value
|
||||
done
|
||||
|
||||
if [[ "$LETSENCRYPT" == yes || "$SECURE" == yes ]]; then
|
||||
if [[ "${LETSENCRYPT}" == yes || "${SECURE}" == yes ]]; then
|
||||
if [ -z "${PASSWORD:+x}" ]; then
|
||||
echo "ERROR: Must set PASSWORD if running in secure mode"
|
||||
echo "$USAGE"
|
||||
echo "${USAGE}"
|
||||
exit 1
|
||||
fi
|
||||
if [ "$LETSENCRYPT" == yes ]; then
|
||||
if [ "${LETSENCRYPT}" == yes ]; then
|
||||
CONFIG=letsencrypt-notebook.yml
|
||||
if [ -z "${SECRETS_VOLUME:+x}" ]; then
|
||||
echo "ERROR: Must set SECRETS_VOLUME if running in letsencrypt mode"
|
||||
echo "$USAGE"
|
||||
echo "${USAGE}"
|
||||
exit 1
|
||||
fi
|
||||
else
|
||||
@@ -59,14 +59,14 @@ fi
|
||||
|
||||
# Setup environment
|
||||
# shellcheck disable=SC1091
|
||||
source "$DIR/env.sh"
|
||||
source "${DIR}/env.sh"
|
||||
|
||||
# Create a Docker volume to store notebooks
|
||||
docker volume create --name "$WORK_VOLUME"
|
||||
docker volume create --name "${WORK_VOLUME}"
|
||||
|
||||
# Bring up a notebook container, using container name as project name
|
||||
echo "Bringing up notebook '$NAME'"
|
||||
docker-compose -f "$DIR/$CONFIG" -p "$NAME" up -d
|
||||
echo "Bringing up notebook '${NAME}'"
|
||||
docker-compose -f "${DIR}/${CONFIG}" -p "${NAME}" up -d
|
||||
|
||||
IP=$(docker-machine ip "$(docker-machine active)")
|
||||
echo "Notebook $NAME listening on $IP:$PORT"
|
||||
echo "Notebook ${NAME} listening on ${IP}:${PORT}"
|
||||
|
||||
@@ -9,7 +9,7 @@ USER root
|
||||
# Add permanent apt-get installs and other root commands here
|
||||
# e.g., RUN apt-get install --yes --no-install-recommends npm nodejs
|
||||
|
||||
USER $NB_UID
|
||||
USER ${NB_UID}
|
||||
|
||||
# Switch back to jovyan to avoid accidental container runs as root
|
||||
# Add permanent pip/conda installs, data files, other user libs here
|
||||
|
||||
@@ -77,7 +77,7 @@ The supplied `assemble` script performs a few key steps.
|
||||
The first steps copy files into the location they need to be when the image is run, from the directory where they are initially placed by the `s2i` command.
|
||||
|
||||
```bash
|
||||
cp -Rf /tmp/src/. /home/$NB_USER
|
||||
cp -Rf /tmp/src/. "/home/${NB_USER}"
|
||||
|
||||
rm -rf /tmp/src
|
||||
```
|
||||
@@ -85,12 +85,12 @@ rm -rf /tmp/src
|
||||
The next steps are:
|
||||
|
||||
```bash
|
||||
if [ -f /home/$NB_USER/environment.yml ]; then
|
||||
conda env update --name root --file /home/$NB_USER/environment.yml
|
||||
if [ -f "/home/${NB_USER}/environment.yml" ]; then
|
||||
conda env update --name root --file "/home/${NB_USER}/environment.yml"
|
||||
conda clean --all -f -y
|
||||
else
|
||||
if [ -f /home/$NB_USER/requirements.txt ]; then
|
||||
pip --no-cache-dir install -r /home/$NB_USER/requirements.txt
|
||||
if [ -f "/home/${NB_USER}/requirements.txt" ]; then
|
||||
pip --no-cache-dir install -r "/home/${NB_USER}/requirements.txt"
|
||||
fi
|
||||
fi
|
||||
```
|
||||
|
||||
@@ -33,4 +33,4 @@ RUN apt-get update --yes && \
|
||||
RUN update-alternatives --install /usr/bin/nano nano /bin/nano-tiny 10
|
||||
|
||||
# Switch back to jovyan to avoid accidental container runs as root
|
||||
USER $NB_UID
|
||||
USER ${NB_UID}
|
||||
|
||||
@@ -13,9 +13,9 @@ USER root
|
||||
# Spark dependencies
|
||||
# Default values can be overridden at build time
|
||||
# (ARGS are in lower case to distinguish them from ENV)
|
||||
ARG spark_version="3.1.1"
|
||||
ARG spark_version="3.1.2"
|
||||
ARG hadoop_version="3.2"
|
||||
ARG spark_checksum="E90B31E58F6D95A42900BA4D288261D71F6C19FA39C1CB71862B792D1B5564941A320227F6AB0E09D946F16B8C1969ED2DEA2A369EC8F9D2D7099189234DE1BE"
|
||||
ARG spark_checksum="2385CB772F21B014CE2ABD6B8F5E815721580D6E8BC42A26D70BBCDDA8D303D886A6F12B36D40F6971B5547B70FAE62B5A96146F0421CB93D4E51491308EF5D5"
|
||||
ARG openjdk_version="11"
|
||||
|
||||
ENV APACHE_SPARK_VERSION="${spark_version}" \
|
||||
@@ -39,7 +39,7 @@ WORKDIR /usr/local
|
||||
# Configure Spark
|
||||
ENV SPARK_HOME=/usr/local/spark
|
||||
ENV SPARK_OPTS="--driver-java-options=-Xms1024M --driver-java-options=-Xmx4096M --driver-java-options=-Dlog4j.logLevel=info" \
|
||||
PATH=$PATH:$SPARK_HOME/bin
|
||||
PATH="${PATH}:${SPARK_HOME}/bin"
|
||||
|
||||
RUN ln -s "spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}" spark && \
|
||||
# Add a link in the before_notebook hook in order to source automatically PYTHONPATH
|
||||
@@ -48,11 +48,11 @@ RUN ln -s "spark-${APACHE_SPARK_VERSION}-bin-hadoop${HADOOP_VERSION}" spark && \
|
||||
|
||||
# Fix Spark installation for Java 11 and Apache Arrow library
|
||||
# see: https://github.com/apache/spark/pull/27356, https://spark.apache.org/docs/latest/#downloading
|
||||
RUN cp -p "$SPARK_HOME/conf/spark-defaults.conf.template" "$SPARK_HOME/conf/spark-defaults.conf" && \
|
||||
echo 'spark.driver.extraJavaOptions -Dio.netty.tryReflectionSetAccessible=true' >> $SPARK_HOME/conf/spark-defaults.conf && \
|
||||
echo 'spark.executor.extraJavaOptions -Dio.netty.tryReflectionSetAccessible=true' >> $SPARK_HOME/conf/spark-defaults.conf
|
||||
RUN cp -p "${SPARK_HOME}/conf/spark-defaults.conf.template" "${SPARK_HOME}/conf/spark-defaults.conf" && \
|
||||
echo 'spark.driver.extraJavaOptions -Dio.netty.tryReflectionSetAccessible=true' >> "${SPARK_HOME}/conf/spark-defaults.conf" && \
|
||||
echo 'spark.executor.extraJavaOptions -Dio.netty.tryReflectionSetAccessible=true' >> "${SPARK_HOME}/conf/spark-defaults.conf"
|
||||
|
||||
USER $NB_UID
|
||||
USER ${NB_UID}
|
||||
|
||||
# Install pyarrow
|
||||
RUN conda install --quiet --yes --satisfied-skip-solve \
|
||||
@@ -61,4 +61,4 @@ RUN conda install --quiet --yes --satisfied-skip-solve \
|
||||
fix-permissions "${CONDA_DIR}" && \
|
||||
fix-permissions "/home/${NB_USER}"
|
||||
|
||||
WORKDIR $HOME
|
||||
WORKDIR "${HOME}"
|
||||
|
||||
@@ -21,7 +21,7 @@ RUN apt-get update --yes && \
|
||||
# Fix for devtools https://github.com/conda-forge/r-devtools-feedstock/issues/4
|
||||
RUN ln -s /bin/tar /bin/gtar
|
||||
|
||||
USER $NB_UID
|
||||
USER ${NB_UID}
|
||||
|
||||
# R packages including IRKernel which gets installed globally.
|
||||
RUN conda install --quiet --yes \
|
||||
|
||||
@@ -12,7 +12,7 @@ RUN apt-get update --yes && \
|
||||
apt-get install --yes --no-install-recommends ffmpeg dvipng cm-super && \
|
||||
apt-get clean && rm -rf /var/lib/apt/lists/*
|
||||
|
||||
USER $NB_UID
|
||||
USER ${NB_UID}
|
||||
|
||||
# Install Python 3 packages
|
||||
RUN conda install --quiet --yes \
|
||||
@@ -62,6 +62,6 @@ ENV XDG_CACHE_HOME="/home/${NB_USER}/.cache/"
|
||||
RUN MPLBACKEND=Agg python -c "import matplotlib.pyplot" && \
|
||||
fix-permissions "/home/${NB_USER}"
|
||||
|
||||
USER $NB_UID
|
||||
USER ${NB_UID}
|
||||
|
||||
WORKDIR $HOME
|
||||
WORKDIR "${HOME}"
|
||||
|
||||
Reference in New Issue
Block a user