hazza/docker-stacks
1
0
Fork 0
mirror of https://github.com/jupyter/docker-stacks.git synced 2025-10-12 20:42:57 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'master' into asalikhov/pre_commit_isort_reqs

Browse Source
This commit is contained in:
Ayaz Salikhov
2022-03-16 17:18:38 +02:00
parent fe36b91545 53fa575a50
commit 2da008cf5a
20 changed files with 58 additions and 47 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
.gitignore vendored
View File

@@ -162,3 +162,6 @@ cython_debug/
# VS Code project configuration
.vscode/
# PyCharm project configuration
.idea/

2
README.md
View File

@@ -1,6 +1,6 @@
# Jupyter Docker Stacks
[![Github actions badge](https://github.com/jupyter/docker-stacks/actions/workflows/docker.yml/badge.svg)](https://github.com/jupyter/docker-stacks/actions/workflows/docker.yml "Docker images build status")
[![GitHub actions badge](https://github.com/jupyter/docker-stacks/actions/workflows/docker.yml/badge.svg)](https://github.com/jupyter/docker-stacks/actions/workflows/docker.yml "Docker images build status")
[![Read the Docs badge](https://img.shields.io/readthedocs/jupyter-docker-stacks.svg)](https://jupyter-docker-stacks.readthedocs.io/en/latest/ "Documentation build status")
[![pre-commit.ci status](https://results.pre-commit.ci/badge/github/jupyter/docker-stacks/master.svg)](https://results.pre-commit.ci/latest/github/jupyter/docker-stacks/master "pre-commit.ci build status")
[![Discourse badge](https://img.shields.io/discourse/users.svg?color=%23f37626&server=https%3A%2F%2Fdiscourse.jupyter.org)](https://discourse.jupyter.org/ "Jupyter Discourse Forum")

2
docs/conf.py
View File

@@ -97,6 +97,6 @@ linkcheck_ignore = [
linkcheck_allowed_redirects = {
r"https://results\.pre-commit\.ci/latest/github/jupyter/docker-stacks/master": r"https://results\.pre-commit\.ci/run/github/.*", # Latest master CI build
r"https://github\.com/jupyter/docker-stacks/issues/new.*": r"https://github\.com/login.*", # Github wants user to be logon to use this features
r"https://github\.com/jupyter/docker-stacks/issues/new.*": r"https://github\.com/login.*", # GitHub wants user to be logon to use this features
r"https://github\.com/orgs/jupyter/teams/docker-image-maintainers/members": r"https://github\.com/login.*",
}

2
docs/contributing/issues.md
View File

@@ -10,7 +10,7 @@ Please review the following guidelines when reporting your problem.
please search the [jupyter/docker-stacks issue tracker](https://github.com/jupyter/docker-stacks/issues)
to see if someone else has already reported the same problem.
If not, please open a [GitHub bug report issue](https://github.com/jupyter/docker-stacks/issues/new?assignees=&labels=type%3ABug&template=bug_report.md&title=)
and provide all of the information requested in the issue template.
and provide all the information requested in the issue template.
Additionally, make sure to check the [Troubleshooting common error](../using/troubleshooting.md) page in the documentation before submitting an issue.
- If the issue you're seeing is with one of the open-source libraries included in the Docker images and is reproducible outside the images,
please file a bug with the appropriate open source project.

2
docs/contributing/lint.md
View File

@@ -27,7 +27,7 @@ make pre-commit-install
### Run
Now pre-commit (and so configured hooks) will run automatically on `git commit` on each changed file.
However it is also possible to trigger it against all files.
However, it is also possible to trigger it against all files.
```{note}
Hadolint pre-commit uses docker to run, so docker should be running while running this command.

2
docs/contributing/packages.md
View File

@@ -11,7 +11,7 @@ We pin major.minor version of python, so this will stay the same even after invo
## Outdated packages
In order to help identifying packages that can be updated you can use the following helper tool.
In order to help to identify packages that can be updated you can use the following helper tool.
It will list all the packages installed in the `Dockerfile` that can be updated -- dependencies are
filtered to focus only on requested packages.

8
docs/contributing/stacks.md
View File

@@ -6,7 +6,7 @@ and the documentation below to help you get started defining, building, and shar
Following these steps will:
1. Setup a project on GitHub containing a Dockerfile based on any of the images we provide.
1. Set up a project on GitHub containing a Dockerfile based on any of the images we provide.
2. Configure GitHub Actions to build and test your image when users submit pull requests to your repository.
3. Configure Docker Hub to build and host your images for others to use.
4. Update the [list of community stacks](../using/selecting.md#community-stacks) in this documentation to include your image.
@@ -70,7 +70,7 @@ git push -u origin main
The cookiecutter template comes with a `.github/workflows/docker.yml` file, which allows you to use GitHub actions to build your Docker image whenever you or someone else submits a pull request.
1. By default the `.github/workflows/docker.yaml` file has the following triggers configuration:
1. By default, the `.github/workflows/docker.yaml` file has the following triggers configuration:
```yaml
on:
@@ -112,7 +112,7 @@ you merge a GitHub pull request to the master branch of your project.
7. Select the GitHub organization and repository containing your image definition from the dropdowns.
![Dockerhub - Create Repository page focusing on the "Select Repository" dropdown menu](../_static/docker-github-settings.png)
8. Click the **Create and Build** button.
9. Click on your avatar on the top-right corner and select Account settings.
9. Click on your avatar in the top-right corner and select Account settings.
![DockerHub page zoomed into the user's settings and accounts menu](../_static/docker-org-select.png)
10. Click on **Security** and then click on the **New Access Token** button.
![DockerHub - Account page with the "Security" tab active and a rectangle highlighting the "New Access Token" button in the UI](../_static/docker-org-security.png)
@@ -126,7 +126,7 @@ you merge a GitHub pull request to the master branch of your project.
13. Head back to your GitHub repository and click on the **Settings tab**.
![GitHub page with the the "Setting" tab active and a rectangle highlighting the "New repository secret" button in the UI](../_static/github-create-secrets.png)
14. Click on the **Secrets** section and then on the **New repository secret** button on the top right corner (see image above).
14. Click on the **Secrets** section and then on the **New repository secret** button in the top right corner (see image above).
15. Create a **DOCKERHUB_TOKEN** secret and paste the Personal Access Token from DockerHub in the **value** field.
![GitHub - Actions/New secret page with the Name field set to "DOCKERHUB_TOKEN"](../_static/github-secret-token.png)
16. Repeat the above step but creating a **DOCKERHUB_USERNAME** and replacing the _value_ field with your DockerHub username.

2
docs/index.rst
View File

@@ -37,6 +37,6 @@ Table of Contents
:maxdepth: 2
:caption: Getting Help
Issue Tracker on Github <https://github.com/jupyter/docker-stacks/issues>
Issue Tracker on GitHub <https://github.com/jupyter/docker-stacks/issues>
Jupyter Discourse Forum <https://discourse.jupyter.org>
Jupyter Website <https://jupyter.org>

2
docs/maintaining/tasks.md
View File

@@ -51,7 +51,7 @@ When there's a new stack definition, do the following before merging the PR with
## Pushing a Build Manually
If automated build in Github Actions has got you down, do the following to push a build manually:
If automated build in GitHub Actions has got you down, do the following to push a build manually:
1. Clone this repository.
2. Check out the git SHA you want to build and publish.

4
docs/using/common.md
View File

@@ -129,7 +129,7 @@ You can further customize the container environment by adding shell scripts (`*.
or executables (`chmod +x`) to be run to the paths below:
- `/usr/local/bin/start-notebook.d/` - handled **before** any of the standard options noted above are applied
- `/usr/local/bin/before-notebook.d/` - handled **after** all of the standard options noted above are applied
- `/usr/local/bin/before-notebook.d/` - handled **after** all the standard options noted above are applied
and ran right before the notebook server launches
See the `run-hooks` function in the [`jupyter/base-notebook start.sh`](https://github.com/jupyter/docker-stacks/blob/master/base-notebook/start.sh)
@@ -216,7 +216,7 @@ docker run -it --rm \
### `start.sh`
The `start-notebook.sh` script inherits most of its option handling capability from a more generic `start.sh` script.
The `start.sh` script supports all of the features described above but allows you to specify an arbitrary command to execute.
The `start.sh` script supports all the features described above but allows you to specify an arbitrary command to execute.
For example, to run the text-based `ipython` console in a container, do the following:
```bash

14
docs/using/recipes.md
View File

@@ -193,8 +193,8 @@ Sometimes it is helpful to run the Jupyter instance behind a nginx proxy, for ex
- you would prefer to access the notebook at a server URL with a path
(`https://example.com/jupyter`) rather than a port (`https://example.com:8888`)
- you may have many different services in addition to Jupyter running on the same server, and want
to nginx to help improve server performance in managing the connections
- you may have many services in addition to Jupyter running on the same server, and want
nginx to help improve server performance in managing the connections
Here is a [quick example NGINX configuration](https://gist.github.com/cboettig/8643341bd3c93b62b5c2) to get started.
You'll need a server, a `.crt` and `.key` file for your server, and `docker` & `docker-compose` installed.
@@ -203,11 +203,11 @@ Customize the `nginx.conf` file to set the desired paths and add other services.
## Host volume mounts and notebook errors
If you are mounting a host directory as `/home/jovyan/work` in your container and you receive
permission errors or connection errors when you create a notebook, be sure that the `jovyan` user
(`UID=1000` by default) has read/write access to the directory on the host.
Alternatively, specify the UID of the `jovyan` user on container startup using the `-e NB_UID` option described in the
[Common Features, Docker Options section](common.md#docker-options)
If you are mounting a host directory as `/home/jovyan/work` in your container,
and you receive permission errors or connection errors when you create a notebook,
be sure that the `jovyan` user (`UID=1000` by default) has read/write access to the directory on the host.
Alternatively, specify the UID of the `jovyan` user on container startup using the `-e NB_UID` option
described in the [Common Features, Docker Options section](common.md#docker-options)
Ref: <https://github.com/jupyter/docker-stacks/issues/199>

2
docs/using/selecting.md
View File

@@ -236,7 +236,7 @@ See the [contributing guide](../contributing/stacks.md) for information about ho
[education_b]: https://mybinder.org/v2/gh/umsi-mads/education-notebook/master
[ihaskell]: https://github.com/jamesdbrock/ihaskell-notebook
[ihaskell_b]: https://mybinder.org/v2/gh/jamesdbrock/learn-you-a-haskell-notebook/master?urlpath=lab/tree/ihaskell_examples/ihaskell/IHaskell.ipynb
[ihaskell_project]: https://github.com/gibiansky/IHaskell
[ihaskell_project]: https://github.com/IHaskell/IHaskell
[java]: https://github.com/jbindinga/java-notebook
[java_b]: https://mybinder.org/v2/gh/jbindinga/java-notebook/master
[ijava]: https://github.com/SpencerPark/IJava

8
docs/using/specifics.md
View File

@@ -102,7 +102,7 @@ rdd.sum()
##### Local Mode in R
In a R notebook with [SparkR][sparkr].
In an R notebook with [SparkR][sparkr].
```R
library(SparkR)
@@ -119,7 +119,7 @@ dapplyCollect(sdf,
# 5050
```
In a R notebook with [sparklyr][sparklyr].
In an R notebook with [sparklyr][sparklyr].
```R
library(sparklyr)
@@ -194,7 +194,7 @@ rdd.sum()
##### Standalone Mode in R
In a R notebook with [SparkR][sparkr].
In an R notebook with [SparkR][sparkr].
```R
library(SparkR)
@@ -211,7 +211,7 @@ dapplyCollect(sdf,
# 5050
```
In a R notebook with [sparklyr][sparklyr].
In an R notebook with [sparklyr][sparklyr].
```R
library(sparklyr)

4
docs/using/troubleshooting.md
View File

@@ -172,8 +172,8 @@ If you have also **created a new user**, you might be experiencing any of the fo
2. **Dynamically assign the user ID and GID**
The above case ensures that the `/home` directory is owned by the a newly created user with an specific `UID` and `GID`, but if you want to assign the `UID` and `GID`
of the new user dynamically you can make the following adjustments:
The above case ensures that the `/home` directory is owned by a newly created user with a specific `UID` and `GID`,
but if you want to assign the `UID` and `GID` of the new user dynamically you can make the following adjustments:
```bash
docker run -it --rm \

7
tagging/taggers.py
View File

@@ -25,15 +25,16 @@ def _get_env_variable(container: Container, variable: str) -> str:
def _get_pip_package_version(container: Container, package: str) -> str:
VERSION_PREFIX = "Version: "
PIP_VERSION_PREFIX = "Version: "
package_info = DockerRunner.run_simple_command(
container,
cmd=f"pip show {package}",
print_result=False,
)
version_line = package_info.split("\n")[1]
assert version_line.startswith(VERSION_PREFIX)
return version_line[len(VERSION_PREFIX) :]
assert version_line.startswith(PIP_VERSION_PREFIX)
return version_line[len(PIP_VERSION_PREFIX) :]
class TaggerInterface:

19
tests/base-notebook/test_packages.py
View File

@@ -10,11 +10,14 @@ It's a basic test aiming to prove that the package is working properly.
The goal is to detect import errors that can be caused by incompatibilities between packages, for example:
- #1012: issue importing `sympy`
- #966: isssue importing `pyarrow`
- #966: issue importing `pyarrow`
This module checks dynamically, through the `CondaPackageHelper`, only the requested packages i.e. packages requested by `mamba install` in the `Dockerfile`s.
This means that it does not check dependencies. This choice is a tradeoff to cover the main requirements while achieving reasonable test duration.
However it could be easily changed (or completed) to cover also dependencies `package_helper.installed_packages()` instead of `package_helper.requested_packages()`.
This module checks dynamically, through the `CondaPackageHelper`,
only the requested packages i.e. packages requested by `mamba install` in the `Dockerfile`s.
This means that it does not check dependencies.
This choice is a tradeoff to cover the main requirements while achieving reasonable test duration.
However it could be easily changed (or completed) to cover also dependencies.
Use `package_helper.installed_packages()` instead of `package_helper.requested_packages()`.
Example:
@@ -26,7 +29,7 @@ Example:
# ---------------------------------------------------------------------------------------------- live log setup ----------------------------------------------------------------------------------------------
# 2022-02-17 16:44:36 [ INFO] Starting container jupyter/base-notebook ... (package_helper.py:55)
# 2022-02-17 16:44:36 [ INFO] Running jupyter/base-notebook with args {'detach': True, 'tty': True, 'command': ['start.sh', 'bash', '-c', 'sleep infinity']} ... (conftest.py:95)
# 2022-02-17 16:44:37 [ INFO] Grabing the list of manually requested packages ... (package_helper.py:83)
# 2022-02-17 16:44:37 [ INFO] Grabbing the list of manually requested packages ... (package_helper.py:83)
# ---------------------------------------------------------------------------------------------- live log call -----------------------------------------------------------------------------------------------
# 2022-02-17 16:44:38 [ INFO] Testing the import of packages ... (test_packages.py:144)
# 2022-02-17 16:44:38 [ INFO] Trying to import mamba (test_packages.py:146)
@@ -96,7 +99,8 @@ def get_package_import_name(package: str) -> str:
def excluded_package_predicate(package: str) -> bool:
"""Return whether a package is excluded from the list (i.e. a package that cannot be tested with standard imports)"""
"""Return whether a package is excluded from the list
(i.e. a package that cannot be tested with standard imports)"""
return package in EXCLUDED_PACKAGES
@@ -140,7 +144,8 @@ def _check_import_packages(
) -> None:
"""Test if packages can be imported
Note: using a list of packages instead of a fixture for the list of packages since pytest prevents use of multiple yields
Note: using a list of packages instead of a fixture for the list of packages
since pytest prevents use of multiple yields
"""
failures = {}
LOGGER.info("Testing the import of packages ...")

2
tests/base-notebook/test_units.py
View File

@@ -11,7 +11,7 @@ LOGGER = logging.getLogger(__name__)
def test_units(container: TrackedContainer) -> None:
"""Various units tests
Add a py file in the `tests/{somestack}-notebook/units` dir and it will be automatically tested
Add a py file in the `tests/{somestack}-notebook/units` dir, and it will be automatically tested
"""
short_image_name = container.image_name[container.image_name.rfind("/") + 1 :]
LOGGER.info(f"Running unit tests for: {short_image_name}")

14
tests/package_helper.py
View File

@@ -68,7 +68,7 @@ class CondaPackageHelper:
def installed_packages(self) -> dict[str, set[str]]:
"""Return the installed packages"""
if self.installed is None:
LOGGER.info("Grabing the list of installed packages ...")
LOGGER.info("Grabbing the list of installed packages ...")
self.installed = CondaPackageHelper._packages_from_json(
self._execute_command(
CondaPackageHelper._conda_export_command(from_history=False)
@@ -79,7 +79,7 @@ class CondaPackageHelper:
def requested_packages(self) -> dict[str, set[str]]:
"""Return the requested package (i.e. `mamba install <package>`)"""
if self.requested is None:
LOGGER.info("Grabing the list of manually requested packages ...")
LOGGER.info("Grabbing the list of manually requested packages ...")
self.requested = CondaPackageHelper._packages_from_json(
self._execute_command(
CondaPackageHelper._conda_export_command(from_history=True)
@@ -111,7 +111,7 @@ class CondaPackageHelper:
if split[1][0].isdigit():
# package + version case
version = set(split[1:])
# The split was incorrect and the package shall not be splitted
# The split was incorrect and the package shall not be split
else:
package = f"{split[0]}={split[1]}"
packages_dict[package] = version
@@ -120,7 +120,9 @@ class CondaPackageHelper:
def available_packages(self) -> dict[str, set[str]]:
"""Return the available packages"""
if self.available is None:
LOGGER.info("Grabing the list of available packages (can take a while) ...")
LOGGER.info(
"Grabbing the list of available packages (can take a while) ..."
)
# Keeping command line output since `mamba search --outdated --json` is way too long ...
self.available = CondaPackageHelper._extract_available(
self._execute_command(["mamba", "search", "--outdated", "--quiet"])
@@ -170,7 +172,7 @@ class CondaPackageHelper:
def semantic_cmp(version_string: str) -> Any:
"""Manage semantic versioning for comparison"""
def mysplit(string: str) -> list[Any]:
def my_split(string: str) -> list[Any]:
def version_substrs(x: str) -> list[str]:
return re.findall(r"([A-z]+|\d+)", x)
@@ -189,7 +191,7 @@ class CondaPackageHelper:
except ValueError:
return str_ord(version_str)
mss = list(chain(*mysplit(version_string)))
mss = list(chain(*my_split(version_string)))
return tuple(map(try_int, mss))
def get_outdated_summary(self, requested_only: bool = True) -> str:

4
tests/scipy-notebook/data/matplotlib/matplotlib_1.py
View File

@@ -1,5 +1,5 @@
# Matplotlit: Create a simple plot example.
# Refs: https://matplotlib.org/3.1.1/gallery/lines_bars_and_markers/simple_plot.html
# Matplotlib: Create a simple plot example.
# Refs: https://matplotlib.org/stable/gallery/lines_bars_and_markers/simple_plot.html
import os

2
tests/scipy-notebook/data/matplotlib/matplotlib_fonts_1.py
View File

@@ -1,4 +1,4 @@
# Matplotlit: Test tex fonts
# Matplotlib: Test tex fonts
import os
import matplotlib