hazza/docker-stacks
1
0
Fork 0
mirror of https://github.com/jupyter/docker-stacks.git synced 2025-10-07 10:04:03 +00:00
Code Issues Packages Projects Releases Wiki Activity

Better tagging directory structure (#2228)

Browse Source
This commit is contained in:
Ayaz Salikhov
2025-02-21 12:48:18 +00:00
committed by GitHub
parent 355813e360
commit e815fde31b
45 changed files with 374 additions and 330 deletions
Download Patch File Download Diff File Expand all files Collapse all files

27
tagging/Home.md
View File

@@ -1,27 +0,0 @@
# Jupyter Docker Stacks build manifests
<!-- Note: this file is copied to wiki from the main repo, edits on wiki page will be overridden -->
Welcome!
Please see [the documentation](https://jupyter-docker-stacks.readthedocs.io/en/latest/) for help with
using, contributing to, and maintaining the Jupyter Docker stacks images.
## Build History
This is an auto-generated index of information from the build system.
In this index, you can find image tags, links to commits, and build manifests that describe the image.
All the builds are grouped by year and then month.
Note: we only store the last 4500 manifest files because of GitHub limits.
That's why old manifest files might not be available.
If you want to clone this repo and access the Git history, use the following command: `git clone git@github.com:{REPOSITORY}.wiki.git`
In the tables below, each line represents:
- `YYYY-MM`: link to a page with a list of images built
- `Builds`: # of times build workflow finished
- `Images`: # of single platform images pushed
- `Commits`: # of commits made and a GitHub link
<!-- Everything below is auto-generated, all manual changes will be erased -->
<!-- YEAR_MONTHLY_TABLES -->

125
tagging/README.md
View File

@@ -1,126 +1,3 @@
# Docker stacks tagging and manifest creation
The main purpose of the source code in this folder is to properly tag all the images and to update [build manifests](https://github.com/jupyter/docker-stacks/wiki).
These two processes are closely related, so the source code is widely reused.
A basic example of a tag is a `Python` version tag.
For example, an image `jupyter/base-notebook` with `python 3.10.5` will have a full image name `quay.io/jupyter/base-notebook:python-3.10.5`.
This tag (and all the other tags) are pushed to Quay.io.
Manifest is a description of some important part of the image in a `markdown`.
For example, we dump all the `conda` packages, including their versions.
## Main principles
- All the images are located in a hierarchical tree.
More info on [image relationships](../docs/using/selecting.md#image-relationships).
- We have `tagger` and `manifest` classes, which can be run inside docker containers to obtain tags and build manifest pieces.
- These classes are inherited from the parent image to all the child images.
- Because manifests and tags might change from parent to child, `taggers` and `manifests` are reevaluated on each image.
So, the values are not inherited.
- To tag an image and create a manifest, run `make hook/base-notebook` (or another image of your choice).
## Source code description
In this section, we will briefly describe the source code in this folder and give examples of how to use it.
### DockerRunner
`DockerRunner` is a helper class to easily run a docker container and execute commands inside this container:
```python
from tagging.docker_runner import DockerRunner
with DockerRunner("ubuntu:22.04") as container:
DockerRunner.run_simple_command(container, cmd="env", print_result=True)
```
### GitHelper
`GitHelper` methods are run in the current `git` repo and give the information about the last commit hash and commit message:
```python
from tagging.git_helper import GitHelper
print("Git hash:", GitHelper.commit_hash())
print("Git message:", GitHelper.commit_message())
```
The prefix of commit hash (namely, 12 letters) is used as an image tag to make it easy to inherit from a fixed version of a docker image.
### Tagger
`Tagger` is a class that can be run inside a docker container to calculate some tag for an image.
All the taggers are inherited from `TaggerInterface`:
```python
class TaggerInterface:
"""Common interface for all taggers"""
@staticmethod
def tag_value(container) -> str:
raise NotImplementedError
```
So, the `tag_value(container)` method gets a docker container as an input and returns a tag.
`SHATagger` example:
```python
from tagging.git_helper import GitHelper
from tagging.taggers import TaggerInterface
class SHATagger(TaggerInterface):
@staticmethod
def tag_value(container):
return GitHelper.commit_hash_tag()
```
- `taggers.py` contains all the taggers.
- `tag_image.py` is a Python executable that is used to tag the image.
### Manifest
`ManifestHeader` is a build manifest header.
It contains the following sections: `Build timestamp`, `Docker image size`, and `Git commit` info.
All the other manifest classes are inherited from `ManifestInterface`:
```python
class ManifestInterface:
"""Common interface for all manifests"""
@staticmethod
def markdown_piece(container) -> str:
raise NotImplementedError
```
- The `markdown_piece(container)` method returns a piece of markdown file to be used as a part of the build manifest.
`AptPackagesManifest` example:
```python
from tagging.manifests import ManifestInterface, quoted_output
class AptPackagesManifest(ManifestInterface):
@staticmethod
def markdown_piece(container) -> str:
return f"""\
## Apt Packages
{quoted_output(container, "apt list --installed")}"""
```
- `quoted_output` simply runs the command inside a container using `DockerRunner.run_simple_command` and wraps it to triple quotes to create a valid markdown piece.
It also adds the command which was run to the markdown piece.
- `manifests.py` contains all the manifests.
- `write_manifest.py` is a Python executable that is used to create the build manifest and history line for an image.
### Images Hierarchy
All images' dependencies on each other and what taggers and manifest they make use of are defined in `images_hierarchy.py`.
`get_taggers_and_manifests.py` defines a helper function to get the taggers and manifests for a specific image.
Please, refer to the [tagging section of documentation](https://jupyter-docker-stacks.readthedocs.io/en/latest/maintaing/tagging.html) to see how tags and manifests are created.

0
tagging/apps/__init__.py Normal file
View File

6
tagging/apply_tags.py → tagging/apps/apply_tags.py
View File

@@ -6,9 +6,9 @@ from pathlib import Path
import plumbum
from tagging.common_arguments import common_arguments_parser
from tagging.get_platform import unify_aarch64
from tagging.get_prefix import get_file_prefix_for_platform
from tagging.apps.common_cli_arguments import common_arguments_parser
from tagging.utils.get_platform import unify_aarch64
from tagging.utils.get_prefix import get_file_prefix_for_platform
docker = plumbum.local["docker"]

0
tagging/common_arguments.py → tagging/apps/common_cli_arguments.py
View File

6
tagging/merge_tags.py → tagging/apps/merge_tags.py
View File

@@ -6,9 +6,9 @@ from pathlib import Path
import plumbum
from tagging.common_arguments import common_arguments_parser
from tagging.get_platform import ALL_PLATFORMS
from tagging.get_prefix import get_file_prefix_for_platform
from tagging.apps.common_cli_arguments import common_arguments_parser
from tagging.utils.get_platform import ALL_PLATFORMS
from tagging.utils.get_prefix import get_file_prefix_for_platform
docker = plumbum.local["docker"]

15
tagging/write_manifest.py → tagging/apps/write_manifest.py
View File

@@ -7,12 +7,15 @@ from pathlib import Path
from docker.models.containers import Container
from tagging.common_arguments import common_arguments_parser
from tagging.docker_runner import DockerRunner
from tagging.get_prefix import get_file_prefix, get_tag_prefix
from tagging.get_taggers_and_manifests import get_taggers_and_manifests
from tagging.git_helper import GitHelper
from tagging.manifests import ManifestHeader, ManifestInterface
from tagging.apps.common_cli_arguments import common_arguments_parser
from tagging.hierarchy.get_taggers_and_manifests import (
get_taggers_and_manifests,
)
from tagging.manifests.header import ManifestHeader
from tagging.manifests.manifest_interface import ManifestInterface
from tagging.utils.docker_runner import DockerRunner
from tagging.utils.get_prefix import get_file_prefix, get_tag_prefix
from tagging.utils.git_helper import GitHelper
LOGGER = logging.getLogger(__name__)

10
tagging/write_tags_file.py → tagging/apps/write_tags_file.py
View File

@@ -4,10 +4,12 @@
import logging
from pathlib import Path
from tagging.common_arguments import common_arguments_parser
from tagging.docker_runner import DockerRunner
from tagging.get_prefix import get_file_prefix, get_tag_prefix
from tagging.get_taggers_and_manifests import get_taggers_and_manifests
from tagging.apps.common_cli_arguments import common_arguments_parser
from tagging.hierarchy.get_taggers_and_manifests import (
get_taggers_and_manifests,
)
from tagging.utils.docker_runner import DockerRunner
from tagging.utils.get_prefix import get_file_prefix, get_tag_prefix
LOGGER = logging.getLogger(__name__)

0
tagging/hierarchy/__init__.py Normal file
View File

6
tagging/get_taggers_and_manifests.py → tagging/hierarchy/get_taggers_and_manifests.py
View File

@@ -1,9 +1,9 @@
# Copyright (c) Jupyter Development Team.
# Distributed under the terms of the Modified BSD License.
from tagging.images_hierarchy import ALL_IMAGES
from tagging.manifests import ManifestInterface
from tagging.taggers import TaggerInterface
from tagging.hierarchy.images_hierarchy import ALL_IMAGES
from tagging.manifests.manifest_interface import ManifestInterface
from tagging.taggers.tagger_interface import TaggerInterface
def get_taggers_and_manifests(

24
tagging/images_hierarchy.py → tagging/hierarchy/images_hierarchy.py
View File

@@ -2,16 +2,17 @@
# Distributed under the terms of the Modified BSD License.
from dataclasses import dataclass, field
from tagging.manifests import (
AptPackagesManifest,
CondaEnvironmentManifest,
JuliaPackagesManifest,
ManifestInterface,
RPackagesManifest,
SparkInfoManifest,
)
from tagging.taggers import (
DateTagger,
from tagging.manifests.apt_packages import AptPackagesManifest
from tagging.manifests.conda_environment import CondaEnvironmentManifest
from tagging.manifests.julia_packages import JuliaPackagesManifest
from tagging.manifests.manifest_interface import ManifestInterface
from tagging.manifests.r_packages import RPackagesManifest
from tagging.manifests.spark_info import SparkInfoManifest
from tagging.taggers.date import DateTagger
from tagging.taggers.sha import SHATagger
from tagging.taggers.tagger_interface import TaggerInterface
from tagging.taggers.ubuntu_version import UbuntuVersionTagger
from tagging.taggers.versions import (
JavaVersionTagger,
JuliaVersionTagger,
JupyterHubVersionTagger,
@@ -21,11 +22,8 @@ from tagging.taggers import (
PythonVersionTagger,
PytorchVersionTagger,
RVersionTagger,
SHATagger,
SparkVersionTagger,
TaggerInterface,
TensorflowVersionTagger,
UbuntuVersionTagger,
)

126
tagging/manifests.py
View File

@@ -1,126 +0,0 @@
# Copyright (c) Jupyter Development Team.
# Distributed under the terms of the Modified BSD License.
import plumbum
from docker.models.containers import Container
from tagging.docker_runner import DockerRunner
from tagging.git_helper import GitHelper
docker = plumbum.local["docker"]
def quoted_output(container: Container, cmd: str) -> str:
cmd_output = DockerRunner.run_simple_command(container, cmd, print_result=False)
# For example, `mamba info` adds redundant empty lines
cmd_output = cmd_output.strip("\n")
# For example, R packages list contains trailing backspaces
cmd_output = "\n".join(line.rstrip() for line in cmd_output.split("\n"))
assert cmd_output, f"Command `{cmd}` returned empty output"
return f"""\
`{cmd}`:
```text
{cmd_output}
```"""
class ManifestHeader:
"""ManifestHeader doesn't fall under common interface, and we run it separately"""
@staticmethod
def create_header(
short_image_name: str, registry: str, owner: str, build_timestamp: str
) -> str:
commit_hash = GitHelper.commit_hash()
commit_hash_tag = GitHelper.commit_hash_tag()
commit_message = GitHelper.commit_message()
# Unfortunately, `docker images` doesn't work when specifying `docker.io` as registry
fixed_registry = registry + "/" if registry != "docker.io" else ""
image_size = docker[
"images",
f"{fixed_registry}{owner}/{short_image_name}:latest",
"--format",
"{{.Size}}",
]().rstrip()
return f"""\
# Build manifest for image: {short_image_name}:{commit_hash_tag}
## Build Info
- Build timestamp: {build_timestamp}
- Docker image: `{registry}/{owner}/{short_image_name}:{commit_hash_tag}`
- Docker image size: {image_size}
- Git commit SHA: [{commit_hash}](https://github.com/jupyter/docker-stacks/commit/{commit_hash})
- Git commit message:
```text
{commit_message}
```"""
class ManifestInterface:
"""Common interface for all manifests"""
@staticmethod
def markdown_piece(container: Container) -> str:
raise NotImplementedError
class CondaEnvironmentManifest(ManifestInterface):
@staticmethod
def markdown_piece(container: Container) -> str:
return f"""\
## Python Packages
{DockerRunner.run_simple_command(container, "python --version")}
{quoted_output(container, "conda info")}
{quoted_output(container, "mamba info")}
{quoted_output(container, "mamba list")}"""
class AptPackagesManifest(ManifestInterface):
@staticmethod
def markdown_piece(container: Container) -> str:
return f"""\
## Apt Packages
{quoted_output(container, "apt list --installed")}"""
class RPackagesManifest(ManifestInterface):
@staticmethod
def markdown_piece(container: Container) -> str:
return f"""\
## R Packages
{quoted_output(container, "R --version")}
{quoted_output(container, "R --silent -e 'installed.packages(.Library)[, c(1,3)]'")}"""
class JuliaPackagesManifest(ManifestInterface):
@staticmethod
def markdown_piece(container: Container) -> str:
return f"""\
## Julia Packages
{quoted_output(container, "julia -E 'using InteractiveUtils; versioninfo()'")}
{quoted_output(container, "julia -E 'import Pkg; Pkg.status()'")}"""
class SparkInfoManifest(ManifestInterface):
@staticmethod
def markdown_piece(container: Container) -> str:
return f"""\
## Apache Spark
{quoted_output(container, "/usr/local/spark/bin/spark-submit --version")}"""

0
tagging/manifests/__init__.py Normal file
View File

15
tagging/manifests/apt_packages.py Normal file
View File

@@ -0,0 +1,15 @@
# Copyright (c) Jupyter Development Team.
# Distributed under the terms of the Modified BSD License.
from docker.models.containers import Container
from tagging.manifests.manifest_interface import ManifestInterface
from tagging.utils.quoted_output import quoted_output
class AptPackagesManifest(ManifestInterface):
@staticmethod
def markdown_piece(container: Container) -> str:
return f"""\
## Apt Packages
{quoted_output(container, "apt list --installed")}"""

22
tagging/manifests/conda_environment.py Normal file
View File

@@ -0,0 +1,22 @@
# Copyright (c) Jupyter Development Team.
# Distributed under the terms of the Modified BSD License.
from docker.models.containers import Container
from tagging.manifests.manifest_interface import ManifestInterface
from tagging.utils.docker_runner import DockerRunner
from tagging.utils.quoted_output import quoted_output
class CondaEnvironmentManifest(ManifestInterface):
@staticmethod
def markdown_piece(container: Container) -> str:
return f"""\
## Python Packages
{DockerRunner.run_simple_command(container, "python --version")}
{quoted_output(container, "conda info")}
{quoted_output(container, "mamba info")}
{quoted_output(container, "mamba list")}"""

44
tagging/manifests/header.py Normal file
View File

@@ -0,0 +1,44 @@
# Copyright (c) Jupyter Development Team.
# Distributed under the terms of the Modified BSD License.
import plumbum
from tagging.utils.git_helper import GitHelper
docker = plumbum.local["docker"]
class ManifestHeader:
"""ManifestHeader doesn't fall under common interface, and we run it separately"""
@staticmethod
def create_header(
short_image_name: str, registry: str, owner: str, build_timestamp: str
) -> str:
commit_hash = GitHelper.commit_hash()
commit_hash_tag = GitHelper.commit_hash_tag()
commit_message = GitHelper.commit_message()
# Unfortunately, `docker images` doesn't work when specifying `docker.io` as registry
fixed_registry = registry + "/" if registry != "docker.io" else ""
image_size = docker[
"images",
f"{fixed_registry}{owner}/{short_image_name}:latest",
"--format",
"{{.Size}}",
]().rstrip()
return f"""\
# Build manifest for image: {short_image_name}:{commit_hash_tag}
## Build Info
- Build timestamp: {build_timestamp}
- Docker image: `{registry}/{owner}/{short_image_name}:{commit_hash_tag}`
- Docker image size: {image_size}
- Git commit SHA: [{commit_hash}](https://github.com/jupyter/docker-stacks/commit/{commit_hash})
- Git commit message:
```text
{commit_message}
```"""

17
tagging/manifests/julia_packages.py Normal file
View File

@@ -0,0 +1,17 @@
# Copyright (c) Jupyter Development Team.
# Distributed under the terms of the Modified BSD License.
from docker.models.containers import Container
from tagging.manifests.manifest_interface import ManifestInterface
from tagging.utils.quoted_output import quoted_output
class JuliaPackagesManifest(ManifestInterface):
@staticmethod
def markdown_piece(container: Container) -> str:
return f"""\
## Julia Packages
{quoted_output(container, "julia -E 'using InteractiveUtils; versioninfo()'")}
{quoted_output(container, "julia -E 'import Pkg; Pkg.status()'")}"""

9
tagging/manifests/manifest_interface.py Normal file
View File

@@ -0,0 +1,9 @@
from docker.models.containers import Container
class ManifestInterface:
"""Common interface for all manifests"""
@staticmethod
def markdown_piece(container: Container) -> str:
raise NotImplementedError

17
tagging/manifests/r_packages.py Normal file
View File

@@ -0,0 +1,17 @@
# Copyright (c) Jupyter Development Team.
# Distributed under the terms of the Modified BSD License.
from docker.models.containers import Container
from tagging.manifests.manifest_interface import ManifestInterface
from tagging.utils.quoted_output import quoted_output
class RPackagesManifest(ManifestInterface):
@staticmethod
def markdown_piece(container: Container) -> str:
return f"""\
## R Packages
{quoted_output(container, "R --version")}
{quoted_output(container, "R --silent -e 'installed.packages(.Library)[, c(1,3)]'")}"""

15
tagging/manifests/spark_info.py Normal file
View File

@@ -0,0 +1,15 @@
# Copyright (c) Jupyter Development Team.
# Distributed under the terms of the Modified BSD License.
from docker.models.containers import Container
from tagging.manifests.manifest_interface import ManifestInterface
from tagging.utils.quoted_output import quoted_output
class SparkInfoManifest(ManifestInterface):
@staticmethod
def markdown_piece(container: Container) -> str:
return f"""\
## Apache Spark
{quoted_output(container, "/usr/local/spark/bin/spark-submit --version")}"""

0
tagging/taggers/__init__.py Normal file
View File

13
tagging/taggers/date.py Normal file
View File

@@ -0,0 +1,13 @@
# Copyright (c) Jupyter Development Team.
# Distributed under the terms of the Modified BSD License.
import datetime
from docker.models.containers import Container
from tagging.taggers.tagger_interface import TaggerInterface
class DateTagger(TaggerInterface):
@staticmethod
def tag_value(container: Container) -> str:
return datetime.datetime.now(datetime.UTC).strftime("%Y-%m-%d")

12
tagging/taggers/sha.py Normal file
View File

@@ -0,0 +1,12 @@
# Copyright (c) Jupyter Development Team.
# Distributed under the terms of the Modified BSD License.
from docker.models.containers import Container
from tagging.taggers.tagger_interface import TaggerInterface
from tagging.utils.git_helper import GitHelper
class SHATagger(TaggerInterface):
@staticmethod
def tag_value(container: Container) -> str:
return GitHelper.commit_hash_tag()

9
tagging/taggers/tagger_interface.py Normal file
View File

@@ -0,0 +1,9 @@
from docker.models.containers import Container
class TaggerInterface:
"""Common interface for all taggers"""
@staticmethod
def tag_value(container: Container) -> str:
raise NotImplementedError

19
tagging/taggers/ubuntu_version.py Normal file
View File

@@ -0,0 +1,19 @@
# Copyright (c) Jupyter Development Team.
# Distributed under the terms of the Modified BSD License.
from docker.models.containers import Container
from tagging.taggers.tagger_interface import TaggerInterface
from tagging.utils.docker_runner import DockerRunner
class UbuntuVersionTagger(TaggerInterface):
@staticmethod
def tag_value(container: Container) -> str:
os_release = DockerRunner.run_simple_command(
container,
"cat /etc/os-release",
).split("\n")
for line in os_release:
if line.startswith("VERSION_ID"):
return "ubuntu-" + line.split("=")[1].strip('"')
raise RuntimeError(f"did not find ubuntu version in: {os_release}")

39
tagging/taggers.py → tagging/taggers/versions.py
View File

@@ -1,11 +1,9 @@
# Copyright (c) Jupyter Development Team.
# Distributed under the terms of the Modified BSD License.
import datetime
from docker.models.containers import Container
from tagging.docker_runner import DockerRunner
from tagging.git_helper import GitHelper
from tagging.taggers.tagger_interface import TaggerInterface
from tagging.utils.docker_runner import DockerRunner
def _get_program_version(container: Container, program: str) -> str:
@@ -25,39 +23,6 @@ def _get_pip_package_version(container: Container, package: str) -> str:
return version_line[len(PIP_VERSION_PREFIX) :]
class TaggerInterface:
"""Common interface for all taggers"""
@staticmethod
def tag_value(container: Container) -> str:
raise NotImplementedError
class SHATagger(TaggerInterface):
@staticmethod
def tag_value(container: Container) -> str:
return GitHelper.commit_hash_tag()
class DateTagger(TaggerInterface):
@staticmethod
def tag_value(container: Container) -> str:
return datetime.datetime.now(datetime.UTC).strftime("%Y-%m-%d")
class UbuntuVersionTagger(TaggerInterface):
@staticmethod
def tag_value(container: Container) -> str:
os_release = DockerRunner.run_simple_command(
container,
"cat /etc/os-release",
).split("\n")
for line in os_release:
if line.startswith("VERSION_ID"):
return "ubuntu-" + line.split("=")[1].strip('"')
raise RuntimeError(f"did not find ubuntu version in: {os_release}")
class PythonVersionTagger(TaggerInterface):
@staticmethod
def tag_value(container: Container) -> str:

228
tagging/update_wiki.py
View File

@@ -1,228 +0,0 @@
#!/usr/bin/env python3
# Copyright (c) Jupyter Development Team.
# Distributed under the terms of the Modified BSD License.
import argparse
import datetime
import logging
import shutil
from pathlib import Path
import plumbum
from dateutil import relativedelta
git = plumbum.local["git"]
LOGGER = logging.getLogger(__name__)
THIS_DIR = Path(__file__).parent.resolve()
def calculate_monthly_stat(
year_month_file: Path, year_month_date: datetime.date
) -> tuple[int, int, int]:
year_month_file_content = year_month_file.read_text()
builds = sum(
"jupyter/base-notebook" in line and "aarch64" not in line
for line in year_month_file_content.split("\n")
)
images = year_month_file_content.count("Build manifest")
with plumbum.local.env(TZ="UTC"):
future = (
git[
"log",
"--oneline",
"--since",
f"{year_month_date}.midnight",
"--until",
f"{year_month_date + relativedelta.relativedelta(months=1)}.midnight",
"--first-parent",
]
& plumbum.BG
)
future.wait()
commits = len(future.stdout.splitlines())
return builds, images, commits
def generate_home_wiki_page(wiki_dir: Path, repository: str) -> None:
YEAR_MONTHLY_TABLES = "<!-- YEAR_MONTHLY_TABLES -->\n"
wiki_home_content = (THIS_DIR / "Home.md").read_text()
assert YEAR_MONTHLY_TABLES in wiki_home_content
wiki_home_content = wiki_home_content[
: wiki_home_content.find(YEAR_MONTHLY_TABLES) + len(YEAR_MONTHLY_TABLES)
]
wiki_home_content = wiki_home_content.format(REPOSITORY=repository)
YEAR_TABLE_HEADER = """\
## {year}
| Month | Builds | Images | Commits |
| ---------------------- | ------ | ------ | ----------------------------------------------------------------------------------------------- |
"""
GITHUB_COMMITS_URL = (
f"[{{}}](https://github.com/{repository}/commits/main/?since={{}}&until={{}})"
)
for year_dir in sorted((wiki_dir / "monthly-files").glob("*"), reverse=True):
wiki_home_content += "\n" + YEAR_TABLE_HEADER.format(year=year_dir.name)
year_builds, year_images, year_commits = 0, 0, 0
for year_month_file in sorted(year_dir.glob("*.md"), reverse=True):
year_month = year_month_file.stem
year_month_date = datetime.date(
year=int(year_month[:4]), month=int(year_month[5:]), day=1
)
builds, images, commits = calculate_monthly_stat(
year_month_file, year_month_date
)
year_builds += builds
year_images += images
year_commits += commits
commits_url = GITHUB_COMMITS_URL.format(
commits,
year_month_date,
year_month_date + relativedelta.relativedelta(day=31),
)
monthly_line = f"| [`{year_month}`](./{year_month}) | {builds: <6} | {images: <6} | {commits_url: <95} |\n"
wiki_home_content += monthly_line
year_commits_url = GITHUB_COMMITS_URL.format(
year_commits, f"{year_dir.name}-01-01", f"{year_dir.name}-12-31"
)
year_total_line = f"| **Total** | {year_builds: <6} | {year_images: <6} | {year_commits_url: <95} |\n"
wiki_home_content += year_total_line
(wiki_dir / "Home.md").write_text(wiki_home_content)
LOGGER.info("Updated Home page")
def update_monthly_wiki_page(
wiki_dir: Path, year_month: str, build_history_line: str
) -> None:
MONTHLY_PAGE_HEADER = f"""\
# Images built during {year_month}
| Date | Image | Links |
| - | - | - |
"""
year = year_month[:4]
monthly_page = wiki_dir / "monthly-files" / year / (year_month + ".md")
if not monthly_page.exists():
monthly_page.parent.mkdir(parents=True, exist_ok=True)
monthly_page.write_text(MONTHLY_PAGE_HEADER)
LOGGER.info(f"Created monthly page: {monthly_page.relative_to(wiki_dir)}")
monthly_page_content = monthly_page.read_text()
assert MONTHLY_PAGE_HEADER in monthly_page_content
monthly_page_content = monthly_page_content.replace(
MONTHLY_PAGE_HEADER, MONTHLY_PAGE_HEADER + build_history_line + "\n"
)
monthly_page.write_text(monthly_page_content)
LOGGER.info(f"Updated monthly page: {monthly_page.relative_to(wiki_dir)}")
def get_manifest_timestamp(manifest_file: Path) -> str:
file_content = manifest_file.read_text()
TIMESTAMP_PREFIX = "Build timestamp: "
TIMESTAMP_LENGTH = 20
timestamp = file_content[
file_content.find(TIMESTAMP_PREFIX) + len(TIMESTAMP_PREFIX) :
][:TIMESTAMP_LENGTH]
# Should be good enough till year 2100
assert timestamp.startswith("20"), timestamp
assert timestamp.endswith("Z"), timestamp
return timestamp
def get_manifest_year_month(manifest_file: Path) -> str:
return get_manifest_timestamp(manifest_file)[:7]
def remove_old_manifests(wiki_dir: Path) -> None:
MAX_NUMBER_OF_MANIFESTS = 4500
manifest_files: list[tuple[str, Path]] = []
for file in (wiki_dir / "manifests").rglob("*.md"):
manifest_files.append((get_manifest_timestamp(file), file))
manifest_files.sort(reverse=True)
for _, file in manifest_files[MAX_NUMBER_OF_MANIFESTS:]:
file.unlink()
LOGGER.info(f"Removed manifest: {file.relative_to(wiki_dir)}")
def update_wiki(
*,
wiki_dir: Path,
hist_lines_dir: Path,
manifests_dir: Path,
repository: str,
allow_no_files: bool,
) -> None:
LOGGER.info("Updating wiki")
manifest_files = list(manifests_dir.rglob("*.md"))
if not allow_no_files:
assert manifest_files, "expected to have some manifest files"
for manifest_file in manifest_files:
year_month = get_manifest_year_month(manifest_file)
year = year_month[:4]
copy_to = wiki_dir / "manifests" / year / year_month / manifest_file.name
copy_to.parent.mkdir(parents=True, exist_ok=True)
shutil.copy(manifest_file, copy_to)
LOGGER.info(f"Added manifest file: {copy_to.relative_to(wiki_dir)}")
build_history_line_files = sorted(hist_lines_dir.rglob("*.txt"))
if not allow_no_files:
assert (
build_history_line_files
), "expected to have some build history line files"
for build_history_line_file in build_history_line_files:
build_history_line = build_history_line_file.read_text()
assert build_history_line.startswith("| `")
year_month = build_history_line[3:10]
update_monthly_wiki_page(wiki_dir, year_month, build_history_line)
generate_home_wiki_page(wiki_dir, repository)
remove_old_manifests(wiki_dir)
if __name__ == "__main__":
logging.basicConfig(level=logging.INFO)
arg_parser = argparse.ArgumentParser()
arg_parser.add_argument(
"--wiki-dir",
required=True,
type=Path,
help="Directory of the wiki repo",
)
arg_parser.add_argument(
"--hist-lines-dir",
required=True,
type=Path,
help="Directory with history lines",
)
arg_parser.add_argument(
"--manifests-dir",
required=True,
type=Path,
help="Directory with manifest files",
)
arg_parser.add_argument(
"--repository",
required=True,
help="Repository name on GitHub",
)
arg_parser.add_argument(
"--allow-no-files",
action="store_true",
help="Allow no manifest or history line files",
)
args = arg_parser.parse_args()
update_wiki(**vars(args))

0
tagging/utils/__init__.py Normal file
View File

0
tagging/docker_runner.py → tagging/utils/docker_runner.py
View File

0
tagging/get_platform.py → tagging/utils/get_platform.py
View File

2
tagging/get_prefix.py → tagging/utils/get_prefix.py
View File

@@ -1,6 +1,6 @@
# Copyright (c) Jupyter Development Team.
# Distributed under the terms of the Modified BSD License.
from tagging.get_platform import get_platform
from tagging.utils.get_platform import get_platform
DEFAULT_VARIANT = "default"

0
tagging/git_helper.py → tagging/utils/git_helper.py
View File

22
tagging/utils/quoted_output.py Normal file
View File

@@ -0,0 +1,22 @@
# Copyright (c) Jupyter Development Team.
# Distributed under the terms of the Modified BSD License.
from docker.models.containers import Container
from tagging.utils.docker_runner import DockerRunner
def quoted_output(container: Container, cmd: str) -> str:
cmd_output = DockerRunner.run_simple_command(container, cmd, print_result=False)
# For example, `mamba info` adds redundant empty lines
cmd_output = cmd_output.strip("\n")
# For example, R packages list contains trailing backspaces
cmd_output = "\n".join(line.rstrip() for line in cmd_output.split("\n"))
assert cmd_output, f"Command `{cmd}` returned empty output"
return f"""\
`{cmd}`:
```text
{cmd_output}
```"""