Merge pull request #3665 from minrk/openapi-test

Tests for our openapi spec

.github/workflows/test.yml

@@ -14,8 +14,28 @@ on:
 env:
   # UTF-8 content may be interpreted as ascii and causes errors without this.
   LANG: C.UTF-8
+  PYTEST_ADDOPTS: "--verbose --color=yes"
 
 jobs:
+  rest-api:
+    runs-on: ubuntu-20.04
+    steps:
+      - uses: actions/checkout@v2
+      - name: Validate REST API
+        uses: char0n/swagger-editor-validate@182d1a5d26ff5c2f4f452c43bd55e2c7d8064003
+        with:
+          definition-file: docs/source/_static/rest-api.yml
+
+      - uses: actions/setup-python@v2
+        with:
+          python-version: "3.9"
+      # in addition to the doc requirements
+      # the docs *tests* require pre-commit and pytest
+      - run: |
+          pip install -r docs/requirements.txt pytest pre-commit -e .
+      - run: |
+          pytest docs/
+
   # Run "pytest jupyterhub/tests" in various configurations
   pytest:
     runs-on: ubuntu-20.04
@@ -182,10 +202,8 @@ jobs:
           fi
 
       - name: Run pytest
-        # FIXME: --color=yes explicitly set because:
-        #        https://github.com/actions/runner/issues/241
         run: |
-          pytest -v --maxfail=2 --color=yes --cov=jupyterhub jupyterhub/tests
+          pytest --maxfail=2 --cov=jupyterhub jupyterhub/tests
       - name: Run yarn jest test
         run: |
           cd jsx && yarn && yarn test

CHECKLIST-Release.md

@@ -1,26 +0,0 @@
-# Release checklist
-
-- [ ] Upgrade Docs prior to Release
-
-  - [ ] Change log
-  - [ ] New features documented
-  - [ ] Update the contributor list - thank you page
-
-- [ ] Upgrade and test Reference Deployments
-
-- [ ] Release software
-
-  - [ ] Make sure 0 issues in milestone
-  - [ ] Follow release process steps
-  - [ ] Send builds to PyPI (Warehouse) and Conda Forge
-
-- [ ] Blog post and/or release note
-
-- [ ] Notify users of release
-
-  - [ ] Email Jupyter and Jupyter In Education mailing lists
-  - [ ] Tweet (optional)
-
-- [ ] Increment the version number for the next release
-
-- [ ] Update roadmap

README.md

@@ -56,9 +56,11 @@ Basic principles for operation are:
   servers.
 
 JupyterHub also provides a
-[REST API](https://petstore3.swagger.io/?url=https://raw.githubusercontent.com/jupyter/jupyterhub/HEAD/docs/rest-api.yml#/default)
+[REST API][]
 for administration of the Hub and its users.
 
+[rest api]: https://juptyerhub.readthedocs.io/en/latest/reference/rest-api.html
+
 ## Installation
 
 ### Check prerequisites
@@ -239,7 +241,7 @@ You can also talk with us on our JupyterHub [Gitter](https://gitter.im/jupyterhu
 - [Reporting Issues](https://github.com/jupyterhub/jupyterhub/issues)
 - [JupyterHub tutorial](https://github.com/jupyterhub/jupyterhub-tutorial)
 - [Documentation for JupyterHub](https://jupyterhub.readthedocs.io/en/latest/) | [PDF (latest)](https://media.readthedocs.org/pdf/jupyterhub/latest/jupyterhub.pdf) | [PDF (stable)](https://media.readthedocs.org/pdf/jupyterhub/stable/jupyterhub.pdf)
-- [Documentation for JupyterHub's REST API](https://petstore3.swagger.io/?url=https://raw.githubusercontent.com/jupyter/jupyterhub/HEAD/docs/rest-api.yml#/default)
+- [Documentation for JupyterHub's REST API][rest api]
 - [Documentation for Project Jupyter](http://jupyter.readthedocs.io/en/latest/index.html) | [PDF](https://media.readthedocs.org/pdf/jupyter/latest/jupyter.pdf)
 - [Project Jupyter website](https://jupyter.org)
 - [Project Jupyter community](https://jupyter.org/community)

RELEASE.md

@@ -0,0 +1,74 @@
+# How to make a release
+
+`jupyterhub` is a package [available on
+PyPI](https://pypi.org/project/jupyterhub/) and
+[conda-forge](https://conda-forge.org/).
+These are instructions on how to make a release on PyPI.
+The PyPI release is done automatically by CI when a tag is pushed.
+
+For you to follow along according to these instructions, you need:
+
+- To have push rights to the [jupyterhub GitHub
+  repository](https://github.com/jupyterhub/jupyterhub).
+
+## Steps to make a release
+
+1. Checkout main and make sure it is up to date.
+
+   ```shell
+   ORIGIN=${ORIGIN:-origin} # set to the canonical remote, e.g. 'upstream' if 'origin' is not the official repo
+   git checkout main
+   git fetch $ORIGIN main
+   git reset --hard $ORIGIN/main
+   ```
+
+1. Update [CHANGELOG.md](CHANGELOG.md). Doing this can be made easier with the
+   help of [github-activity](https://github.com/choldgraf/github-activity).
+
+1. Set the `version_info` variable in [\_version.py](jupyterhub/_version.py)
+   appropriately, update `docs/source/_static/rest-api.yml` to match and make a commit.
+
+   ```shell
+   git add jupyterhub/_version.py
+   git add docs/source/_static/rest-api.yml
+   VERSION=...  # e.g. 1.2.3
+   git commit -m "release $VERSION"
+   ```
+
+1. Reset the `version_info` variable in
+   [\_version.py](jupyterhub/_version.py) appropriately with an incremented
+   patch version and a `dev` element, then make a commit.
+
+   ```shell
+   git add jupyterhub/_version.py
+   git add docs/source/_static/rest-api.yml
+   git commit -m "back to dev"
+   ```
+
+1. Push your two commits to main.
+
+   ```shell
+   # first push commits without a tags to ensure the
+   # commits comes through, because a tag can otherwise
+   # be pushed all alone without company of rejected
+   # commits, and we want have our tagged release coupled
+   # with a specific commit in main
+   git push $ORIGIN main
+   ```
+
+1. Create a git tag for the pushed release commit and push it.
+
+   ```shell
+   git tag -a $VERSION -m $VERSION HEAD~1
+
+   # then verify you tagged the right commit
+   git log
+
+   # then push it
+   git push $ORIGIN $VERSION
+   ```
+
+1. Following the release to PyPI, an automated PR should arrive to
+   [conda-forge/jupyterhub-feedstock](https://github.com/conda-forge/jupyterhub-feedstock),
+   check for the tests to succeed on this PR and then merge it to successfully
+   update the package for `conda` on the conda-forge channel.

docs/Makefile

@@ -53,14 +53,6 @@ help:
 clean:
 	rm -rf $(BUILDDIR)/*
 
-node_modules: package.json
-	npm install && touch node_modules
-
-rest-api: source/_static/rest-api/index.html
-
-source/_static/rest-api/index.html: rest-api.yml node_modules
-	npm run rest-api
-
 metrics: source/reference/metrics.rst
 
 source/reference/metrics.rst: generate-metrics.py
@@ -71,7 +63,7 @@ scopes: source/rbac/scope-table.md
 source/rbac/scope-table.md: source/rbac/generate-scope-table.py
 	python3 source/rbac/generate-scope-table.py
 
-html: rest-api metrics scopes
+html: metrics scopes
 	$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
 	@echo
 	@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."

docs/package.json

@@ -1,14 +0,0 @@
-{
-  "name": "jupyterhub-docs-build",
-  "version": "0.8.0",
-  "description": "build JupyterHub swagger docs",
-  "scripts": {
-    "rest-api": "bootprint openapi ./rest-api.yml source/_static/rest-api"
-  },
-  "author": "",
-  "license": "BSD-3-Clause",
-  "devDependencies": {
-    "bootprint": "^1.0.0",
-    "bootprint-openapi": "^1.0.0"
-  }
-}

docs/source/_static/custom.css

@@ -2,3 +2,9 @@
 .navbar-brand {
   height: 4rem !important;
 }
+
+/* hide redundant funky-formatted swagger-ui version */
+
+.swagger-ui .info .title small {
+  display: none !important;
+}

docs/source/_static/rest-api.yml

@@ -1,9 +1,11 @@
-# see me at: https://petstore3.swagger.io/?url=https://raw.githubusercontent.com/jupyterhub/jupyterhub/HEAD/docs/rest-api.yml#/default
+# see me at: https://jupyterhub.readthedocs.io/en/latest/reference/rest-api.html
 swagger: "2.0"
 info:
   title: JupyterHub
   description: The REST API for JupyterHub
-  version: 1.4.0
+  # version should match jupyterhub/_version.py
+  # `make scopes` ensures this is in sync
+  version: 2.0.0b3
   license:
     name: BSD-3-Clause
 schemes: [http, https]
@@ -15,14 +17,17 @@ securityDefinitions:
   oauth2:
     type: oauth2
     flow: accessCode
-    authorizationUrl: "/hub/api/oauth2/authorize" # what are the absolute URIs here? is oauth2 correct here or shall we use just authorizations?
-    tokenUrl: "/hub/api/oauth2/token"
+    # these must be absolute until we update to openapi 3
+    authorizationUrl: "https://hub.example/hub/api/oauth2/authorize"
+    tokenUrl: "https://hub.example/hub/api/oauth2/token"
     scopes: # Generated based on scope table in jupyterhub/scopes.py
       (no_scope): Identify the owner of the requesting entity.
       self:
         The user’s own resources _(metascope for users, resolves to (no_scope)
         for services)_
-      all: Everything that the token-owning entity can access _(metascope for tokens)_
+      inherit:
+        Everything that the token-owning entity can access _(metascope for
+        tokens)_
       admin:users:
         Read, write, create and delete users and their authentication state,
         not including their servers or tokens.
@@ -30,6 +35,8 @@ securityDefinitions:
       users:
         Read and write permissions to user models (excluding servers, tokens
         and authentication state).
+      delete:users: Delete users.
+      list:users: List users, including at least their names.
       read:users:
         Read user models (excluding including servers, tokens and authentication
         state).
@@ -47,14 +54,18 @@ securityDefinitions:
       read:servers:
         Read users’ names and their server models (excluding the server
         state).
+      delete:servers: Stop and delete users' servers.
       tokens: Read, write, create and delete user tokens.
       read:tokens: Read user tokens.
       admin:groups: Read and write group information, create and delete groups.
       groups:
         Read and write group information, including adding/removing users to/from
         groups.
+      list:groups: List groups, including at least their names.
       read:groups: Read group models.
       read:groups:name: Read group names.
+      delete:groups: Delete groups.
+      list:services: List services, including at least their names.
       read:services: Read service models.
       read:services:name: Read service names.
       read:hub: Read detailed information about the Hub.
@@ -174,7 +185,7 @@ paths:
             If unspecified, return all users.
         - name: limit
           in: query
-          requred: false
+          required: false
           type: number
           description: |
             Return a finite number of users.
@@ -779,7 +790,7 @@ paths:
             If unspecified, return all routes.
         - name: limit
           in: query
-          requred: false
+          required: false
           type: number
           description: |
             Return a finite number of routes.
@@ -870,7 +881,7 @@ paths:
       summary: Identify a user or service from an API token
       security:
         - oauth2:
-            - (noscope)
+            - (no_scope)
       parameters:
         - name: token
           in: path

docs/source/api/index.rst

@@ -17,11 +17,6 @@ information on:
 - making an API request programmatically using the requests library
 - learning more about JupyterHub's API
 
-The same JupyterHub API spec, as found here, is available in an interactive form
-`here (on swagger's petstore) <https://petstore3.swagger.io/?url=https://raw.githubusercontent.com/jupyterhub/jupyterhub/HEAD/docs/rest-api.yml#!/default>`__.
-The `OpenAPI Initiative`_ (fka Swagger™) is a project used to describe
-and document RESTful APIs.
-
 JupyterHub API Reference:
 
 .. toctree::

docs/source/conf.py

@@ -215,7 +215,7 @@ if on_rtd:
     # build both metrics and rest-api, since RTD doesn't run make
     from subprocess import check_call as sh
 
-    sh(['make', 'metrics', 'rest-api', 'scopes'], cwd=docs)
+    sh(['make', 'metrics', 'scopes'], cwd=docs)
 
 # -- Spell checking -------------------------------------------------------
 

docs/source/index.rst

@@ -43,7 +43,7 @@ JupyterHub performs the following functions:
   notebook servers
 
 For convenient administration of the Hub, its users, and services,
-JupyterHub also provides a `REST API`_.
+JupyterHub also provides a :doc:`REST API <reference/rest-api>`.
 
 The JupyterHub team and Project Jupyter value our community, and JupyterHub
 follows the Jupyter `Community Guides <https://jupyter.readthedocs.io/en/latest/community/content-community.html>`_.
@@ -155,4 +155,3 @@ Questions? Suggestions?
 
 .. _JupyterHub: https://github.com/jupyterhub/jupyterhub
 .. _Jupyter notebook: https://jupyter-notebook.readthedocs.io/en/latest/
-.. _REST API: https://petstore3.swagger.io/?url=https://raw.githubusercontent.com/jupyterhub/jupyterhub/HEAD/docs/rest-api.yml#!/default

docs/source/rbac/generate-scope-table.py

@@ -5,10 +5,12 @@ from pathlib import Path
 from pytablewriter import MarkdownTableWriter
 from ruamel.yaml import YAML
 
+import jupyterhub
 from jupyterhub.scopes import scope_definitions
 
 HERE = os.path.abspath(os.path.dirname(__file__))
-PARENT = Path(HERE).parent.parent.absolute()
+DOCS = Path(HERE).parent.parent.absolute()
+REST_API_YAML = DOCS.joinpath("source", "_static", "rest-api.yml")
 
 
 class ScopeTableGenerator:
@@ -98,22 +100,24 @@ class ScopeTableGenerator:
 
     def write_api(self):
         """Generates the API description in markdown format and writes it into `rest-api.yml`"""
-        filename = f"{PARENT}/rest-api.yml"
+        filename = REST_API_YAML
         yaml = YAML(typ='rt')
         yaml.preserve_quotes = True
         scope_dict = {}
-        with open(filename, 'r+') as f:
+        with open(filename) as f:
             content = yaml.load(f.read())
-            f.seek(0)
-            for scope in self.scopes:
-                description = self.scopes[scope]['description']
-                doc_description = self.scopes[scope].get('doc_description', '')
-                if doc_description:
-                    description = doc_description
-                scope_dict[scope] = description
-            content['securityDefinitions']['oauth2']['scopes'] = scope_dict
+
+        content["info"]["version"] = jupyterhub.__version__
+        for scope in self.scopes:
+            description = self.scopes[scope]['description']
+            doc_description = self.scopes[scope].get('doc_description', '')
+            if doc_description:
+                description = doc_description
+            scope_dict[scope] = description
+        content['securityDefinitions']['oauth2']['scopes'] = scope_dict
+
+        with open(filename, 'w') as f:
             yaml.dump(content, f)
-            f.truncate()
 
 
 def main():

docs/source/reference/index.rst

@@ -16,6 +16,7 @@ what happens under-the-hood when you deploy and configure your JupyterHub.
    proxy
    separate-proxy
    rest
+   rest-api
    server-api
    monitoring
    database

docs/source/reference/rest-api.md

@@ -0,0 +1,27 @@
+# JupyterHub REST API
+
+Below is an interactive view of JupyterHub's OpenAPI specification.
+
+<!-- client-rendered openapi UI copied from FastAPI -->
+
+<link type="text/css" rel="stylesheet" href="https://cdn.jsdelivr.net/npm/swagger-ui-dist@3/swagger-ui.css">
+<script src="https://cdn.jsdelivr.net/npm/swagger-ui-dist@3/swagger-ui-bundle.js"></script>
+<!-- `SwaggerUIBundle` is now available on the page -->
+
+<!-- render the ui here -->
+<div id="openapi-ui"></div>
+
+<script>
+const ui = SwaggerUIBundle({
+  url: '../_static/rest-api.yml',
+  dom_id: '#openapi-ui',
+  presets: [
+    SwaggerUIBundle.presets.apis,
+    SwaggerUIBundle.SwaggerUIStandalonePreset
+  ],
+  layout: "BaseLayout",
+  deepLinking: true,
+  showExtensions: true,
+  showCommonExtensions: true,
+});
+</script>

docs/source/reference/rest-api.rst

@@ -1,14 +0,0 @@
-:orphan:
-
-===================
-JupyterHub REST API
-===================
-
-.. this doc exists as a resolvable link target
-.. which _static files are not
-
-.. meta::
-    :http-equiv=refresh: 0;url=../_static/rest-api/index.html
-
-The rest API docs are `here <../_static/rest-api/index.html>`_
-if you are not redirected automatically.

docs/source/reference/rest.md

@@ -302,12 +302,8 @@ or kubernetes pods.
 
 ## Learn more about the API
 
-You can see the full [JupyterHub REST API][] for details. This REST API Spec can
-be viewed in a more [interactive style on swagger's petstore][].
-Both resources contain the same information and differ only in its display.
-Note: The Swagger specification is being renamed the [OpenAPI Initiative][].
+You can see the full [JupyterHub REST API][] for details.
 
-[interactive style on swagger's petstore]: https://petstore3.swagger.io/?url=https://raw.githubusercontent.com/jupyterhub/jupyterhub/HEAD/docs/rest-api.yml#!/default
 [openapi initiative]: https://www.openapis.org/
 [jupyterhub rest api]: ./rest-api
 [jupyter notebook rest api]: https://petstore3.swagger.io/?url=https://raw.githubusercontent.com/jupyter/notebook/HEAD/notebook/services/api/api.yaml

docs/test_docs.py

@@ -0,0 +1,45 @@
+import sys
+from pathlib import Path
+from subprocess import run
+
+from ruamel.yaml import YAML
+
+yaml = YAML(typ="safe")
+
+here = Path(__file__).absolute().parent
+root = here.parent
+
+
+def test_rest_api_version():
+    version_py = root.joinpath("jupyterhub", "_version.py")
+    rest_api_yaml = root.joinpath("docs", "source", "_static", "rest-api.yml")
+    ns = {}
+    with version_py.open() as f:
+        exec(f.read(), {}, ns)
+    jupyterhub_version = ns["__version__"]
+
+    with rest_api_yaml.open() as f:
+        rest_api = yaml.load(f)
+        rest_api_version = rest_api["info"]["version"]
+
+    assert jupyterhub_version == rest_api_version
+
+
+def test_restapi_scopes():
+    run([sys.executable, "source/rbac/generate-scope-table.py"], cwd=here, check=True)
+    run(
+        ['pre-commit', 'run', 'prettier', '--files', 'source/_static/rest-api.yml'],
+        cwd=here,
+        check=False,
+    )
+    run(
+        [
+            "git",
+            "diff",
+            "--no-pager",
+            "--exit-code",
+            str(here.joinpath("source", "_static", "rest-api.yml")),
+        ],
+        cwd=here,
+        check=True,
+    )

jupyterhub/_version.py

@@ -1,13 +1,14 @@
 """JupyterHub version info"""
 # Copyright (c) Jupyter Development Team.
 # Distributed under the terms of the Modified BSD License.
-
 version_info = (
     2,
     0,
     0,
     "b3",  # release (b1, rc1, or "" for final or dev)
     # "dev",  # dev or nothing for beta/rc/stable releases
+    # when updating, make sure to update version in docs/source/_static/rest-api.yml as well
+    # `cd docs; make scopes` will do this
 )
 
 # pep 440 version: no dot before beta/rc, but before .dev