Merge master into rbac

docs/source/getting-started/authenticators-users-basics.md

@@ -91,6 +91,7 @@ JupyterHub's [OAuthenticator][] currently supports the following
 popular services:
 
 - Auth0
+- Azure AD
 - Bitbucket
 - CILogon
 - GitHub

docs/source/installation-guide-hard.rst

@@ -3,4 +3,4 @@
 JupyterHub the hard way
 =======================
 
-This guide has moved to https://github.com/manics/jupyterhub-the-hard-way/blob/jupyterhub-alternative-doc/docs/installation-guide-hard.md
+This guide has moved to https://github.com/jupyterhub/jupyterhub-the-hard-way/blob/master/docs/installation-guide-hard.md

examples/service-fastapi/Dockerfile

@@ -0,0 +1,13 @@
+FROM jupyterhub/jupyterhub
+
+# Create test user (PAM auth) and install single-user Jupyter 
+RUN useradd testuser --create-home --shell /bin/bash
+RUN echo 'testuser:passwd' | chpasswd
+RUN pip install jupyter 
+
+COPY app ./app
+COPY jupyterhub_config.py .
+COPY requirements.txt /tmp/requirements.txt
+RUN pip install -r /tmp/requirements.txt
+
+CMD ["jupyterhub", "--ip", "0.0.0.0"]

examples/service-fastapi/README.md

@@ -0,0 +1,107 @@
+# Fastapi
+
+[FastAPI](https://fastapi.tiangolo.com/) is a popular new web framework attractive for its type hinting, async support, automatic doc generation (Swagger), and more. Their [Feature highlights](https://fastapi.tiangolo.com/features/) sum it up nicely.
+
+# Swagger UI with OAuth demo
+
+![Fastapi Service Example](./fastapi_example.gif)
+
+# Try it out locally
+
+1. Install `fastapi` and other dependencies, then launch Jupyterhub
+
+```
+pip install -r requirements.txt
+jupyterhub --ip=127.0.0.1
+```
+
+2. Visit http://127.0.0.1:8000/services/fastapi or http://127.0.0.1:8000/services/fastapi/docs
+
+3. Try interacting programmatically. If you create a new token in your control panel or pull out the `JUPYTERHUB_API_TOKEN` in the single user environment, you can skip the third step here.
+
+```
+$ curl -X GET http://127.0.0.1:8000/services/fastapi/
+{"Hello":"World"}
+
+$ curl -X GET http://127.0.0.1:8000/services/fastapi/me
+{"detail":"Must login with token parameter, cookie, or header"}
+
+$ curl -X POST http://127.0.0.1:8000/hub/api/authorizations/token \
+       -d '{"username": "myname", "password": "mypasswd!"}' \
+       | jq '.token'
+"3fee13ce6d2845da9bd5f2c2170d3428"
+
+$ curl -X GET http://127.0.0.1:8000/services/fastapi/me \
+       -H "Authorization: Bearer 3fee13ce6d2845da9bd5f2c2170d3428" \
+       | jq .
+{
+  "name": "myname",
+  "admin": false,
+  "groups": [],
+  "server": null,
+  "pending": null,
+  "last_activity": "2021-04-07T18:05:11.587638+00:00",
+  "servers": null
+}
+```
+
+# Try it out in Docker
+
+1. Build and run the Docker image locally
+
+```bash
+sudo docker build . -t service-fastapi
+sudo docker run -it -p 8000:8000 service-fastapi
+```
+
+2. Visit http://127.0.0.1:8000/services/fastapi/docs. When going through the OAuth flow or getting a token from the control panel, you can log in with `testuser` / `passwd`.
+
+# PUBLIC_HOST
+
+If you are running your service behind a proxy, or on a Docker / Kubernetes infrastructure, you might run into an error during OAuth that says `Mismatching redirect URI`. In the Jupterhub logs, there will be a warning along the lines of: `[W 2021-04-06 23:40:06.707 JupyterHub provider:498] Redirect uri https://jupyterhub.my.cloud/services/fastapi/oauth_callback != /services/fastapi/oauth_callback`. This happens because Swagger UI adds the request host, as seen in the browser, to the Authorization URL.
+
+To solve that problem, the `oauth_redirect_uri` value in the service initialization needs to match what Swagger will auto-generate and what the service will use when POST'ing to `/oauth2/token`. In this example, setting the `PUBLIC_HOST` environment variable to your public-facing Hub domain (e.g. `https://jupyterhub.my.cloud`) should make it work.
+
+# Notes on security.py
+
+FastAPI has a concept of a [dependency injection](https://fastapi.tiangolo.com/tutorial/dependencies) using a `Depends` object (and a subclass `Security`) that is automatically instantiated/executed when it is a parameter for your endpoint routes. You can utilize a `Depends` object for re-useable common parameters or authentication mechanisms like the [`get_user`](https://fastapi.tiangolo.com/tutorial/security/get-current-user) pattern.
+
+JupyterHub OAuth has three ways to authenticate: a `token` url parameter; a `Authorization: Bearer <token>` header; and a (deprecated) `jupyterhub-services` cookie. FastAPI has helper functions that let us create `Security` (dependency injection) objects for each of those. When you need to allow multiple / optional authentication dependencies (`Security` objects), then you can use the argument `auto_error=False` and it will return `None` instead of raising an `HTTPException`.
+
+Endpoints that need authentication (`/me` and `/debug` in this example) can leverage the `get_user` pattern and effectively pull the user model from the Hub API when a request has authenticated with cookie / token / header all using the simple syntax,
+
+```python
+from .security import get_current_user
+from .models import User
+
+@router.get("/new_endpoint")
+async def new_endpoint(user: User = Depends(get_current_user)):
+    "Function that needs to work with an authenticated user"
+    return {"Hello": user.name}
+```
+
+# Notes on client.py
+
+FastAPI is designed to be an asynchronous web server, so the interactions with the Hub API should be made asynchronously as well. Instead of using `requests` to get user information from a token/cookie, this example uses [`httpx`](https://www.python-httpx.org/). `client.py` defines a small function that creates a `Client` (equivalent of `requests.Session`) with the Hub API url as it's `base_url` and adding the `JUPYTERHUB_API_TOKEN` to every header.
+
+Consider this a very minimal alternative to using `jupyterhub.services.auth.HubOAuth`
+
+```python
+# client.py
+import os
+
+def get_client():
+    base_url = os.environ["JUPYTERHUB_API_URL"]
+    token = os.environ["JUPYTERHUB_API_TOKEN"]
+    headers = {"Authorization": "Bearer %s" % token}
+    return httpx.AsyncClient(base_url=base_url, headers=headers)
+```
+
+```python
+# other modules
+from .client import get_client
+
+async with get_client() as client:
+    resp = await client.get('/endpoint')
+    ...
+```

examples/service-fastapi/app/__init__.py

@@ -0,0 +1 @@
+from .app import app

examples/service-fastapi/app/app.py

@@ -0,0 +1,25 @@
+import os
+
+from fastapi import FastAPI
+
+from .service import router
+
+### When managed by Jupyterhub, the actual endpoints
+### will be served out prefixed by /services/:name.
+### One way to handle this with FastAPI is to use an APIRouter.
+### All routes are defined in service.py
+
+app = FastAPI(
+    title="Example FastAPI Service",
+    version="0.1",
+    ### Serve out Swagger from the service prefix (<hub>/services/:name/docs)
+    openapi_url=router.prefix + "/openapi.json",
+    docs_url=router.prefix + "/docs",
+    redoc_url=router.prefix + "/redoc",
+    ### Add our service client id to the /docs Authorize form automatically
+    swagger_ui_init_oauth={"clientId": os.environ["JUPYTERHUB_CLIENT_ID"]},
+    ### Default /docs/oauth2 redirect will cause Hub
+    ### to raise oauth2 redirect uri mismatch errors
+    swagger_ui_oauth2_redirect_url=os.environ["JUPYTERHUB_OAUTH_CALLBACK_URL"],
+)
+app.include_router(router)

examples/service-fastapi/app/client.py

@@ -0,0 +1,11 @@
+import os
+
+import httpx
+
+
+# a minimal alternative to using HubOAuth class
+def get_client():
+    base_url = os.environ["JUPYTERHUB_API_URL"]
+    token = os.environ["JUPYTERHUB_API_TOKEN"]
+    headers = {"Authorization": "Bearer %s" % token}
+    return httpx.AsyncClient(base_url=base_url, headers=headers)

examples/service-fastapi/app/models.py

@@ -0,0 +1,46 @@
+from datetime import datetime
+from typing import Any
+from typing import List
+from typing import Optional
+
+from pydantic import BaseModel
+
+
+# https://jupyterhub.readthedocs.io/en/stable/_static/rest-api/index.html
+class Server(BaseModel):
+    name: str
+    ready: bool
+    pending: Optional[str]
+    url: str
+    progress_url: str
+    started: datetime
+    last_activity: datetime
+    state: Optional[Any]
+    user_options: Optional[Any]
+
+
+class User(BaseModel):
+    name: str
+    admin: bool
+    groups: List[str]
+    server: Optional[str]
+    pending: Optional[str]
+    last_activity: datetime
+    servers: Optional[List[Server]]
+
+
+# https://stackoverflow.com/questions/64501193/fastapi-how-to-use-httpexception-in-responses
+class AuthorizationError(BaseModel):
+    detail: str
+
+
+class HubResponse(BaseModel):
+    msg: str
+    request_url: str
+    token: str
+    response_code: int
+    hub_response: dict
+
+
+class HubApiError(BaseModel):
+    detail: HubResponse

examples/service-fastapi/app/security.py

@@ -0,0 +1,61 @@
+import os
+
+from fastapi import HTTPException
+from fastapi import Security
+from fastapi import status
+from fastapi.security import OAuth2AuthorizationCodeBearer
+from fastapi.security.api_key import APIKeyQuery
+
+from .client import get_client
+from .models import User
+
+### Endpoints can require authentication using Depends(get_current_user)
+### get_current_user will look for a token in url params or
+### Authorization: bearer token (header).
+### Hub technically supports cookie auth too, but it is deprecated so
+### not being included here.
+auth_by_param = APIKeyQuery(name="token", auto_error=False)
+
+auth_url = os.environ["PUBLIC_HOST"] + "/hub/api/oauth2/authorize"
+auth_by_header = OAuth2AuthorizationCodeBearer(
+    authorizationUrl=auth_url, tokenUrl="get_token", auto_error=False
+)
+### ^^ The flow for OAuth2 in Swagger is that the "authorize" button
+### will redirect user (browser) to "auth_url", which is the Hub login page.
+### After logging in, the browser will POST to our internal /get_token endpoint
+### with the auth code.  That endpoint POST's to Hub /oauth2/token with
+### our client_secret (JUPYTERHUB_API_TOKEN) and that code to get an
+### access_token, which it returns to browser, which places in Authorization header.
+
+### For consideration: optimize performance with a cache instead of
+### always hitting the Hub api?
+async def get_current_user(
+    auth_by_param: str = Security(auth_by_param),
+    auth_by_header: str = Security(auth_by_header),
+):
+    token = auth_by_param or auth_by_header
+    if token is None:
+        raise HTTPException(
+            status.HTTP_401_UNAUTHORIZED,
+            detail="Must login with token parameter or Authorization bearer header",
+        )
+
+    async with get_client() as client:
+        endpoint = "/user"
+        # normally we auth to Hub API with service api token,
+        # but this time auth as the user token to get user model
+        headers = {"Authorization": f"Bearer {token}"}
+        resp = await client.get(endpoint, headers=headers)
+        if resp.is_error:
+            raise HTTPException(
+                status.HTTP_400_BAD_REQUEST,
+                detail={
+                    "msg": "Error getting user info from token",
+                    "request_url": str(resp.request.url),
+                    "token": token,
+                    "response_code": resp.status_code,
+                    "hub_response": resp.json(),
+                },
+            )
+    user = User(**resp.json())
+    return user

examples/service-fastapi/app/service.py

@@ -0,0 +1,70 @@
+import os
+
+from fastapi import APIRouter
+from fastapi import Depends
+from fastapi import Form
+from fastapi import Request
+
+from .client import get_client
+from .models import AuthorizationError
+from .models import HubApiError
+from .models import User
+from .security import get_current_user
+
+# APIRouter prefix cannot end in /
+service_prefix = os.getenv("JUPYTERHUB_SERVICE_PREFIX", "").rstrip("/")
+router = APIRouter(prefix=service_prefix)
+
+
+@router.post("/get_token", include_in_schema=False)
+async def get_token(code: str = Form(...)):
+    "Callback function for OAuth2AuthorizationCodeBearer scheme"
+    # The only thing we need in this form post is the code
+    # Everything else we can hardcode / pull from env
+    async with get_client() as client:
+        redirect_uri = (
+            os.environ["PUBLIC_HOST"] + os.environ["JUPYTERHUB_OAUTH_CALLBACK_URL"],
+        )
+        data = {
+            "client_id": os.environ["JUPYTERHUB_CLIENT_ID"],
+            "client_secret": os.environ["JUPYTERHUB_API_TOKEN"],
+            "grant_type": "authorization_code",
+            "code": code,
+            "redirect_uri": redirect_uri,
+        }
+        resp = await client.post("/oauth2/token", data=data)
+    ### resp.json() is {'access_token': <token>, 'token_type': 'Bearer'}
+    return resp.json()
+
+
+@router.get("/")
+async def index():
+    "Non-authenticated function that returns {'Hello': 'World'}"
+    return {"Hello": "World"}
+
+
+# response_model and responses dict translate to OpenAPI (Swagger) hints
+# compare and contrast what the /me endpoint looks like in Swagger vs /debug
+@router.get(
+    "/me",
+    response_model=User,
+    responses={401: {'model': AuthorizationError}, 400: {'model': HubApiError}},
+)
+async def me(user: User = Depends(get_current_user)):
+    "Authenticated function that returns the User model"
+    return user
+
+
+@router.get("/debug")
+async def index(request: Request, user: User = Depends(get_current_user)):
+    """
+    Authenticated function that returns a few pieces of debug
+     * Environ of the service process
+     * Request headers
+     * User model
+    """
+    return {
+        "env": dict(os.environ),
+        "headers": dict(request.headers),
+        "user": user,
+    }

examples/service-fastapi/jupyterhub_config.py

@@ -0,0 +1,31 @@
+import os
+import warnings
+
+# When Swagger performs OAuth2 in the browser, it will set
+# the request host + relative path as the redirect uri, causing a
+# uri mismatch if the oauth_redirect_uri is just the relative path
+# is set in the c.JupyterHub.services entry (as per default).
+# Therefore need to know the request host ahead of time.
+if "PUBLIC_HOST" not in os.environ:
+    msg = (
+        "env PUBLIC_HOST is not set, defaulting to http://127.0.0.1:8000.  "
+        "This can cause problems with OAuth.  "
+        "Set PUBLIC_HOST to your public (browser accessible) host."
+    )
+    warnings.warn(msg)
+    public_host = "http://127.0.0.1:8000"
+else:
+    public_host = os.environ["PUBLIC_HOST"].rstrip('/')
+service_name = "fastapi"
+oauth_redirect_uri = f"{public_host}/services/{service_name}/oauth_callback"
+
+c.JupyterHub.services = [
+    {
+        "name": service_name,
+        "url": "http://127.0.0.1:10202",
+        "command": ["uvicorn", "app:app", "--port", "10202"],
+        "admin": True,
+        "oauth_redirect_uri": oauth_redirect_uri,
+        "environment": {"PUBLIC_HOST": public_host},
+    }
+]

examples/service-fastapi/requirements.txt

@@ -0,0 +1,4 @@
+fastapi
+httpx
+python-multipart
+uvicorn

jupyterhub/alembic/versions/1cebaf56856c_session_id.py

@@ -23,7 +23,7 @@ tables = ('oauth_access_tokens', 'oauth_codes')
 
 def add_column_if_table_exists(table, column):
     engine = op.get_bind().engine
-    if table not in engine.table_names():
+    if table not in sa.inspect(engine).get_table_names():
         # table doesn't exist, no need to upgrade
         # because jupyterhub will create it on launch
         logger.warning("Skipping upgrade of absent table: %s", table)

jupyterhub/alembic/versions/4dc2d5a8c53c_user_options.py

@@ -17,7 +17,8 @@ from jupyterhub.orm import JSONDict
 
 
 def upgrade():
-    tables = op.get_bind().engine.table_names()
+    engine = op.get_bind().engine
+    tables = sa.inspect(engine).get_table_names()
     if 'spawners' in tables:
         op.add_column('spawners', sa.Column('user_options', JSONDict()))
 

jupyterhub/alembic/versions/56cc5a70207e_token_tracking.py

@@ -20,7 +20,8 @@ logger = logging.getLogger('alembic')
 
 
 def upgrade():
-    tables = op.get_bind().engine.table_names()
+    engine = op.get_bind().engine
+    tables = sa.inspect(engine).get_table_names()
     op.add_column('api_tokens', sa.Column('created', sa.DateTime(), nullable=True))
     op.add_column(
         'api_tokens', sa.Column('last_activity', sa.DateTime(), nullable=True)

jupyterhub/alembic/versions/99a28a4418e1_user_created.py

@@ -31,7 +31,7 @@ def upgrade():
         % (now,)
     )
 
-    tables = c.engine.table_names()
+    tables = sa.inspect(c.engine).get_table_names()
 
     if 'spawners' in tables:
         op.add_column('spawners', sa.Column('started', sa.DateTime, nullable=True))

jupyterhub/alembic/versions/d68c98b66cd4_client_description.py

@@ -16,7 +16,8 @@ import sqlalchemy as sa
 
 
 def upgrade():
-    tables = op.get_bind().engine.table_names()
+    engine = op.get_bind().engine
+    tables = sa.inspect(engine).get_table_names()
     if 'oauth_clients' in tables:
         op.add_column(
             'oauth_clients', sa.Column('description', sa.Unicode(length=1023))

jupyterhub/app.py

@@ -9,6 +9,7 @@ import json
 import logging
 import os
 import re
+import secrets
 import signal
 import socket
 import sys
@@ -382,6 +383,42 @@ class JupyterHub(Application):
         Default is two weeks.
         """,
     ).tag(config=True)
+
+    oauth_token_expires_in = Integer(
+        help="""Expiry (in seconds) of OAuth access tokens.
+
+        The default is to expire when the cookie storing them expires,
+        according to `cookie_max_age_days` config.
+
+        These are the tokens stored in cookies when you visit
+        a single-user server or service.
+        When they expire, you must re-authenticate with the Hub,
+        even if your Hub authentication is still valid.
+        If your Hub authentication is valid,
+        logging in may be a transparent redirect as you refresh the page.
+        
+        This does not affect JupyterHub API tokens in general,
+        which do not expire by default.
+        Only tokens issued during the oauth flow
+        accessing services and single-user servers are affected.
+
+        .. versionadded:: 1.4
+            OAuth token expires_in was not previously configurable.
+        .. versionchanged:: 1.4
+            Default now uses cookie_max_age_days so that oauth tokens
+            which are generally stored in cookies,
+            expire when the cookies storing them expire.
+            Previously, it was one hour.
+        """,
+        config=True,
+    )
+
+    @default("oauth_token_expires_in")
+    def _cookie_max_age_seconds(self):
+        """default to cookie max age, where these tokens are stored"""
+        # convert cookie max age days to seconds
+        return int(self.cookie_max_age_days * 24 * 3600)
+
     redirect_to_server = Bool(
         True, help="Redirect user to server (if running), instead of control panel."
     ).tag(config=True)
@@ -1502,7 +1539,7 @@ class JupyterHub(Application):
         if not secret:
             secret_from = 'new'
             self.log.debug("Generating new %s", trait_name)
-            secret = os.urandom(COOKIE_SECRET_BYTES)
+            secret = secrets.token_bytes(COOKIE_SECRET_BYTES)
 
         if secret_file and secret_from == 'new':
             # if we generated a new secret, store it in the secret_file
@@ -2253,6 +2290,7 @@ class JupyterHub(Application):
             lambda: self.db,
             url_prefix=url_path_join(base_url, 'api/oauth2'),
             login_url=url_path_join(base_url, 'login'),
+            token_expires_in=self.oauth_token_expires_in,
         )
 
     def cleanup_oauth_clients(self):

jupyterhub/oauth/provider.py

@@ -558,20 +558,25 @@ class JupyterHubOAuthServer(WebApplicationServer):
 
         hash its client_secret before putting it in the database.
         """
-        # clear existing clients with same ID
+        # Update client if it already exists, else create it
-        for orm_client in self.db.query(orm.OAuthClient).filter_by(
+        # Sqlalchemy doesn't have a good db agnostic UPSERT,
-            identifier=client_id
+        # so we do this manually. It's protected inside a
-        ):
+        # transaction, so should fail if there are multiple
-            self.db.delete(orm_client)
+        # rows with the same identifier.
-        self.db.commit()
+        orm_client = (
-
+            self.db.query(orm.OAuthClient).filter_by(identifier=client_id).one_or_none()
+        )
+        if orm_client is None:
             orm_client = orm.OAuthClient(
                 identifier=client_id,
-            secret=hash_token(client_secret),
-            redirect_uri=redirect_uri,
-            description=description,
             )
             self.db.add(orm_client)
+            app_log.info(f'Creating oauth client {client_id}')
+        else:
+            app_log.info(f'Updating oauth client {client_id}')
+        orm_client.secret = hash_token(client_secret)
+        orm_client.redirect_uri = redirect_uri
+        orm_client.description = description
         self.db.commit()
 
     def fetch_by_client_id(self, client_id):
@@ -579,9 +584,9 @@ class JupyterHubOAuthServer(WebApplicationServer):
         return self.db.query(orm.OAuthClient).filter_by(identifier=client_id).first()
 
 
-def make_provider(session_factory, url_prefix, login_url):
+def make_provider(session_factory, url_prefix, login_url, **oauth_server_kwargs):
     """Make an OAuth provider"""
     db = session_factory()
     validator = JupyterHubRequestValidator(db)
-    server = JupyterHubOAuthServer(db, validator)
+    server = JupyterHubOAuthServer(db, validator, **oauth_server_kwargs)
     return server

jupyterhub/orm.py

@@ -873,7 +873,7 @@ def check_db_revision(engine):
     - Empty databases are tagged with the current revision
     """
     # Check database schema version
-    current_table_names = set(engine.table_names())
+    current_table_names = set(inspect(engine).get_table_names())
     my_table_names = set(Base.metadata.tables.keys())
 
     from .dbutil import _temp_alembic_ini

jupyterhub/singleuser/mixins.py

@@ -12,7 +12,9 @@ import asyncio
 import json
 import os
 import random
+import secrets
 import warnings
+from datetime import datetime
 from datetime import timezone
 from textwrap import dedent
 from urllib.parse import urlparse
@@ -251,7 +253,7 @@ class SingleUserNotebookAppMixin(Configurable):
     cookie_secret = Bytes()
 
     def _cookie_secret_default(self):
-        return os.urandom(32)
+        return secrets.token_bytes(32)
 
     user = CUnicode().tag(config=True)
     group = CUnicode().tag(config=True)

jupyterhub/tests/test_services_auth.py

@@ -475,6 +475,10 @@ async def test_oauth_logout(app, mockservice_url):
     session_id = s.cookies['jupyterhub-session-id']
 
     assert len(auth_tokens()) == 1
+    token = auth_tokens()[0]
+    assert token.expires_in is not None
+    # verify that oauth_token_expires_in has its desired effect
+    assert abs(app.oauth_token_expires_in - token.expires_in) < 30
 
     # hit hub logout URL
     r = await s.get(public_url(app, path='hub/logout'))

jupyterhub/user.py

@@ -826,10 +826,7 @@ class User:
             try:
                 await maybe_future(spawner.run_post_stop_hook())
             except:
-                spawner.clear_state()
+                self.log.exception("Error in Spawner.post_stop_hook for %s", self)
-                spawner.orm_spawner.state = spawner.get_state()
-                self.db.commit()
-                raise
             spawner.clear_state()
             spawner.orm_spawner.state = spawner.get_state()
             self.db.commit()

jupyterhub/utils.py

@@ -8,6 +8,7 @@ import hashlib
 import inspect
 import os
 import random
+import secrets
 import socket
 import ssl
 import sys
@@ -325,7 +326,7 @@ def hash_token(token, salt=8, rounds=16384, algorithm='sha512'):
     """
     h = hashlib.new(algorithm)
     if isinstance(salt, int):
-        salt = b2a_hex(os.urandom(salt))
+        salt = b2a_hex(secrets.token_bytes(salt))
     if isinstance(salt, bytes):
         bsalt = salt
         salt = salt.decode('utf8')