Merge branch 'main' into pr/small-services-docs-fixes

docs/source/explanation/oauth.md

@@ -1,3 +1,5 @@
+(jupyterhub-oauth)=
+
 # JupyterHub and OAuth
 
 JupyterHub uses [OAuth 2](https://oauth.net/2/) as an internal mechanism for authenticating users.

docs/source/reference/gallery-jhub-deployments.md

@@ -16,8 +16,6 @@ Please submit pull requests to update information or to add new institutions or
 
 - [BIDS - Berkeley Institute for Data Science](https://bids.berkeley.edu/)
 
-  - [Teaching with Jupyter notebooks and JupyterHub](https://bids.berkeley.edu/resources/videos/teaching-ipythonjupyter-notebooks-and-jupyterhub)
-
 - [Data 8](http://data8.org/)
 
   - [GitHub organization](https://github.com/data-8)

docs/source/reference/services.md

@@ -4,20 +4,19 @@
 
 ## Definition of a Service
 
-When working with JupyterHub, a **Service** is defined as a process that interacts
-with the Hub's REST API. A Service may perform a specific
-action or task. For example, the following tasks can each be a unique Service:
+When working with JupyterHub, a **Service** is defined as something (usually a process) that can interact with the Hub's REST API.
+A Service may perform a specific action or task.
+For example, the following tasks can each be a unique Service:
 
-- shutting down individuals' single user notebook servers that have been idle
-  for some time
-- registering additional web servers which should use the Hub's authentication
-  and be served behind the Hub's proxy.
+- shutting down individuals' single user notebook servers that have been idle for some time
+- an additional web application which uses the Hub as an OAuth provider to authenticate and authorize user access
+- a script run once in a while, which performs any API action
+- automating requests to running user servers, such as activity data collection
 
-Two key features help define a Service:
+Two key features help differentiate Services:
 
 - Is the Service **managed** by JupyterHub?
-- Does the Service have a web server that should be added to the proxy's
-  table?
+- Does the Service have a web server that should be added to the proxy's table?
 
 Currently, these characteristics distinguish two types of Services:
 
@@ -30,24 +29,32 @@ Currently, these characteristics distinguish two types of Services:
 A Service may have the following properties:
 
 - `name: str` - the name of the service
-- `admin: bool (default - false)` - whether the service should have
-  administrative privileges
-- `url: str (default - None)` - The URL where the service is/should be. If a
-  url is specified for where the Service runs its own web server,
-  the service will be added to the proxy at `/services/:name`
-- `api_token: str (default - None)` - For Externally-Managed Services you need to specify
-  an API token to perform API requests to the Hub
+- `url: str (default - None)` - The URL where the service should be running (from the proxy's perspective).
+  Typically a localhost URL for Hub-managed services.
+  If a url is specified,
+  the service will be added to the proxy at `/services/:name`.
+- `api_token: str (default - None)` - For Externally-Managed Services,
+  you need to specify an API token to perform API requests to the Hub.
+  For Hub-managed services, this token is generated at startup,
+  and available via `$JUPYTERHUB_API_TOKEN`.
+  For OAuth services, this is the client secret.
 - `display: bool (default - True)` - When set to true, display a link to the
-  service's URL under the 'Services' dropdown in user's hub home page.
-
+  service's URL under the 'Services' dropdown in users' hub home page.
+  Only has an effect if `url` is also specified.
 - `oauth_no_confirm: bool (default - False)` - When set to true,
   skip the OAuth confirmation page when users access this service.
-
   By default, when users authenticate with a service using JupyterHub,
   they are prompted to confirm that they want to grant that service
   access to their credentials.
   Skipping the confirmation page is useful for admin-managed services that are considered part of the Hub
   and shouldn't need extra prompts for login.
+- `oauth_client_id: str (default - 'service-$name')` -
+  This never needs to be set, but you can specify a service's OAuth client id.
+  It must start with `service-`.
+- `oauth_redirect_uri: str (default: '/services/:name/oauth_redirect')` -
+  Set the OAuth redirect URI.
+  Required if the redirect URI differs from the default or the service is not to be added to the proxy at `/services/:name`
+  (i.e. `url` is not set, but there is still a public web service using OAuth).
 
 If a service is also to be managed by the Hub, it has a few extra options:
 
@@ -55,19 +62,19 @@ If a service is also to be managed by the Hub, it has a few extra options:
   externally. - If a command is specified for launching the Service, the Service will
   be started and managed by the Hub.
 - `environment: dict` - additional environment variables for the Service.
-- `user: str` - the name of a system user to manage the Service. If
-  unspecified, run as the same user as the Hub.
+- `user: str` - the name of a system user to manage the Service.
+  If unspecified, run as the same user as the Hub.
 
 ## Hub-Managed Services
 
 A **Hub-Managed Service** is started by the Hub, and the Hub is responsible
-for the Service's actions. A Hub-Managed Service can only be a local
+for the Service's operation. A Hub-Managed Service can only be a local
 subprocess of the Hub. The Hub will take care of starting the process and
 restart the service if the service stops.
 
-While Hub-Managed Services share some similarities with notebook Spawners,
+While Hub-Managed Services share some similarities with single-user server Spawners,
 there are no plans for Hub-Managed Services to support the same spawning
-abstractions as a notebook Spawner.
+abstractions as a Spawner.
 
 If you wish to run a Service in a Docker container or other deployment
 environments, the Service can be registered as an
@@ -174,6 +181,106 @@ c.JupyterHub.services = [
 In this case, the `url` field will be passed along to the Service as
 `JUPYTERHUB_SERVICE_URL`.
 
+## Service credentials
+
+A service has direct access to the Hub API via its `api_token`.
+Exactly what actions the service can take are governed by the service's [role assignments](define-role-target):
+
+```python
+c.JupyterHub.services = [
+    {
+        "name": "user-lister",
+        "command": ["python3", "/path/to/user-lister"],
+    }
+]
+
+c.JupyterHub.load_roles = [
+    {
+        "name": "list-users",
+        "scopes": ["list:users", "read:users"],
+        "services": ["user-lister"]
+    }
+]
+```
+
+When a service has a configured URL or explicit `oauth_client_id` or `oauth_redirect_uri`, it can operate as an [OAuth client](jupyterhub-oauth).
+When a user visits an oauth-authenticated service,
+completion of authentication results in issuing an oauth token.
+
+This token is:
+
+- owned by the authenticated user
+- associated with the oauth client of the service
+- governed by the service's `oauth_client_allowed_scopes` configuration
+
+This token enables the service to act _on behalf of_ the user.
+
+When an oauthenticated service makes a request to the Hub (or other Hub-authenticated service), it has two credentials available to authenticate the request:
+
+- the service's own `api_token`, which acts _as_ the service,
+  and is governed by the service's own role assignments.
+- the user's oauth token issued to the service during the oauth flow,
+  which acts _as_ the user.
+
+Choosing which one to use depends on "who" should be considered taking the action represented by the request.
+
+A service's own permissions governs how it can act without any involvement of a user.
+The service's `oauth_client_allowed_scopes` configuration allows individual users to _delegate_ permission for the service to act on their behalf.
+This allows services to have little to no permissions of their own,
+but allow users to take actions _via_ the service,
+using their own credentials.
+
+An example of such a service would be a web application for instructors,
+presenting a dashboard of actions which can be taken for students in their courses.
+The service would need no permission to do anything with the JupyterHub API on its own,
+but it could employ the user's oauth credentials to list users,
+manage student servers, etc.
+
+This service might look like:
+
+```python
+c.JupyterHub.services = [
+    {
+        "name": "grader-dashboard",
+        "command": ["python3", "/path/to/grader-dashboard"],
+        "url": "http://127.0.0.1:12345",
+        "oauth_client_allowed_scopes": [
+            "list:users",
+            "read:users",
+        ]
+    }
+]
+
+c.JupyterHub.load_roles = [
+    {
+        "name": "grader",
+        "scopes": [
+            "list:users!group=class-a",
+            "read:users!group=class-a",
+            "servers!group=class-a",
+            "access:servers!group=class-a",
+            "access:services",
+        ],
+        "groups": ["graders"]
+    }
+]
+```
+
+In this example, the `grader-dashboard` service does not have permission to take any actions with the Hub API on its own because it has not been assigned any role.
+But when a grader accesses the service,
+the dashboard will have a token with permission to list and read information about any users that the grader can access.
+The dashboard will _not_ have permission to do additional things as the grader.
+
+The dashboard will be able to:
+
+- list users in class A (`list:users!group=class-a`)
+- read information about users in class A (`read:users!group=class-a`)
+
+The dashboard will _not_ be able to:
+
+- start, stop, or access user servers (`servers`, `access:servers`), even though the grader has this permission (it's not in `oauth_client_allowed_scopes`)
+- take any action without the grader granting permission via oauth
+
 ## Adding or removing services at runtime
 
 Only externally-managed services can be added at runtime by using JupyterHub’s REST API.
@@ -281,11 +388,11 @@ To use HubAuth, you must set the `.api_token` instance variable. This can be
 done via the HubAuth constructor, direct assignment to a HubAuth object, or via the
 `JUPYTERHUB_API_TOKEN` environment variable. A number of the examples in the
 root of the jupyterhub git repository set the `JUPYTERHUB_API_TOKEN` variable
-so consider having a look at those for futher reading
+so consider having a look at those for further reading
 ([cull-idle](https://github.com/jupyterhub/jupyterhub/tree/master/examples/cull-idle),
 [external-oauth](https://github.com/jupyterhub/jupyterhub/tree/master/examples/external-oauth),
 [service-notebook](https://github.com/jupyterhub/jupyterhub/tree/master/examples/service-notebook)
-and [service-whoiami](https://github.com/jupyterhub/jupyterhub/tree/master/examples/service-whoami))
+and [service-whoami](https://github.com/jupyterhub/jupyterhub/tree/master/examples/service-whoami))
 
 Most of the logic for authentication implementation is found in the
 {meth}`.HubAuth.user_for_token` methods,

examples/external-oauth/jupyterhub_config.py

@@ -9,7 +9,6 @@ if not api_token:
     )
 
 # tell JupyterHub to register the service as an external oauth client
-
 c.JupyterHub.services = [
     {
         'name': 'external-oauth',
@@ -18,3 +17,26 @@ c.JupyterHub.services = [
         'oauth_redirect_uri': 'http://127.0.0.1:5555/oauth_callback',
     }
 ]
+
+# Grant all JupyterHub users ability to access services
+c.JupyterHub.load_roles = [
+    {
+        'name': 'user',
+        'description': 'Allow all users to access all services',
+        'scopes': ['access:services', 'self'],
+    }
+]
+
+# Boilerplate to make sure the example runs - this is not relevant
+# to external oauth services.
+
+# Allow authentication with any username and any password
+from jupyterhub.auth import DummyAuthenticator
+
+c.JupyterHub.authenticator_class = DummyAuthenticator
+
+# Optionally set a global password that all users must use
+# c.DummyAuthenticator.password = "your_password"
+
+# only listen on localhost for testing.
+c.JupyterHub.bind_url = 'http://127.0.0.1:8000'

jupyterhub/apihandlers/base.py

@@ -532,7 +532,7 @@ class APIHandler(BaseHandler):
         if next_offset < total_count:
             # if there's a next page
             next_url_parsed = urlparse(self.request.full_url())
-            query = parse_qs(next_url_parsed.query)
+            query = parse_qs(next_url_parsed.query, keep_blank_values=True)
             query['offset'] = [next_offset]
             query['limit'] = [limit]
             next_url_parsed = next_url_parsed._replace(

jupyterhub/tests/test_api.py

@@ -7,7 +7,7 @@ import uuid
 from copy import deepcopy
 from datetime import datetime, timedelta
 from unittest import mock
-from urllib.parse import quote, urlparse
+from urllib.parse import parse_qs, quote, urlparse
 
 import pytest
 from pytest import fixture, mark
@@ -268,20 +268,22 @@ def max_page_limit(app):
 @mark.user
 @mark.role
 @mark.parametrize(
-    "n, offset, limit, accepts_pagination, expected_count",
+    "n, offset, limit, accepts_pagination, expected_count, include_stopped_servers",
     [
-        (10, None, None, False, 10),
-        (10, None, None, True, 10),
-        (10, 5, None, True, 5),
-        (10, 5, None, False, 5),
-        (10, 5, 1, True, 1),
-        (10, 10, 10, True, 0),
+        (10, None, None, False, 10, False),
+        (10, None, None, True, 10, False),
+        (10, 5, None, True, 5, False),
+        (10, 5, None, False, 5, False),
+        (10, None, 5, True, 5, True),
+        (10, 5, 1, True, 1, True),
+        (10, 10, 10, True, 0, False),
         (  # default page limit, pagination expected
             30,
             None,
             None,
             True,
             'default',
+            False,
         ),
         (
             # default max page limit, pagination not expected
@@ -290,6 +292,7 @@ def max_page_limit(app):
             None,
             False,
             'max',
+            False,
         ),
         (
             # limit exceeded
@@ -298,6 +301,7 @@ def max_page_limit(app):
             500,
             False,
             'max',
+            False,
         ),
     ],
 )
@@ -310,6 +314,7 @@ async def test_get_users_pagination(
     expected_count,
     default_page_limit,
     max_page_limit,
+    include_stopped_servers,
 ):
     db = app.db
 
@@ -336,6 +341,11 @@ async def test_get_users_pagination(
     if limit:
         params['limit'] = limit
     url = url_concat(url, params)
+    if include_stopped_servers:
+        # assumes limit is set. There doesn't seem to be a way to set valueless query
+        # params using url_cat
+        url += "&include_stopped_servers"
+
     headers = auth_header(db, 'admin')
     if accepts_pagination:
         headers['Accept'] = PAGINATION_MEDIA_TYPE
@@ -348,6 +358,11 @@ async def test_get_users_pagination(
             "_pagination",
         }
         pagination = response["_pagination"]
+        if include_stopped_servers and pagination["next"]:
+            next_query = parse_qs(
+                urlparse(pagination["next"]["url"]).query, keep_blank_values=True
+            )
+            assert "include_stopped_servers" in next_query
         users = response["items"]
     else:
         users = response