Merge pull request #2080 from minrk/rel-0.9.2

prepare to release 0.9.2

CONTRIBUTING.md

@@ -95,4 +95,4 @@ make html
 
 ```bash
 open build/html/index.html
-```
+```

README.md

@@ -11,8 +11,8 @@
 
 
 [![PyPI](https://img.shields.io/pypi/v/jupyterhub.svg)](https://pypi.python.org/pypi/jupyterhub)
-[![Documentation Status](https://readthedocs.org/projects/jupyterhub/badge/?version=latest)](http://jupyterhub.readthedocs.org/en/latest/?badge=latest)
-[![Documentation Status](http://readthedocs.org/projects/jupyterhub/badge/?version=0.7.2)](http://jupyterhub.readthedocs.io/en/0.7.2/?badge=0.7.2)
+[![Documentation Status](https://readthedocs.org/projects/jupyterhub/badge/?version=latest)](https://jupyterhub.readthedocs.org/en/latest/?badge=latest)
+[![Documentation Status](http://readthedocs.org/projects/jupyterhub/badge/?version=0.7.2)](https://jupyterhub.readthedocs.io/en/0.7.2/?badge=0.7.2)
 [![Build Status](https://travis-ci.org/jupyterhub/jupyterhub.svg?branch=master)](https://travis-ci.org/jupyterhub/jupyterhub)
 [![Circle CI](https://circleci.com/gh/jupyterhub/jupyterhub.svg?style=shield&circle-token=b5b65862eb2617b9a8d39e79340b0a6b816da8cc)](https://circleci.com/gh/jupyterhub/jupyterhub)
 [![codecov.io](https://codecov.io/github/jupyterhub/jupyterhub/coverage.svg?branch=master)](https://codecov.io/github/jupyterhub/jupyterhub?branch=master)
@@ -124,7 +124,7 @@ more configuration of the system.
 
 ## Configuration
 
-The [Getting Started](http://jupyterhub.readthedocs.io/en/latest/getting-started/index.html) section of the
+The [Getting Started](https://jupyterhub.readthedocs.io/en/latest/getting-started/index.html) section of the
 documentation explains the common steps in setting up JupyterHub.
 
 The [**JupyterHub tutorial**](https://github.com/jupyterhub/jupyterhub-tutorial)
@@ -233,7 +233,7 @@ our JupyterHub [Gitter](https://gitter.im/jupyterhub/jupyterhub) channel.
 
 - [Reporting Issues](https://github.com/jupyterhub/jupyterhub/issues)
 - [JupyterHub tutorial](https://github.com/jupyterhub/jupyterhub-tutorial)
-- [Documentation for JupyterHub](http://jupyterhub.readthedocs.io/en/latest/) | [PDF (latest)](https://media.readthedocs.org/pdf/jupyterhub/latest/jupyterhub.pdf) | [PDF (stable)](https://media.readthedocs.org/pdf/jupyterhub/stable/jupyterhub.pdf)
+- [Documentation for JupyterHub](https://jupyterhub.readthedocs.io/en/latest/) | [PDF (latest)](https://media.readthedocs.org/pdf/jupyterhub/latest/jupyterhub.pdf) | [PDF (stable)](https://media.readthedocs.org/pdf/jupyterhub/stable/jupyterhub.pdf)
 - [Documentation for JupyterHub's REST API](http://petstore.swagger.io/?url=https://raw.githubusercontent.com/jupyter/jupyterhub/master/docs/rest-api.yml#/default)
 - [Documentation for Project Jupyter](http://jupyter.readthedocs.io/en/latest/index.html) | [PDF](https://media.readthedocs.org/pdf/jupyter/latest/jupyter.pdf)
 - [Project Jupyter website](https://jupyter.org)

dev-requirements.txt

@@ -8,3 +8,6 @@ pytest>=3.3
 notebook
 requests-mock
 virtualenv
+# temporary pin of attrs for jsonschema 0.3.0a1
+# seems to be a pip bug
+attrs>=17.4.0

docs/environment.yml

@@ -1,3 +1,5 @@
+# ReadTheDocs uses the `environment.yaml` so make sure to update that as well
+# if you change the dependencies of JupyterHub in the various `requirements.txt`
 name: jhub_docs
 channels:
   - conda-forge
@@ -17,3 +19,4 @@ dependencies:
   - recommonmark==0.4.0
   - async_generator
   - prometheus_client
+  - attrs>=17.4.0

docs/requirements.txt

@@ -1,3 +1,5 @@
+# ReadTheDocs uses the `environment.yaml` so make sure to update that as well
+# if you change this file
 -r ../requirements.txt
 sphinx>=1.7
 recommonmark==0.4.0

docs/source/changelog.md

@@ -9,6 +9,16 @@ command line for details.
 
 ## 0.9
 
+### [0.9.2] 2018-08-10
+
+JupyterHub 0.9.2 contains small bugfixes and improvements.
+
+- Documentation and example improvements
+- Add `Spawner.consecutive_failure_limit` config for aborting the Hub if too many spawns fail in a row.
+- Fix for handling SIGTERM when run with asyncio (tornado 5)
+- Windows compatibility fixes
+
+
 ### [0.9.1] 2018-07-04
 
 JupyterHub 0.9.1 contains a number of small bugfixes on top of 0.9.
@@ -392,7 +402,8 @@ Fix removal of `/login` page in 0.4.0, breaking some OAuth providers.
 First preview release
 
 
-[Unreleased]: https://github.com/jupyterhub/jupyterhub/compare/0.9.1...HEAD
+[Unreleased]: https://github.com/jupyterhub/jupyterhub/compare/0.9.2...HEAD
+[0.9.2]: https://github.com/jupyterhub/jupyterhub/compare/0.9.1...0.9.2
 [0.9.1]: https://github.com/jupyterhub/jupyterhub/compare/0.9.0...0.9.1
 [0.9.0]: https://github.com/jupyterhub/jupyterhub/compare/0.8.1...0.9.0
 [0.8.1]: https://github.com/jupyterhub/jupyterhub/compare/0.8.0...0.8.1

docs/source/getting-started/authenticators-users-basics.md

@@ -96,4 +96,4 @@ A generic implementation, which you can use for OAuth authentication
 with any provider, is also available.
 
 [PAM]: https://en.wikipedia.org/wiki/Pluggable_authentication_module
-[OAuthenticator]: https://github.com/jupyterhub/oauthenticator
+[OAuthenticator]: https://github.com/jupyterhub/oauthenticator

docs/source/reference/authenticators.md

@@ -226,5 +226,5 @@ Beginning with version 0.8, JupyterHub is an OAuth provider.
 [OAuth]: https://en.wikipedia.org/wiki/OAuth
 [GitHub OAuth]: https://developer.github.com/v3/oauth/
 [OAuthenticator]: https://github.com/jupyterhub/oauthenticator
-[pre_spawn_start(user, spawner)]: http://jupyterhub.readthedocs.io/en/latest/api/auth.html#jupyterhub.auth.Authenticator.pre_spawn_start
-[post_spawn_stop(user, spawner)]: http://jupyterhub.readthedocs.io/en/latest/api/auth.html#jupyterhub.auth.Authenticator.post_spawn_stop
+[pre_spawn_start(user, spawner)]: https://jupyterhub.readthedocs.io/en/latest/api/auth.html#jupyterhub.auth.Authenticator.pre_spawn_start
+[post_spawn_stop(user, spawner)]: https://jupyterhub.readthedocs.io/en/latest/api/auth.html#jupyterhub.auth.Authenticator.post_spawn_stop

docs/source/reference/config-ghoauth.md

@@ -79,4 +79,4 @@ export OAUTH_CALLBACK_URL=https://example.com/hub/oauth_callback
 export CONFIGPROXY_AUTH_TOKEN=super-secret
 # append log output to log file /var/log/jupyterhub.log
 jupyterhub -f /etc/jupyterhub/jupyterhub_config.py &>> /var/log/jupyterhub.log
-```
+```

docs/source/reference/services.md

@@ -204,10 +204,10 @@ which implements the requests to the Hub.
 To use HubAuth, you must set the `.api_token`, either programmatically when constructing the class,
 or via the `JUPYTERHUB_API_TOKEN` environment variable.
 
-Most of the logic for authentication implementation is found in the
-[`HubAuth.user_for_cookie`](services.auth.html#jupyterhub.services.auth.HubAuth.user_for_cookie)
-and in the
-[`HubAuth.user_for_token`](services.auth.html#jupyterhub.services.auth.HubAuth.user_for_token)
+Most of the logic for authentication implementation is found in the 
+[`HubAuth.user_for_cookie`][HubAuth.user_for_cookie] 
+and in the 
+[`HubAuth.user_for_token`][HubAuth.user_for_token]
 methods, which makes a request of the Hub, and returns:
 
 - None, if no user could be identified, or
@@ -359,14 +359,16 @@ and taking note of the following process:
    ```
 
 An example of using an Externally-Managed Service and authentication is
-in [nbviewer README]_ section on securing the notebook viewer,
+in [nbviewer README][nbviewer example] section on securing the notebook viewer,
 and an example of its configuration is found [here](https://github.com/jupyter/nbviewer/blob/master/nbviewer/providers/base.py#L94).
-nbviewer can also be run as a Hub-Managed Service as described [nbviewer README]_
+nbviewer can also be run as a Hub-Managed Service as described [nbviewer README][nbviewer example]
 section on securing the notebook viewer.
 
 
 [requests]: http://docs.python-requests.org/en/master/
 [services_auth]: ../api/services.auth.html
 [HubAuth]: ../api/services.auth.html#jupyterhub.services.auth.HubAuth
+[HubAuth.user_for_cookie]: ../api/services.auth.html#jupyterhub.services.auth.HubAuth.user_for_cookie
+[HubAuth.user_for_token]: ../api/services.auth.html#jupyterhub.services.auth.HubAuth.user_for_token
 [HubAuthenticated]: ../api/services.auth.html#jupyterhub.services.auth.HubAuthenticated
 [nbviewer example]: https://github.com/jupyter/nbviewer#securing-the-notebook-viewer

examples/bootstrap-script/README.md

@@ -130,4 +130,4 @@ else
 fi
 
 exit 0
-```
+```

examples/cull-idle/cull_idle_servers.py

@@ -186,10 +186,16 @@ def cull_idle(url, api_token, inactive_limit, cull_users=False, max_age=0, concu
                 log_name, format_td(age), format_td(inactive))
             return False
 
+        if server_name:
+            # culling a named server
+            delete_url = url + "/users/%s/servers/%s" % (
+                quote(user['name']), quote(server['name'])
+            )
+        else:
+            delete_url = url + '/users/%s/server' % quote(user['name'])
+
         req = HTTPRequest(
-            url=url + '/users/%s/server' % quote(user['name']),
-            method='DELETE',
-            headers=auth_header,
+            url=delete_url, method='DELETE', headers=auth_header,
         )
         resp = yield fetch(req)
         if resp.code == 202:

examples/service-announcement/README.md

@@ -0,0 +1,60 @@
+
+# Simple Announcement Service Example
+
+This is a simple service that allows administrators to manage announcements
+that appear when JupyterHub renders pages.
+
+To run the service as a hub-managed service simply include in your JupyterHub
+configuration file something like:
+
+    c.JupyterHub.services = [
+            {
+                'name': 'announcement',
+                'url': 'http://127.0.0.1:8888',
+                'command': ["python", "-m", "announcement"],
+            }
+    ]
+
+This starts the announcements service up at `/services/announcement` when
+JupyterHub launches.  By default the announcement text is empty.
+
+The `announcement` module has a configurable port (default 8888) and an API
+prefix setting.  By default the API prefix is `JUPYTERHUB_SERVICE_PREFIX` if
+that environment variable is set or `/` if it is not.
+
+## Managing the Announcement
+
+Admin users can set the announcement text with an API token:
+
+    $ curl -X POST -H "Authorization: token <token>"                        \
+        -d "{'announcement':'JupyterHub will be upgraded on August 14!'}"   \
+        https://.../services/announcement
+
+Anyone can read the announcement:
+
+    $ curl https://.../services/announcement | python -m json.tool
+    {
+        announcement: "JupyterHub will be upgraded on August 14!",
+        timestamp: "...",
+        user: "..."
+    }
+
+The time the announcement was posted is recorded in the `timestamp` field and
+the user who posted the announcement is recorded in the `user` field.
+
+To clear the announcement text, just DELETE.  Only admin users can do this.
+
+    $ curl -X POST -H "Authorization: token <token>"                        \
+        https://.../services/announcement
+
+## Seeing the Announcement in JupyterHub
+
+To be able to render the announcement, include the provide `page.html` template
+that extends the base `page.html` template.  Set `c.JupyterHub.template_paths`
+in JupyterHub's configuration to include the path to the extending template.
+The template changes the `announcement` element and does a JQuery `$.get()` call
+to retrieve the announcement text.
+
+JupyterHub's configurable announcement template variables can be set for various
+pages like login, logout, spawn, and home.  Including the template provided in
+this example overrides all of those.

examples/service-announcement/announcement.py

@@ -0,0 +1,73 @@
+
+import argparse
+import datetime
+import json
+import os
+
+from jupyterhub.services.auth import HubAuthenticated
+from tornado import escape, gen, ioloop, web
+
+
+class AnnouncementRequestHandler(HubAuthenticated, web.RequestHandler):
+    """Dynamically manage page announcements"""
+
+    hub_users = []
+    allow_admin = True
+
+    def initialize(self, storage):
+        """Create storage for announcement text"""
+        self.storage = storage
+
+    @web.authenticated
+    def post(self):
+        """Update announcement"""
+        doc = escape.json_decode(self.request.body)
+        self.storage["announcement"] = doc["announcement"]
+        self.storage["timestamp"] = datetime.datetime.now().isoformat()
+        self.storage["user"] = user["name"]
+        self.write_to_json(self.storage)
+
+    def get(self):
+        """Retrieve announcement"""
+        self.write_to_json(self.storage)
+
+    @web.authenticated
+    def delete(self):
+        """Clear announcement"""
+        self.storage["announcement"] = ""
+        self.write_to_json(self.storage)
+
+    def write_to_json(self, doc):
+        """Write dictionary document as JSON"""
+        self.set_header("Content-Type", "application/json; charset=UTF-8")
+        self.write(escape.utf8(json.dumps(doc)))
+
+
+def main():
+    args = parse_arguments()
+    application = create_application(**vars(args))
+    application.listen(args.port)
+    ioloop.IOLoop.current().start()
+
+
+def parse_arguments():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--api-prefix", "-a",
+            default=os.environ.get("JUPYTERHUB_SERVICE_PREFIX", "/"),
+            help="application API prefix")
+    parser.add_argument("--port", "-p",
+            default=8888,
+            help="port for API to listen on",
+            type=int)
+    return parser.parse_args()
+
+
+def create_application(api_prefix="/", 
+        handler=AnnouncementRequestHandler,
+        **kwargs):
+    storage = dict(announcement="", timestamp="", user="")
+    return web.Application([(api_prefix, handler, dict(storage=storage))])
+
+
+if __name__ == "__main__":
+    main()

examples/service-announcement/jupyterhub_config.py

@@ -0,0 +1,15 @@
+
+# To run the announcement service managed by the hub, add this.
+
+c.JupyterHub.services = [
+        {
+            'name': 'announcement',
+            'url': 'http://127.0.0.1:8888',
+            'command': ["python", "-m", "announcement"],
+        }
+]
+
+# The announcements need to get on the templates somehow, see page.html
+# for an example of how to do this.
+
+c.JupyterHub.template_paths = ["templates"]

examples/service-announcement/templates/page.html

@@ -0,0 +1,14 @@
+{% extends "templates/page.html" %}
+{% block announcement %}
+<div class="container text-center announcement">
+</div>
+{% endblock %}
+
+{% block script %}
+{{ super() }}
+<script>
+$.get("/services/announcement/", function(data) {
+    $(".announcement").html(data["announcement"]);
+});
+</script>
+{% endblock %}

jupyterhub/_version.py

@@ -6,8 +6,8 @@
 version_info = (
     0,
     9,
-    1,
-    "",  # release (b1, rc1, or "" for final)
+    2,
+    "",  # release (b1, rc1, or "" for final or dev)
     # "dev",  # dev or nothing
 )
 

jupyterhub/app.py

@@ -1921,8 +1921,7 @@ class JupyterHub(Application):
 
     def sigterm(self, signum, frame):
         self.log.critical("Received SIGTERM, shutting down")
-        self.io_loop.stop()
-        self.atexit()
+        raise SystemExit(128 + signum)
 
     _atexit_ran = False
 
@@ -1932,6 +1931,7 @@ class JupyterHub(Application):
             return
         self._atexit_ran = True
         # run the cleanup step (in a new loop, because the interrupted one is unclean)
+        asyncio.set_event_loop(asyncio.new_event_loop())
         IOLoop.clear_current()
         loop = IOLoop()
         loop.make_current()

jupyterhub/handlers/base.py

@@ -657,6 +657,7 @@ class BaseHandler(RequestHandler):
         # hook up spawner._spawn_future so that other requests can await
         # this result
         finish_spawn_future = spawner._spawn_future = maybe_future(finish_user_spawn())
+
         def _clear_spawn_future(f):
             # clear spawner._spawn_future when it's done
             # keep an exception around, though, to prevent repeated implicit spawns
@@ -665,10 +666,44 @@ class BaseHandler(RequestHandler):
                 spawner._spawn_future = None
             # Now we're all done. clear _spawn_pending flag
             spawner._spawn_pending = False
+
         finish_spawn_future.add_done_callback(_clear_spawn_future)
 
+        # when spawn finishes (success or failure)
+        # update failure count and abort if consecutive failure limit
+        # is reached
+        def _track_failure_count(f):
+            if f.exception() is None:
+                # spawn succeeded, reset failure count
+                self.settings['failure_count'] = 0
+                return
+            # spawn failed, increment count and abort if limit reached
+            self.settings.setdefault('failure_count', 0)
+            self.settings['failure_count'] += 1
+            failure_count = self.settings['failure_count']
+            failure_limit = spawner.consecutive_failure_limit
+            if failure_limit and 1 < failure_count < failure_limit:
+                self.log.warning(
+                    "%i consecutive spawns failed.  "
+                    "Hub will exit if failure count reaches %i before succeeding",
+                    failure_count, failure_limit,
+                )
+            if failure_limit and failure_count >= failure_limit:
+                self.log.critical(
+                    "Aborting due to %i consecutive spawn failures", failure_count
+                )
+                # abort in 2 seconds to allow pending handlers to resolve
+                # mostly propagating errors for the current failures
+                def abort():
+                    raise SystemExit(1)
+                IOLoop.current().call_later(2, abort)
+
+        finish_spawn_future.add_done_callback(_track_failure_count)
+
         try:
-            await gen.with_timeout(timedelta(seconds=self.slow_spawn_timeout), finish_spawn_future)
+            await gen.with_timeout(
+                timedelta(seconds=self.slow_spawn_timeout), finish_spawn_future
+            )
         except gen.TimeoutError:
             # waiting_for_response indicates server process has started,
             # but is yet to become responsive.
@@ -866,6 +901,11 @@ class PrefixRedirectHandler(BaseHandler):
     """
     def get(self):
         uri = self.request.uri
+        # Since self.base_url will end with trailing slash.
+        # Ensure uri will end with trailing slash when matching
+        # with self.base_url.
+        if not uri.endswith('/'):
+            uri += '/'
         if uri.startswith(self.base_url):
             path = self.request.uri[len(self.base_url):]
         else:

jupyterhub/proxy.py

@@ -447,6 +447,14 @@ class ConfigurableHTTPProxy(Proxy):
 
     _check_running_callback = Any(help="PeriodicCallback to check if the proxy is running")
 
+    def _check_pid(self, pid):
+        if os.name == 'nt':
+            import psutil
+            if not psutil.pid_exists(pid):
+                raise ProcessLookupError
+        else:
+            os.kill(pid, 0)
+
     def __init__(self, **kwargs):
         super().__init__(**kwargs)
         # check for required token if proxy is external
@@ -471,7 +479,7 @@ class ConfigurableHTTPProxy(Proxy):
             return
 
         try:
-            os.kill(pid, 0)
+            self._check_pid(pid)
         except ProcessLookupError:
             self.log.warning("Proxy no longer running at pid=%s", pid)
             self._remove_pid_file()
@@ -486,12 +494,12 @@ class ConfigurableHTTPProxy(Proxy):
                 break
             time.sleep(1)
             try:
-                os.kill(pid, 0)
+                self._check_pid(pid)
             except ProcessLookupError:
                 break
 
         try:
-            os.kill(pid, 0)
+            self._check_pid(pid)
         except ProcessLookupError:
             self.log.warning("Stopped proxy at pid=%s", pid)
             self._remove_pid_file()

jupyterhub/services/service.py

@@ -42,6 +42,7 @@ A hub-managed service with no URL::
 import copy
 import pipes
 import shutil
+import os
 from subprocess import Popen
 
 from traitlets import (
@@ -106,6 +107,8 @@ class _ServiceSpawner(LocalProcessSpawner):
     def start(self):
         """Start the process"""
         env = self.get_env()
+        if os.name == 'nt':
+            env['SYSTEMROOT'] = os.environ['SYSTEMROOT']
         cmd = self.cmd
 
         self.log.info("Spawning %s", ' '.join(pipes.quote(s) for s in cmd))

jupyterhub/spawner.py

@@ -195,6 +195,19 @@ class Spawner(LoggingConfigurable):
         """
     ).tag(config=True)
 
+    consecutive_failure_limit = Integer(
+        0,
+        help="""
+        Maximum number of consecutive failures to allow before
+        shutting down JupyterHub.
+
+        This helps JupyterHub recover from a certain class of problem preventing launch
+        in contexts where the Hub is automatically restarted (e.g. systemd, docker, kubernetes).
+
+        A limit of 0 means no limit and consecutive failures will not be tracked.
+        """,
+    ).tag(config=True)
+
     start_timeout = Integer(60,
         help="""
         Timeout (in seconds) before giving up on starting of single-user server.