0.8.0b5

improve reporting of spawn failure

.dockerignore

@@ -4,3 +4,7 @@ jupyterhub_cookie_secret
 jupyterhub.sqlite
 jupyterhub_config.py
 node_modules
+docs
+.git
+dist
+build

.gitignore

@@ -3,7 +3,7 @@ node_modules
 *~
 .cache
 .DS_Store
-build
+/build
 dist
 docs/_build
 docs/source/_static/rest-api

.travis.yml

@@ -45,3 +45,5 @@ matrix:
       env: JUPYTERHUB_TEST_DB_URL=mysql+mysqlconnector://root@127.0.0.1/jupyterhub
     - python: 3.6
       env: JUPYTERHUB_TEST_DB_URL=postgresql://postgres@127.0.0.1/jupyterhub
+  allow_failures:
+    - python: nightly

MANIFEST.in

@@ -10,6 +10,7 @@ graft onbuild
 graft jupyterhub
 graft scripts
 graft share
+graft singleuser
 
 # Documentation
 graft docs

README.md

@@ -99,7 +99,7 @@ more configuration of the system.
 
 ## Configuration
 
-The [Getting Started](http://jupyterhub.readthedocs.io/en/latest/getting-started.html) section of the
+The [Getting Started](http://jupyterhub.readthedocs.io/en/latest/getting-started/index.html) section of the
 documentation explains the common steps in setting up JupyterHub.
 
 The [**JupyterHub tutorial**](https://github.com/jupyterhub/jupyterhub-tutorial)

docs/source/api/services.auth.rst

@@ -16,6 +16,12 @@ Module: :mod:`jupyterhub.services.auth`
 .. autoconfigurable:: HubAuth
     :members:
 
+:class:`HubOAuth`
+----------------
+
+.. autoconfigurable:: HubOAuth
+    :members:
+
 
 :class:`HubAuthenticated`
 -------------------------
@@ -23,3 +29,13 @@ Module: :mod:`jupyterhub.services.auth`
 .. autoclass:: HubAuthenticated
     :members:
 
+:class:`HubOAuthenticated`
+-------------------------
+
+.. autoclass:: HubOAuthenticated
+
+:class:`HubOAuthCallbackHandler`
+--------------------------------
+
+.. autoclass:: HubOAuthCallbackHandler
+

docs/source/changelog.md

@@ -7,12 +7,90 @@ command line for details.
 
 ## [Unreleased] 0.8
 
+JupyterHub 0.8 is a big release!
+
+Perhaps the biggest change is the use of OAuth to negotiate authentication
+between the Hub and single-user services.
+Due to this change, it is important that the single-user server
+and Hub are both running the same version of JupyterHub.
+If you are using containers (e.g. via DockerSpawner or KubeSpawner),
+this means upgrading jupyterhub in your user images at the same time as the Hub.
+In most cases, a
+
+    pip install jupyterhub==version
+
+in your Dockerfile is sufficient.
+
 #### Added
 
+- JupyterHub now defined a `.Proxy` API for custom
+  proxy implementations other than the default.
+  The defaults are unchanged,
+  but configuration of the proxy is now done on the `ConfigurableHTTPProxy` class instead of the top-level JupyterHub.
+  TODO: docs for writing a custom proxy.
+- Single-user servers and services
+  (anything that uses HubAuth)
+  can now accept token-authenticated requests via the Authentication header.
+- Authenticators can now store state in the Hub's database.
+  To do so, the `.authenticate` method should return a dict of the form
+
+  ```python
+  {
+      'username': 'name'
+      'state': {}
+  }
+  ```
+
+  This data will be encrypted and requires `JUPYTERHUB_CRYPT_KEY` environment variable to be set
+  and the `Authenticator.enable_auth_state` flag to be True.
+  If these are not set, auth_state returned by the Authenticator will not be stored.
+- There is preliminary support for multiple (named) servers per user in the REST API.
+  Named servers can be created via API requests, but there is currently no UI for managing them.
+- Add `LocalProcessSpawner.popen_kwargs` and `LocalProcessSpawner.shell_cmd`
+  for customizing how user server processes are launched.
+- Add `Authenticator.auto_login` flag for skipping the "Login with..." page explicitly.
+- Add `JupyterHub.hub_connect_ip` configuration
+  for the ip that should be used when connecting to the Hub.
+  This is promoting (and deprecating) `DockerSpawner.hub_ip_connect`
+  for use by all Spawners.
+- Add `Spawner.pre_spawn_hook(spawner)` hook for customizing
+  pre-spawn events.
+- Add `JupyterHub.active_server_limit` and `JupyterHub.concurrent_spawn_limit`
+  for limiting the total number of running user servers and the number of pending spawns, respectively.
+
+
 #### Changed
 
+- more arguments to spawners are now passed via environment variables (`.get_env()`)
+  rather than CLI arguments (`.get_args()`)
+- internally generated tokens no longer get extra hash rounds,
+  significantly speeding up authentication.
+  The hash rounds were deemed unnecessary because the tokens were already
+  generated with high entropy.
+- `JUPYTERHUB_API_TOKEN` env is available at all times,
+  rather than being removed during single-user start.
+  The token is now accessible to kernel processes,
+  enabling user kernels to make authenticated API requests to Hub-authenticated services.
+- Cookie secrets should be 32B hex instead of large base64 secrets.
+- pycurl is used by default, if available.
+
 #### Fixed
 
+So many things fixed!
+
+- Collisions are checked when users are renamed
+- Fix bug where OAuth authenticators could not logout users
+  due to being redirected right back through the login process.
+- If there are errors loading your config files,
+  JupyterHub will refuse to start with an informative error.
+  Previously, the bad config would be ignored and JupyterHub would launch with default configuration.
+- Raise 403 error on unauthorized user rather than redirect to login,
+  which could cause redirect loop.
+- Set `httponly` on cookies because it's prudent.
+- Improve support for MySQL as the database backend
+- Many race conditions and performance problems under heavy load have been fixed.
+- Fix alembic tagging of database schema versions.
+
 #### Removed
 
 - End support for Python 3.3 

docs/source/contributor-list.md

@@ -31,6 +31,7 @@ contribution on JupyterHub:
 - JamiesHQ
 - jbweston
 - jdavidheiser
+- jencabral
 - jhamrick
 - josephtate
 - kinuax

docs/source/index.rst

@@ -67,6 +67,8 @@ Contents
 **Tutorials**
 
 * :doc:`tutorials/index`
+* :doc:`tutorials/upgrade-dot-eight`
+* `Zero to JupyterHub with Kubernetes <https://zero-to-jupyterhub.readthedocs.io/en/latest/>`_
 
 **Troubleshooting**
 

docs/source/reference/config-examples.md

@@ -49,9 +49,6 @@ c.JupyterHub.cookie_secret_file = pjoin(runtime_dir, 'cookie_secret')
 c.JupyterHub.db_url = pjoin(runtime_dir, 'jupyterhub.sqlite')
 # or `--db=/path/to/jupyterhub.sqlite` on the command-line
 
-# put the log file in /var/log
-c.JupyterHub.extra_log_file = '/var/log/jupyterhub.log'
-
 # use GitHub OAuthenticator for local users
 c.JupyterHub.authenticator_class = 'oauthenticator.LocalGitHubOAuthenticator'
 c.GitHubOAuthenticator.oauth_callback_url = os.environ['OAUTH_CALLBACK_URL']
@@ -79,7 +76,8 @@ export GITHUB_CLIENT_ID=github_id
 export GITHUB_CLIENT_SECRET=github_secret
 export OAUTH_CALLBACK_URL=https://example.com/hub/oauth_callback
 export CONFIGPROXY_AUTH_TOKEN=super-secret
-jupyterhub -f /etc/jupyterhub/jupyterhub_config.py
+# append log output to log file /var/log/jupyterhub.log
+jupyterhub -f /etc/jupyterhub/jupyterhub_config.py &>> /var/log/jupyterhub.log
 ```
 
 ## Using nginx reverse proxy

docs/source/tutorials/index.rst

@@ -4,4 +4,11 @@ Tutorials
 This section provides links to documentation that helps a user do a specific
 task.
 
-- `Zero to JupyterHub with Kubernetes <https://zero-to-jupyterhub.readthedocs.io/en/latest/>`_
+* :doc:`upgrade-dot-eight`
+* `Zero to JupyterHub with Kubernetes <https://zero-to-jupyterhub.readthedocs.io/en/latest/>`_
+
+.. toctree::
+   :maxdepth: 1
+   :hidden:
+
+   upgrade-dot-eight

docs/source/tutorials/upgrade-dot-eight.rst

@@ -0,0 +1,93 @@
+.. upgrade-dot-eight:
+
+Upgrading to JupyterHub version 0.8
+===================================
+
+This document will assist you in upgrading an existing JupyterHub deployment
+from version 0.7 to version 0.8.
+
+Upgrade checklist
+-----------------
+
+0. Review the release notes. Review any deprecated features and pay attention
+   to any backwards incompatible changes
+1. Backup JupyterHub database:
+    - ``jupyterhub.sqlite`` when using the default sqlite database
+    - Your JupyterHub database when using an RDBMS
+2. Backup the existing JupyterHub configuration file: ``jupyterhub_config.py``
+3. Shutdown the Hub
+4. Upgrade JupyterHub
+    - ``pip install -U jupyterhub`` when using ``pip``
+    - ``conda upgrade jupyterhub`` when using ``conda``
+5. Upgrade the database using run ```jupyterhub upgrade-db``
+6. Update the JupyterHub configuration file ``jupyterhub_config.py``
+
+Backup JupyterHub database
+--------------------------
+
+To prevent unintended loss of data or configuration information, you should
+back up the JupyterHub database (the default SQLite database or a RDBMS
+database using PostgreSQL, MySQL, or others supported by SQLAlchemy):
+
+- If using the default SQLite database, back up the ``jupyterhub.sqlite``
+  database.
+- If using an RDBMS database such as PostgreSQL, MySQL, or other supported by
+  SQLAlchemy, back up the JupyterHub database.
+
+.. note::
+
+    Losing the Hub database is often not a big deal. Information that resides only
+    in the Hub database includes:
+
+    - active login tokens (user cookies, service tokens)
+    - users added via GitHub UI, instead of config files
+    - info about running servers
+
+    If the following conditions are true, you should be fine clearing the Hub
+    database and starting over:
+
+    - users specified in config file
+    - user servers are stopped during upgrade
+    - don't mind causing users to login again after upgrade
+
+Backup JupyterHub configuration file
+------------------------------------
+
+Backup up your configuration file, ``jupyterhub_config.py``, to a secure
+location.
+
+Shutdown JupyterHub
+-------------------
+
+- Prior to shutting down JupyterHub, you should notify the Hub users of the
+  scheduled downtime.
+- Shutdown the JupyterHub service.
+
+Upgrade JupyterHub
+------------------
+
+Follow directions that correspond to your package manager, ``pip`` or ``conda``,
+for the new JupyterHub release:
+
+- ``pip install -U jupyterhub`` for ``pip``
+- ``conda upgrade jupyterhub`` for ``conda``
+
+Upgrade the proxy, authenticator, or spawner if needed.
+
+Upgrade JupyterHub database
+---------------------------
+
+To run the upgrade process for JupyterHub databases, enter::
+
+    jupyterhub upgrade-db
+
+Update the JupyterHub configuration file
+----------------------------------------
+
+Create a new JupyterHub configuration file or edit a copy of the existing
+file ``jupyterhub_config.py``.
+
+Start JupyterHub
+----------------
+
+Start JupyterHub with the same command that you used before the upgrade.

examples/cull-idle/cull_idle_servers.py

@@ -40,8 +40,11 @@ from tornado.options import define, options, parse_command_line
 
 
 @coroutine
-def cull_idle(url, api_token, timeout):
-    """cull idle single-user servers"""
+def cull_idle(url, api_token, timeout, cull_users=False):
+    """Shutdown idle single-user servers
+
+    If cull_users, inactive *users* will be deleted as well.
+    """
     auth_header = {
             'Authorization': 'token %s' % api_token
         }
@@ -54,26 +57,50 @@ def cull_idle(url, api_token, timeout):
     resp = yield client.fetch(req)
     users = json.loads(resp.body.decode('utf8', 'replace'))
     futures = []
-    for user in users:
-        last_activity = parse_date(user['last_activity'])
-        if user['server'] and last_activity < cull_limit:
-            app_log.info("Culling %s (inactive since %s)", user['name'], last_activity)
+
+    @coroutine
+    def cull_one(user, last_activity):
+        """cull one user"""
+
+        # shutdown server first. Hub doesn't allow deleting users with running servers.
+        if user['server']:
+            app_log.info("Culling server for %s (inactive since %s)", user['name'], last_activity)
             req = HTTPRequest(url=url + '/users/%s/server' % user['name'],
                 method='DELETE',
                 headers=auth_header,
             )
-            futures.append((user['name'], client.fetch(req)))
-        elif user['server'] and last_activity > cull_limit:
+            yield client.fetch(req)
+        if cull_users:
+            app_log.info("Culling user %s (inactive since %s)", user['name'], last_activity)
+            req = HTTPRequest(url=url + '/users/%s' % user['name'],
+                method='DELETE',
+                headers=auth_header,
+            )
+            yield client.fetch(req)
+
+    for user in users:
+        if not user['server'] and not cull_users:
+            # server not running and not culling users, nothing to do
+            continue
+        last_activity = parse_date(user['last_activity'])
+        if last_activity < cull_limit:
+            futures.append((user['name'], cull_one(user, last_activity)))
+        else:
             app_log.debug("Not culling %s (active since %s)", user['name'], last_activity)
     
     for (name, f) in futures:
         yield f
         app_log.debug("Finished culling %s", name)
 
+
 if __name__ == '__main__':
     define('url', default=os.environ.get('JUPYTERHUB_API_URL'), help="The JupyterHub API URL")
     define('timeout', default=600, help="The idle timeout (in seconds)")
     define('cull_every', default=0, help="The interval (in seconds) for checking for idle servers to cull")
+    define('cull_users', default=False,
+        help="""Cull users in addition to servers.
+                This is for use in temporary-user cases such as tmpnb.""",
+    )
     
     parse_command_line()
     if not options.cull_every:
@@ -82,7 +109,7 @@ if __name__ == '__main__':
     api_token = os.environ['JUPYTERHUB_API_TOKEN']
     
     loop = IOLoop.current()
-    cull = lambda : cull_idle(options.url, api_token, options.timeout)
+    cull = lambda : cull_idle(options.url, api_token, options.timeout, options.cull_users)
     # run once before scheduling periodic call
     loop.run_sync(cull)
     # schedule periodic cull

examples/service-whoami-flask/README.md

@@ -8,7 +8,7 @@ Uses `jupyterhub.services.HubAuth` to authenticate requests with the Hub in a [f
 
         jupyterhub --ip=127.0.0.1
 
-2. Visit http://127.0.0.1:8000/services/whoami
+2. Visit http://127.0.0.1:8000/services/whoami or http://127.0.0.1:8000/services/whoami-oauth
 
 After logging in with your local-system credentials, you should see a JSON dump of your user info:
 

examples/service-whoami-flask/jupyterhub_config.py

@@ -9,5 +9,13 @@ c.JupyterHub.services = [
         'environment': {
             'FLASK_APP': 'whoami-flask.py',
         }
+    },
+    {
+        'name': 'whoami-oauth',
+        'url': 'http://127.0.0.1:10201',
+        'command': ['flask', 'run', '--port=10201'],
+        'environment': {
+            'FLASK_APP': 'whoami-oauth.py',
         }
+    },
 ]

examples/service-whoami-flask/whoami-flask.py

@@ -17,7 +17,7 @@ prefix = os.environ.get('JUPYTERHUB_SERVICE_PREFIX', '/')
 
 auth = HubAuth(
     api_token=os.environ['JUPYTERHUB_API_TOKEN'],
-    cookie_cache_max_age=60,
+    cache_max_age=60,
 )
 
 app = Flask(__name__)

examples/service-whoami-flask/whoami-oauth.py

@@ -0,0 +1,70 @@
+#!/usr/bin/env python3
+"""
+whoami service authentication with the Hub
+"""
+
+from functools import wraps
+import json
+import os
+
+from flask import Flask, redirect, request, Response, make_response
+
+from jupyterhub.services.auth import HubOAuth
+
+
+prefix = os.environ.get('JUPYTERHUB_SERVICE_PREFIX', '/')
+
+auth = HubOAuth(
+    api_token=os.environ['JUPYTERHUB_API_TOKEN'],
+    cache_max_age=60,
+)
+
+app = Flask(__name__)
+
+
+def authenticated(f):
+    """Decorator for authenticating with the Hub via OAuth"""
+    @wraps(f)
+    def decorated(*args, **kwargs):
+        token = request.cookies.get(auth.cookie_name)
+        if token:
+            user = auth.user_for_token(token)
+        else:
+            user = None
+        if user:
+            return f(user, *args, **kwargs)
+        else:
+            # redirect to login url on failed auth
+            state = auth.generate_state(next_url=request.path)
+            response = make_response(redirect(auth.login_url + '&state=%s' % state))
+            response.set_cookie(auth.state_cookie_name, state)
+            return response
+    return decorated
+
+
+@app.route(prefix)
+@authenticated
+def whoami(user):
+    return Response(
+        json.dumps(user, indent=1, sort_keys=True),
+        mimetype='application/json',
+        )
+
+@app.route(prefix + 'oauth_callback')
+def oauth_callback():
+    code = request.args.get('code', None)
+    if code is None:
+        return 403
+
+    # validate state field
+    arg_state = request.args.get('state', None)
+    cookie_state = request.cookies.get(auth.state_cookie_name)
+    if arg_state != cookie_state:
+        # state doesn't match
+        return 403
+
+    token = auth.token_for_code(code)
+    next_url = auth.get_next_url(cookie_state) or prefix
+    response = make_response(redirect(next_url))
+    response.set_cookie(auth.cookie_name, token)
+    return response

examples/service-whoami/README.md

@@ -2,13 +2,15 @@
 
 Uses `jupyterhub.services.HubAuthenticated` to authenticate requests with the Hub.
 
+There is an implementation each of cookie-based `HubAuthenticated` and OAuth-based `HubOAuthenticated`.
+
 ## Run
 
 1. Launch JupyterHub and the `whoami service` with
 
         jupyterhub --ip=127.0.0.1
 
-2. Visit http://127.0.0.1:8000/services/whoami
+2. Visit http://127.0.0.1:8000/services/whoami or http://127.0.0.1:8000/services/whoami-oauth
 
 After logging in with your local-system credentials, you should see a JSON dump of your user info:
 

examples/service-whoami/jupyterhub_config.py

@@ -6,5 +6,10 @@ c.JupyterHub.services = [
         'name': 'whoami',
         'url': 'http://127.0.0.1:10101',
         'command': [sys.executable, './whoami.py'],
-    }
+    },
+    {
+        'name': 'whoami-oauth',
+        'url': 'http://127.0.0.1:10102',
+        'command': [sys.executable, './whoami-oauth.py'],
+    },
 ]

examples/service-whoami/whoami-oauth.py

@@ -13,10 +13,10 @@ from tornado.ioloop import IOLoop
 from tornado.httpserver import HTTPServer
 from tornado.web import RequestHandler, Application, authenticated
 
-from jupyterhub.services.auth import HubAuthenticated
+from jupyterhub.services.auth import HubOAuthenticated, HubOAuthCallbackHandler
+from jupyterhub.utils import url_path_join
 
-
-class WhoAmIHandler(HubAuthenticated, RequestHandler):
+class WhoAmIHandler(HubOAuthenticated, RequestHandler):
     hub_users = {getuser()} # the users allowed to access this service
 
     @authenticated
@@ -27,9 +27,10 @@ class WhoAmIHandler(HubAuthenticated, RequestHandler):
 
 def main():
     app = Application([
-        (os.environ['JUPYTERHUB_SERVICE_PREFIX'] + '/?', WhoAmIHandler),
+        (os.environ['JUPYTERHUB_SERVICE_PREFIX'], WhoAmIHandler),
+        (url_path_join(os.environ['JUPYTERHUB_SERVICE_PREFIX'], 'oauth_callback'), HubOAuthCallbackHandler),
         (r'.*', WhoAmIHandler),
-    ])
+    ], cookie_secret=os.urandom(32))
     
     http_server = HTTPServer(app)
     url = urlparse(os.environ['JUPYTERHUB_SERVICE_URL'])

examples/service-whoami/whoami.py

@@ -27,7 +27,7 @@ def main():
     app = Application([
         (os.environ['JUPYTERHUB_SERVICE_PREFIX'] + '/?', WhoAmIHandler),
         (r'.*', WhoAmIHandler),
-    ], login_url='/hub/login')
+    ])
     
     http_server = HTTPServer(app)
     url = urlparse(os.environ['JUPYTERHUB_SERVICE_URL'])

jupyterhub/_version.py

@@ -7,7 +7,7 @@ version_info = (
     0,
     8,
     0,
-    'b1',
+    'b5',
 )
 
 __version__ = '.'.join(map(str, version_info))
@@ -28,6 +28,7 @@ def _check_version(hub_version, singleuser_version, log):
         from distutils.version import LooseVersion as V
         hub_major_minor = V(hub_version).version[:2]
         singleuser_major_minor = V(singleuser_version).version[:2]
+        extra = ""
         if singleuser_major_minor == hub_major_minor:
             # patch-level mismatch or lower, log difference at debug-level
             # because this should be fine
@@ -35,8 +36,11 @@ def _check_version(hub_version, singleuser_version, log):
         else:
             # log warning-level for more significant mismatch, such as 0.8 vs 0.9, etc.
             log_method = log.warning
-        log_method("jupyterhub version %s != jupyterhub-singleuser version %s",
-            hub_version, singleuser_version,
+            extra = " This could cause failure to authenticate and result in redirect loops!"
+        log_method(
+            "jupyterhub version %s != jupyterhub-singleuser version %s." + extra,
+            hub_version,
+            singleuser_version,
         )
     else:
         log.debug("jupyterhub and jupyterhub-singleuser both on version %s" % hub_version)

jupyterhub/apihandlers/auth.py

@@ -41,15 +41,27 @@ class TokenAPIHandler(APIHandler):
             # for authenticators where that's possible
             data = self.get_json_body()
             try:
-                authenticated = yield self.authenticate(self, data)
+                user = yield self.login_user(data)
             except Exception as e:
                 self.log.error("Failure trying to authenticate with form data: %s" % e)
-                authenticated = None
-            if authenticated is None:
+                user = None
+            if user is None:
                 raise web.HTTPError(403)
-            user = self.find_user(authenticated['name'])
+        else:
+            data = self.get_json_body()
+            # admin users can request 
+            if data and data.get('username') != user.name:
+                if user.admin:
+                    user = self.find_user(data['username'])
+                    if user is None:
+                        raise web.HTTPError(400, "No such user '%s'" % data['username'])
+                else:
+                    raise web.HTTPError(403, "Only admins can request tokens for other users.")
         api_token = user.new_api_token()
-        self.write(json.dumps({'token': api_token}))
+        self.write(json.dumps({
+            'token': api_token,
+            'user': self.user_model(user),
+        }))
 
 
 class CookieAPIHandler(APIHandler):

jupyterhub/apihandlers/base.py

@@ -104,22 +104,17 @@ class APIHandler(BaseHandler):
             'pending': None,
             'last_activity': user.last_activity.isoformat(),
         }
-        if user.spawners['']._spawn_pending:
-            model['pending'] = 'spawn'
-        elif user.spawners['']._stop_pending:
-            model['pending'] = 'stop'
+        model['pending'] = user.spawners[''].pending or None
 
         if self.allow_named_servers:
             servers = model['servers'] = {}
             for name, spawner in user.spawners.items():
                 if spawner.ready:
                     servers[name] = s = {'name': name}
-                    if spawner._spawn_pending:
-                        s['pending'] = 'spawn'
-                    elif spawner._stop_pending:
-                        s['pending'] = 'stop'
+                    if spawner.pending:
+                        s['pending'] = spawner.pending
                     if spawner.server:
-                        s['url'] = user.url + name
+                        s['url'] = user.url + name + '/'
         return model
 
     def group_model(self, group):

jupyterhub/apihandlers/users.py

@@ -178,19 +178,34 @@ class UserAPIHandler(APIHandler):
 
 class UserServerAPIHandler(APIHandler):
     """Start and stop single-user servers"""
+
     @gen.coroutine
     @admin_or_self
     def post(self, name, server_name=''):
         user = self.find_user(name)
-        if server_name:
-            if not self.allow_named_servers:
+        if server_name and not self.allow_named_servers:
             raise web.HTTPError(400, "Named servers are not enabled.")
+        if self.allow_named_servers and not server_name:
+            server_name = user.default_server_name()
         spawner = user.spawners[server_name]
+        pending = spawner.pending
+        if pending == 'spawn':
+            self.set_header('Content-Type', 'text/plain')
+            self.set_status(202)
+            return
+        elif pending:
+            raise web.HTTPError(400, "%s is pending %s" % (spawner._log_name, pending))
+
         if spawner.ready:
             # include notify, so that a server that died is noticed immediately
+            # set _spawn_pending flag to prevent races while we wait
+            spawner._spawn_pending = True
+            try:
                 state = yield spawner.poll_and_notify()
+            finally:
+                spawner._spawn_pending = False
             if state is None:
-                raise web.HTTPError(400, "%s's server %s is already running" % (name, server_name))
+                raise web.HTTPError(400, "%s is already running" % spawner._log_name)
 
         options = self.get_json_body()
         yield self.spawn_single_user(user, server_name, options=options)
@@ -209,17 +224,21 @@ class UserServerAPIHandler(APIHandler):
                 raise web.HTTPError(404, "%s has no server named '%s'" % (name, server_name))
 
         spawner = user.spawners[server_name]
-
-        if spawner._stop_pending:
+        if spawner.pending == 'stop':
+            self.log.debug("%s already stopping", spawner._log_name)
             self.set_header('Content-Type', 'text/plain')
             self.set_status(202)
             return
+
         if not spawner.ready:
-            raise web.HTTPError(400, "%s's server %s is not running" % (name, server_name))
+            raise web.HTTPError(
+                400, "%s is not running %s" %
+                (spawner._log_name, '(pending: %s)' % spawner.pending if spawner.pending else '')
+            )
         # include notify, so that a server that died is noticed immediately
         status = yield spawner.poll_and_notify()
         if status is not None:
-            raise web.HTTPError(400, "%s's server %s is not running" % (name, server_name))
+            raise web.HTTPError(400, "%s is not running" % spawner._log_name)
         yield self.stop_single_user(user, server_name)
         status = 202 if spawner._stop_pending else 204
         self.set_header('Content-Type', 'text/plain')

jupyterhub/app.py

@@ -62,7 +62,7 @@ from .utils import (
 from .auth import Authenticator, PAMAuthenticator
 from .crypto import CryptKeeper
 from .spawner import Spawner, LocalProcessSpawner
-from .objects import Hub
+from .objects import Hub, Server
 
 # For faking stats
 from .emptyclass import EmptyClass
@@ -291,13 +291,13 @@ class JupyterHub(Application):
     ssl_key = Unicode('',
         help="""Path to SSL key file for the public facing interface of the proxy
 
-        Use with ssl_cert
+        When setting this, you should also set ssl_cert
         """
     ).tag(config=True)
     ssl_cert = Unicode('',
         help="""Path to SSL certificate file for the public facing interface of the proxy
 
-        Use with ssl_key
+        When setting this, you should also set ssl_key
         """
     ).tag(config=True)
     ip = Unicode('',
@@ -801,12 +801,10 @@ class JupyterHub(Application):
         self.handlers = self.add_url_prefix(self.hub_prefix, h)
         # some extra handlers, outside hub_prefix
         self.handlers.extend([
-            (r"%s" % self.hub_prefix.rstrip('/'), web.RedirectHandler,
-                {
-                    "url": self.hub_prefix,
-                    "permanent": False,
-                }
-            ),
+            # add trailing / to `/hub`
+            (self.hub_prefix.rstrip('/'), handlers.AddSlashHandler),
+            # add trailing / to ``/user|services/:name`
+            (r"%s(user|services)/([^/]+)" % self.base_url, handlers.AddSlashHandler),
             (r"(?!%s).*" % self.hub_prefix, handlers.PrefixRedirectHandler),
             (r'(.*)', handlers.Template404),
         ])
@@ -1180,7 +1178,7 @@ class JupyterHub(Application):
             if not service.url:
                 continue
             try:
-                yield service.orm.server.wait_up(timeout=1)
+                yield Server.from_orm(service.orm.server).wait_up(timeout=1)
             except TimeoutError:
                 self.log.warning("Cannot connect to %s service %s at %s", service.kind, name, service.url)
             else:
@@ -1557,7 +1555,7 @@ class JupyterHub(Application):
                 tries = 10 if service.managed else 1
                 for i in range(tries):
                     try:
-                        yield service.orm.server.wait_up(http=True, timeout=1)
+                        yield Server.from_orm(service.orm.server).wait_up(http=True, timeout=1)
                     except TimeoutError:
                         if service.managed:
                             status = yield service.spawner.poll()

jupyterhub/handlers/base.py

@@ -20,7 +20,7 @@ from .. import __version__
 from .. import orm
 from ..objects import Server
 from ..spawner import LocalProcessSpawner
-from ..utils import url_path_join, exponential_backoff
+from ..utils import default_server_name, url_path_join
 
 # pattern for the authentication token header
 auth_header_pat = re.compile(r'^(?:token|bearer)\s+([^\s]+)$', flags=re.IGNORECASE)
@@ -347,7 +347,7 @@ class BaseHandler(RequestHandler):
         else:
             self.statsd.incr('login.failure')
             self.statsd.timing('login.authenticate.failure', auth_timer.ms)
-            self.log.warning("Failed login for %s", data.get('username', 'unknown user'))
+            self.log.warning("Failed login for %s", (data or {}).get('username', 'unknown user'))
 
 
     #---------------------------------------------------------------
@@ -376,8 +376,19 @@ class BaseHandler(RequestHandler):
 
     @gen.coroutine
     def spawn_single_user(self, user, server_name='', options=None):
-        if server_name in user.spawners and user.spawners[server_name].pending == 'spawn':
-            raise RuntimeError("Spawn already pending for: %s" % user.name)
+        # in case of error, include 'try again from /hub/home' message
+        self.extra_error_html = self.spawn_home_error
+
+        user_server_name = user.name
+        if self.allow_named_servers and not server_name:
+            server_name = default_server_name(user)
+
+        if server_name:
+            user_server_name = '%s:%s' % (user.name, server_name)
+
+        if server_name in user.spawners and user.spawners[server_name].pending:
+            pending = user.spawners[server_name].pending
+            raise RuntimeError("%s pending %s" % (user_server_name, pending))
 
         # count active servers and pending spawns
         # we could do careful bookkeeping to avoid
@@ -397,26 +408,20 @@ class BaseHandler(RequestHandler):
             )
             raise web.HTTPError(
                 429,
-                    "User startup rate limit exceeded. Try again in a few minutes.")
+                "User startup rate limit exceeded. Try again in a few minutes.",
+            )
         if active_server_limit and active_count >= active_server_limit:
             self.log.info(
                 '%s servers active, no space available',
                 active_count,
             )
-                raise web.HTTPError(
-                    429,
-                    "Active user limit exceeded. Try again in a few minutes.")
+            raise web.HTTPError(429, "Active user limit exceeded. Try again in a few minutes.")
 
         tic = IOLoop.current().time()
-        user_server_name = user.name
-        if server_name:
-            user_server_name = '%s:%s' % (user.name, server_name)
-        else:
-            user_server_name = user.name
 
         self.log.debug("Initiating spawn for %s", user_server_name)
 
-        f = user.spawn(server_name, options)
+        spawn_future = user.spawn(server_name, options)
 
         self.log.debug("%i%s concurrent spawns",
             spawn_pending_count,
@@ -426,22 +431,24 @@ class BaseHandler(RequestHandler):
             '/%i' % active_server_limit if active_server_limit else '')
 
         spawner = user.spawners[server_name]
+        # set spawn_pending now, so there's no gap where _spawn_pending is False
+        # while we are waiting for _proxy_pending to be set
+        spawner._spawn_pending = True
 
         @gen.coroutine
-        def finish_user_spawn(f=None):
+        def finish_user_spawn():
             """Finish the user spawn by registering listeners and notifying the proxy.
 
             If the spawner is slow to start, this is passed as an async callback,
             otherwise it is called immediately.
             """
-            if f and f.exception() is not None:
-                # failed, don't add to the proxy
-                return
+            # wait for spawn Future
+            yield spawn_future
             toc = IOLoop.current().time()
             self.log.info("User %s took %.3f seconds to start", user_server_name, toc-tic)
             self.statsd.timing('spawner.success', (toc - tic) * 1000)
-            try:
             spawner._proxy_pending = True
+            try:
                 yield self.proxy.add_user(user, server_name)
             except Exception:
                 self.log.exception("Failed to add %s to proxy!", user_server_name)
@@ -452,36 +459,53 @@ class BaseHandler(RequestHandler):
             finally:
                 spawner._proxy_pending = False
 
+        # hook up spawner._spawn_future so that other requests can await
+        # this result
+        finish_spawn_future = spawner._spawn_future = finish_user_spawn()
+        def _clear_spawn_future(f):
+            # clear spawner._spawn_future when it's done
+            # keep an exception around, though, to prevent repeated implicit spawns
+            # if spawn is failing
+            if f.exception() is None:
+                spawner._spawn_future = None
+            # Now we're all done. clear _spawn_pending flag
+            spawner._spawn_pending = False
+        finish_spawn_future.add_done_callback(_clear_spawn_future)
+
         try:
-            yield gen.with_timeout(timedelta(seconds=self.slow_spawn_timeout), f)
+            yield gen.with_timeout(timedelta(seconds=self.slow_spawn_timeout), finish_spawn_future)
         except gen.TimeoutError:
             # waiting_for_response indicates server process has started,
             # but is yet to become responsive.
-            if not spawner._waiting_for_response:
+            if spawner._spawn_pending and not spawner._waiting_for_response:
                 # still in Spawner.start, which is taking a long time
                 # we shouldn't poll while spawn is incomplete.
                 self.log.warning("User %s is slow to start (timeout=%s)",
                                  user_server_name, self.slow_spawn_timeout)
-                # schedule finish for when the user finishes spawning
-                IOLoop.current().add_future(f, finish_user_spawn)
-            else:
+                return
+
             # start has finished, but the server hasn't come up
             # check if the server died while we were waiting
-                status = yield user.spawner.poll()
-                if status is None:
-                    # hit timeout, but server's running. Hope that it'll show up soon enough,
+            status = yield spawner.poll()
+            if status is not None:
+                toc = IOLoop.current().time()
+                self.statsd.timing('spawner.failure', (toc - tic) * 1000)
+                raise web.HTTPError(500, "Spawner failed to start [status=%s]. The logs for %s may contain details." % (
+                    status, spawner._log_name))
+
+            if spawner._waiting_for_response:
+                # hit timeout waiting for response, but server's running.
+                # Hope that it'll show up soon enough,
                 # though it's possible that it started at the wrong URL
                 self.log.warning("User %s is slow to become responsive (timeout=%s)",
                                  user_server_name, self.slow_spawn_timeout)
-                    self.log.debug("Expecting server for %s at: %s", user_server_name, spawner.server.url)
-                    # schedule finish for when the user finishes spawning
-                    IOLoop.current().add_future(f, finish_user_spawn)
-                else:
-                    toc = IOLoop.current().time()
-                    self.statsd.timing('spawner.failure', (toc - tic) * 1000)
-                    raise web.HTTPError(500, "Spawner failed to start [status=%s]" % status)
-        else:
-            yield finish_user_spawn()
+                self.log.debug("Expecting server for %s at: %s",
+                               user_server_name, spawner.server.url)
+            if spawner._proxy_pending:
+                # User.spawn finished, but it hasn't been added to the proxy
+                # Could be due to load or a slow proxy
+                self.log.warning("User %s is slow to be added to the proxy (timeout=%s)",
+                                 user_server_name, self.slow_spawn_timeout)
 
     @gen.coroutine
     def user_stopped(self, user, server_name):
@@ -501,41 +525,55 @@ class BaseHandler(RequestHandler):
         if name not in user.spawners:
             raise KeyError("User %s has no such spawner %r", user.name, name)
         spawner = user.spawners[name]
-        if spawner._stop_pending:
-            raise RuntimeError("Stop already pending for: %s:%s" % (user.name, name))
-        tic = IOLoop.current().time()
-        yield self.proxy.delete_user(user, name)
-        f = user.stop()
-        @gen.coroutine
-        def finish_stop(f=None):
-            """Finish the stop action by noticing that the user is stopped.
+        if spawner.pending:
+            raise RuntimeError("%s pending %s" % (spawner._log_name, spawner.pending))
+        # set user._stop_pending before doing anything async
+        # to avoid races
+        spawner._stop_pending = True
 
-            If the spawner is slow to stop, this is passed as an async callback,
-            otherwise it is called immediately.
+        @gen.coroutine
+        def stop():
+            """Stop the server
+
+            1. remove it from the proxy
+            2. stop the server
+            3. notice that it stopped
             """
-            if f and f.exception() is not None:
-                # failed, don't do anything
-                return
+            tic = IOLoop.current().time()
+            try:
+                yield self.proxy.delete_user(user, name)
+                yield user.stop(name)
+            finally:
+                spawner._stop_pending = False
             toc = IOLoop.current().time()
-            self.log.info("User %s server took %.3f seconds to stop", user.name, toc-tic)
+            self.log.info("User %s server took %.3f seconds to stop", user.name, toc - tic)
 
         try:
-            yield gen.with_timeout(timedelta(seconds=self.slow_stop_timeout), f)
+            yield gen.with_timeout(timedelta(seconds=self.slow_stop_timeout), stop())
         except gen.TimeoutError:
             if spawner._stop_pending:
                 # hit timeout, but stop is still pending
                 self.log.warning("User %s:%s server is slow to stop", user.name, name)
-                # schedule finish for when the server finishes stopping
-                IOLoop.current().add_future(f, finish_stop)
             else:
                 raise
-        else:
-            yield finish_stop()
 
     #---------------------------------------------------------------
     # template rendering
     #---------------------------------------------------------------
 
+    @property
+    def spawn_home_error(self):
+        """Extra message pointing users to try spawning again from /hub/home.
+
+        Should be added to `self.extra_error_html` for any handler
+        that could serve a failed spawn message.
+        """
+        home = url_path_join(self.hub.base_url, 'home')
+        return (
+            "You can try restarting your server from the "
+            "<a href='{home}'>home page</a>.".format(home=home)
+        )
+
     def get_template(self, name):
         """Return the jinja template object for a given name"""
         return self.settings['jinja2_env'].get_template(name)
@@ -583,6 +621,7 @@ class BaseHandler(RequestHandler):
             status_code=status_code,
             status_message=status_message,
             message=message,
+            extra_error_html=getattr(self, 'extra_error_html', ''),
             exception=exception,
         )
 
@@ -636,10 +675,13 @@ class UserSpawnHandler(BaseHandler):
         current_user = self.get_current_user()
 
         if current_user and current_user.name == name:
+            # if spawning fails for any reason, point users to /hub/home to retry
+            self.extra_error_html = self.spawn_home_error
+
             # If people visit /user/:name directly on the Hub,
             # the redirects will just loop, because the proxy is bypassed.
             # Try to check for that and warn,
-            # though the user-facing behavior is unchainged
+            # though the user-facing behavior is unchanged
             host_info = urlparse(self.request.full_url())
             port = host_info.port
             if not port:
@@ -651,9 +693,36 @@ class UserSpawnHandler(BaseHandler):
                     Make sure to connect to the proxied public URL %s
                     """, self.request.full_url(), self.proxy.public_url)
 
-            # logged in as correct user, spawn the server
+            # logged in as correct user, check for pending spawn
             spawner = current_user.spawner
-            if spawner._spawn_pending or spawner._proxy_pending:
+
+            # First, check for previous failure.
+            if (
+                not spawner.active
+                and spawner._spawn_future
+                and spawner._spawn_future.done()
+                and spawner._spawn_future.exception()
+            ):
+                # Condition: spawner not active and _spawn_future exists and contains an Exception
+                # Implicit spawn on /user/:name is not allowed if the user's last spawn failed.
+                # We should point the user to Home if the most recent spawn failed.
+                self.log.error("Preventing implicit spawn for %s because last spawn failed: %s",
+                    spawner._log_name, spawner._spawn_future.exception())
+                raise spawner._spawn_future.exception()
+
+            # check for pending spawn
+            if spawner.pending and spawner._spawn_future:
+                # wait on the pending spawn
+                self.log.debug("Waiting for %s pending %s", spawner._log_name, spawner.pending)
+                try:
+                    yield gen.with_timeout(timedelta(seconds=self.slow_spawn_timeout), spawner._spawn_future)
+                except gen.TimeoutError:
+                    self.log.info("Pending spawn for %s didn't finish in %.1f seconds", spawner._log_name, self.slow_spawn_timeout)
+                    pass
+
+            # we may have waited above, check pending again:
+            if spawner.pending:
+                self.log.info("%s is pending %s", spawner._log_name, spawner.pending)
                 # spawn has started, but not finished
                 self.statsd.incr('redirects.user_spawn_pending', 1)
                 html = self.render_template("spawn_pending.html", user=current_user)
@@ -661,7 +730,12 @@ class UserSpawnHandler(BaseHandler):
                 return
 
             # spawn has supposedly finished, check on the status
+            if spawner.ready:
                 status = yield spawner.poll()
+            else:
+                status = 0
+
+            # server is not running, trigger spawn
             if status is not None:
                 if spawner.options_form:
                     self.redirect(url_concat(url_path_join(self.hub.base_url, 'spawn'),
@@ -670,6 +744,15 @@ class UserSpawnHandler(BaseHandler):
                 else:
                     yield self.spawn_single_user(current_user)
 
+            # spawn didn't finish, show pending page
+            if spawner.pending:
+                self.log.info("%s is pending %s", spawner._log_name, spawner.pending)
+                # spawn has started, but not finished
+                self.statsd.incr('redirects.user_spawn_pending', 1)
+                html = self.render_template("spawn_pending.html", user=current_user)
+                self.finish(html)
+                return
+
             # We do exponential backoff here - since otherwise we can get stuck in a redirect loop!
             # This is important in many distributed proxy implementations - those are often eventually
             # consistent and can take upto a couple of seconds to actually apply throughout the cluster.
@@ -679,9 +762,23 @@ class UserSpawnHandler(BaseHandler):
                 self.log.warning("Invalid redirects argument %r", self.get_argument('redirects'))
                 redirects = 0
 
-            if redirects >= self.settings.get('user_redirect_limit', 5):
+            # check redirect limit to prevent browser-enforced limits.
+            # In case of version mismatch, raise on only two redirects.
+            if redirects >= self.settings.get(
+                'user_redirect_limit', 4
+            ) or (redirects >= 2 and spawner._jupyterhub_version != __version__):
                 # We stop if we've been redirected too many times.
-                raise web.HTTPError(500, "Redirect loop detected.")
+                msg = "Redirect loop detected."
+                if spawner._jupyterhub_version != __version__:
+                    msg += (
+                        " Notebook has jupyterhub version {singleuser}, but the Hub expects {hub}."
+                        " Try installing jupyterhub=={hub} in the user environment"
+                        " if you continue to have problems."
+                    ).format(
+                        singleuser=spawner._jupyterhub_version or 'unknown (likely < 0.8)',
+                        hub=__version__,
+                    )
+                raise web.HTTPError(500, msg)
 
             # set login cookie anew
             self.set_login_cookie(current_user)
@@ -755,6 +852,13 @@ class CSPReportHandler(BaseHandler):
         self.statsd.incr('csp_report')
 
 
+class AddSlashHandler(BaseHandler):
+    """Handler for adding trailing slash to URLs that need them"""
+    def get(self, *args):
+        src = urlparse(self.request.uri)
+        dest = src._replace(path=src.path + '/')
+        self.redirect(urlunparse(dest))
+
 default_handlers = [
     (r'/user/([^/]+)(/.*)?', UserSpawnHandler),
     (r'/user-redirect/(.*)?', UserRedirectHandler),

jupyterhub/handlers/login.py

@@ -84,10 +84,11 @@ class LoginHandler(BaseHandler):
 
         if user:
             already_running = False
-            if user.spawner:
+            if user.spawner.ready:
                 status = yield user.spawner.poll()
                 already_running = (status is None)
-            if not already_running and not user.spawner.options_form:
+            if not already_running and not user.spawner.options_form \
+                    and not user.spawner.pending:
                 # logging in triggers spawn
                 yield self.spawn_single_user(user)
             self.redirect(self.get_next_url())

jupyterhub/handlers/pages.py

@@ -67,9 +67,13 @@ class HomeHandler(BaseHandler):
         if user.running:
             # trigger poll_and_notify event in case of a server that died
             yield user.spawner.poll_and_notify()
+        # send the user to /spawn if they aren't running,
+        # to establish that this is an explicit spawn request rather
+        # than an implicit one, which can be caused by any link to `/user/:name`
+        url = user.url if user.running else url_path_join(self.hub.base_url, 'spawn')
         html = self.render_template('home.html',
             user=user,
-            url=user.url,
+            url=url,
         )
         self.finish(html)
 
@@ -92,7 +96,10 @@ class SpawnHandler(BaseHandler):
 
     @web.authenticated
     def get(self):
-        """GET renders form for spawning with user-specified options"""
+        """GET renders form for spawning with user-specified options
+
+        or triggers spawn via redirect if there is no form.
+        """
         user = self.get_current_user()
         if not self.allow_named_servers and user.running:
             url = user.url
@@ -102,7 +109,12 @@ class SpawnHandler(BaseHandler):
         if user.spawner.options_form:
             self.finish(self._render_form())
         else:
-            # not running, no form. Trigger spawn.
+            # Explicit spawn request: clear _spawn_future
+            # which may have been saved to prevent implicit spawns
+            # after a failure.
+            if user.spawner._spawn_future and user.spawner._spawn_future.done():
+                user.spawner._spawn_future = None
+            # not running, no form. Trigger spawn by redirecting to /user/:name
             self.redirect(user.url)
 
     @web.authenticated
@@ -115,6 +127,10 @@ class SpawnHandler(BaseHandler):
             self.log.warning("User is already running: %s", url)
             self.redirect(url)
             return
+        if user.spawner.pending:
+            raise web.HTTPError(
+                400, "%s is pending %s" % (user.spawner._log_name, user.spawner.pending)
+            )
         form_options = {}
         for key, byte_list in self.request.body_arguments.items():
             form_options[key] = [ bs.decode('utf8') for bs in byte_list ]
@@ -146,14 +162,19 @@ class AdminHandler(BaseHandler):
         available = {'name', 'admin', 'running', 'last_activity'}
         default_sort = ['admin', 'name']
         mapping = {
-            'running': '_server_id'
+            'running': orm.Spawner.server_id,
         }
+        for name in available:
+            if name not in mapping:
+                mapping[name] = getattr(orm.User, name)
+
         default_order = {
             'name': 'asc',
             'last_activity': 'desc',
             'admin': 'desc',
             'running': 'desc',
         }
+
         sorts = self.get_arguments('sort') or default_sort
         orders = self.get_arguments('order')
 
@@ -176,11 +197,11 @@ class AdminHandler(BaseHandler):
 
         # this could be one incomprehensible nested list comprehension
         # get User columns
-        cols = [ getattr(orm.User, mapping.get(c, c)) for c in sorts ]
+        cols = [ mapping[c] for c in sorts ]
         # get User.col.desc() order objects
         ordered = [ getattr(c, o)() for c, o in zip(cols, orders) ]
 
-        users = self.db.query(orm.User).order_by(*ordered)
+        users = self.db.query(orm.User).join(orm.Spawner).order_by(*ordered)
         users = [ self._user_from_orm(u) for u in users ]
         running = [ u for u in users if u.running ]
 

jupyterhub/proxy.py

@@ -231,9 +231,10 @@ class Proxy(LoggingConfigurable):
                       user.name, spawner.proxy_spec, spawner.server.host,
                       )
 
-        if spawner._spawn_pending:
+        if spawner.pending and spawner.pending != 'spawn':
             raise RuntimeError(
-                "User %s's spawn is pending, shouldn't be added to the proxy yet!", user.name)
+                "%s is pending %s, shouldn't be added to the proxy yet!" % (spawner._log_name, spawner.pending)
+            )
 
         yield self.add_route(
             spawner.proxy_spec,
@@ -326,7 +327,7 @@ class Proxy(LoggingConfigurable):
                                 spec, route['target'], spawner.server,
                             )
                             futures.append(self.add_user(user, name))
-                elif spawner._proxy_pending:
+                elif spawner._spawn_pending:
                     good_routes.add(spawner.proxy_spec)
 
         # check service routes
@@ -374,7 +375,7 @@ class Proxy(LoggingConfigurable):
         self.log.info("Setting up routes on new proxy")
         yield self.add_hub_route(self.app.hub)
         yield self.add_all_users(self.app.users)
-        yield self.add_all_services(self.app.services)
+        yield self.add_all_services(self.app._service_map)
         self.log.info("New proxy back up and good to go")
 
 

jupyterhub/services/auth.py

@@ -9,11 +9,15 @@ model describing the authenticated user.
 authenticate with the Hub.
 
 """
+
+import base64
+import json
 import os
 import re
 import socket
 import time
 from urllib.parse import quote, urlencode
+import uuid
 import warnings
 
 import requests
@@ -239,7 +243,8 @@ class HubAuth(Configurable):
         headers.setdefault('Authorization', 'token %s' % self.api_token)
         try:
             r = requests.request(method, url, **kwargs)
-        except requests.ConnectionError:
+        except requests.ConnectionError as e:
+            app_log.error("Error connecting to %s: %s", self.api_url, e)
             msg = "Failed to connect to Hub API at %r." % self.api_url
             msg += "  Is the Hub accessible at this URL (from host: %s)?" % socket.gethostname()
             if '127.0.0.1' in self.api_url:
@@ -397,6 +402,14 @@ class HubOAuth(HubAuth):
         """
         return self.oauth_client_id
 
+    @property
+    def state_cookie_name(self):
+        """The cookie name for storing OAuth state
+
+        This cookie is only live for the duration of the OAuth handshake.
+        """
+        return self.cookie_name + '-oauth-state'
+
     def _get_user_cookie(self, handler):
         token = handler.get_secure_cookie(self.cookie_name)
         if token:
@@ -476,6 +489,84 @@ class HubOAuth(HubAuth):
 
         return token_reply['access_token']
 
+    def _encode_state(self, state):
+        """Encode a state dict as url-safe base64"""
+        # trim trailing `=` because 
+        json_state = json.dumps(state)
+        return base64.urlsafe_b64encode(
+            json_state.encode('utf8')
+        ).decode('ascii').rstrip('=')
+
+    def _decode_state(self, b64_state):
+        """Decode a base64 state
+
+        Always returns a dict.
+        The dict will be empty if the state is invalid.
+        """
+        if isinstance(b64_state, str):
+            b64_state = b64_state.encode('ascii')
+        if len(b64_state) != 4:
+            # restore padding
+            b64_state = b64_state + (b'=' * (4 - len(b64_state) % 4))
+        try:
+            json_state = base64.urlsafe_b64decode(b64_state).decode('utf8')
+        except ValueError:
+            app_log.error("Failed to b64-decode state: %r", b64_state)
+            return {}
+        try:
+            return json.loads(json_state)
+        except ValueError:
+            app_log.error("Failed to json-decode state: %r", json_state)
+            return {}
+
+    def set_state_cookie(self, handler, next_url=None):
+        """Generate an OAuth state and store it in a cookie
+
+        Parameters
+        ----------
+        handler (RequestHandler): A tornado RequestHandler
+        next_url (str): The page to redirect to on successful login
+
+        Returns
+        -------
+        state (str): The OAuth state that has been stored in the cookie (url safe, base64-encoded)
+        """
+        b64_state = self.generate_state(next_url)
+        kwargs = {
+            'path': self.base_url,
+            'httponly': True,
+            'expires_days': 1,
+        }
+        if handler.request.protocol == 'https':
+            kwargs['secure'] = True
+        handler.set_secure_cookie(
+            self.state_cookie_name,
+            b64_state,
+            **kwargs
+        )
+        return b64_state
+
+    def generate_state(self, next_url=None):
+        """Generate a state string, given a next_url redirect target
+
+        Parameters
+        ----------
+        next_url (str): The URL of the page to redirect to on successful login.
+
+        Returns
+        -------
+        state (str): The base64-encoded state string.
+        """
+        return self._encode_state({
+            'uuid': uuid.uuid4().hex,
+            'next_url': next_url
+        })
+
+    def get_next_url(self, b64_state=''):
+        """Get the next_url for redirection, given an encoded OAuth state"""
+        state = self._decode_state(b64_state)
+        return state.get('next_url') or self.base_url
+
     def set_cookie(self, handler, access_token):
         """Set a cookie recording OAuth result"""
         kwargs = {
@@ -565,8 +656,14 @@ class HubAuthenticated(object):
 
     def get_login_url(self):
         """Return the Hub's login URL"""
-        app_log.debug("Redirecting to login url: %s" % self.hub_auth.login_url)
-        return self.hub_auth.login_url
+        login_url = self.hub_auth.login_url
+        app_log.debug("Redirecting to login url: %s", login_url)
+        if isinstance(self.hub_auth, HubOAuthenticated):
+            # add state argument to OAuth url
+            state = self.hub_auth.set_state_cookie(self, next_url=self.request.uri)
+            return url_concat(login_url, {'state': state})
+        else:
+            return login_url
 
     def check_hub_user(self, model):
         """Check whether Hub-authenticated user or service should be allowed.
@@ -634,6 +731,19 @@ class HubAuthenticated(object):
         except Exception:
             self._hub_auth_user_cache = None
             raise
+
+        # store ?token=... tokens passed via url in a cookie for future requests
+        url_token = self.get_argument('token', '')
+        if (
+            user_model
+            and url_token
+            and getattr(self, '_token_authenticated', False)
+            and hasattr(self.hub_auth, 'set_cookie')
+        ):
+            # authenticated via `?token=`
+            # set a cookie for future requests
+            # hub_auth.set_cookie is only available on HubOAuth
+            self.hub_auth.set_cookie(self, url_token)
         return self._hub_auth_user_cache
 
 
@@ -657,6 +767,21 @@ class HubOAuthCallbackHandler(HubOAuthenticated, RequestHandler):
         code = self.get_argument("code", False)
         if not code:
             raise HTTPError(400, "oauth callback made without a token")
+
+        # validate OAuth state
+        arg_state = self.get_argument("state", None)
+        cookie_state = self.get_secure_cookie(self.hub_auth.state_cookie_name)
+        next_url = None
+        if arg_state or cookie_state:
+            # clear cookie state now that we've consumed it
+            self.clear_cookie(self.hub_auth.state_cookie_name)
+            if isinstance(cookie_state, bytes):
+                cookie_state = cookie_state.decode('ascii', 'replace')
+            # check that state matches
+            if arg_state != cookie_state:
+                app_log.debug("oauth state %r != %r", arg_state, cookie_state)
+                raise HTTPError(403, "oauth state does not match")
+            next_url = self.hub_auth.get_next_url(cookie_state)
         # TODO: make async (in a Thread?)
         token = self.hub_auth.token_for_code(code)
         user_model = self.hub_auth.user_for_token(token)
@@ -664,7 +789,6 @@ class HubOAuthCallbackHandler(HubOAuthenticated, RequestHandler):
             raise HTTPError(500, "oauth callback failed to identify a user")
         app_log.info("Logged-in user %s", user_model)
         self.hub_auth.set_cookie(self, token)
-        next_url = self.get_argument('next', '') or self.hub_auth.base_url
-        self.redirect(next_url)
+        self.redirect(next_url or self.hub_auth.base_url)
 
 

jupyterhub/singleuser.py

@@ -22,6 +22,7 @@ except ImportError:
 
 from traitlets import (
     Bool,
+    Bytes,
     Unicode,
     CUnicode,
     default,
@@ -116,20 +117,6 @@ class OAuthCallbackHandler(HubOAuthCallbackHandler, IPythonHandler):
     def hub_auth(self):
         return self.settings['hub_auth']
 
-    def get(self):
-        code = self.get_argument("code", False)
-        if not code:
-            raise HTTPError(400, "oauth callback made without a token")
-        # TODO: make async (in a Thread?)
-        token = self.hub_auth.token_for_code(code)
-        user_model = self.hub_auth.user_for_token(token)
-        if user_model is None:
-            raise HTTPError(500, "oauth callback failed to identify a user")
-        self.log.info("Logged-in user %s", user_model)
-        self.hub_auth.set_cookie(self, token)
-        next_url = self.get_argument('next', '') or self.base_url
-        self.redirect(next_url)
-
 
 # register new hub related command-line aliases
 aliases = dict(notebook_aliases)
@@ -193,6 +180,15 @@ class SingleUserNotebookApp(NotebookApp):
     version = __version__
     classes = NotebookApp.classes + [HubOAuth]
     
+    # don't store cookie secrets
+    cookie_secret_file = ''
+    # always generate a new cookie secret on launch
+    # ensures that each spawn clears any cookies from previous session,
+    # triggering OAuth again
+    cookie_secret = Bytes()
+    def _cookie_secret_default(self):
+        return os.urandom(32)
+
     user = CUnicode().tag(config=True)
     group = CUnicode().tag(config=True)
     

jupyterhub/spawner.py

@@ -18,7 +18,7 @@ from tempfile import mkdtemp
 from sqlalchemy import inspect
 
 from tornado import gen
-from tornado.ioloop import PeriodicCallback, IOLoop
+from tornado.ioloop import PeriodicCallback
 
 from traitlets.config import LoggingConfigurable
 from traitlets import (
@@ -49,9 +49,23 @@ class Spawner(LoggingConfigurable):
     
     # private attributes for tracking status
     _spawn_pending = False
+    _start_pending = False
     _stop_pending = False
     _proxy_pending = False
     _waiting_for_response = False
+    _jupyterhub_version = None
+    _spawn_future = None
+
+    @property
+    def _log_name(self):
+        """Return username:servername or username
+
+        Used in logging for consistency with named servers.
+        """
+        if self.name:
+            return '%s:%s' % (self.user.name, self.name)
+        else:
+            return self.user.name
 
     @property
     def pending(self):
@@ -59,7 +73,7 @@ class Spawner(LoggingConfigurable):
 
         Return False if nothing is pending.
         """
-        if self._spawn_pending or self._proxy_pending:
+        if self._spawn_pending:
             return 'spawn'
         elif self._stop_pending:
             return 'stop'
@@ -89,6 +103,7 @@ class Spawner(LoggingConfigurable):
     authenticator = Any()
     hub = Any()
     orm_spawner = Any()
+    db = Any()
 
     @observe('orm_spawner')
     def _orm_spawner_changed(self, change):

jupyterhub/tests/mocking.py

@@ -7,8 +7,6 @@ import threading
 from unittest import mock
 from urllib.parse import urlparse
 
-import requests
-
 from tornado import gen
 from tornado.concurrent import Future
 from tornado.ioloop import IOLoop
@@ -58,6 +56,13 @@ class MockSpawner(LocalProcessSpawner):
     def _cmd_default(self):
         return [sys.executable, '-m', 'jupyterhub.tests.mocksu']
 
+    use_this_api_token = None
+    def start(self):
+        if self.use_this_api_token:
+            self.api_token = self.use_this_api_token
+        elif self.will_resume:
+            self.use_this_api_token = self.api_token
+        return super().start()
 
 class SlowSpawner(MockSpawner):
     """A spawner that takes a few seconds to start"""

jupyterhub/tests/test_api.py

@@ -89,7 +89,7 @@ def api_request(app, *api_path, **kwargs):
     base_url = app.hub.url
     headers = kwargs.setdefault('headers', {})
 
-    if 'Authorization' not in headers:
+    if 'Authorization' not in headers and not kwargs.pop('noauth', False):
         headers.update(auth_header(app.db, 'admin'))
 
     url = ujoin(base_url, 'api', *api_path)
@@ -654,6 +654,50 @@ def test_active_server_limit(app, request):
     assert counts['pending'] == 0
 
 
+@mark.gen_test
+def test_start_stop_race(app, no_patience, slow_spawn):
+    user = add_user(app.db, app, name='panda')
+    spawner = user.spawner
+    # start the server
+    r = yield api_request(app, 'users', user.name, 'server', method='post')
+    assert r.status_code == 202
+    assert spawner.pending == 'spawn'
+    # additional spawns while spawning shouldn't trigger a new spawn
+    with mock.patch.object(spawner, 'start') as m:
+        r = yield api_request(app, 'users', user.name, 'server', method='post')
+    assert r.status_code == 202
+    assert m.call_count == 0
+
+    # stop while spawning is not okay
+    r = yield api_request(app, 'users', user.name, 'server', method='delete')
+    assert r.status_code == 400
+    while not spawner.ready:
+        yield gen.sleep(0.1)
+
+    spawner.delay = 3
+    # stop the spawner
+    r = yield api_request(app, 'users', user.name, 'server', method='delete')
+    assert r.status_code == 202
+    assert spawner.pending == 'stop'
+    # make sure we get past deleting from the proxy
+    yield gen.sleep(1)
+    # additional stops while stopping shouldn't trigger a new stop
+    with mock.patch.object(spawner, 'stop') as m:
+        r = yield api_request(app, 'users', user.name, 'server', method='delete')
+    assert r.status_code == 202
+    assert m.call_count == 0
+    # start while stopping is not allowed
+    with mock.patch.object(spawner, 'start') as m:
+        r = yield api_request(app, 'users', user.name, 'server', method='post')
+    assert r.status_code == 400
+
+    while spawner.active:
+        yield gen.sleep(0.1)
+    # start after stop is okay
+    r = yield api_request(app, 'users', user.name, 'server', method='post')
+    assert r.status_code == 202
+
+
 @mark.gen_test
 def test_get_proxy(app):
     r = yield api_request(app, 'proxy')
@@ -711,16 +755,16 @@ def test_token(app):
 
 
 @mark.gen_test
-@mark.parametrize("headers, data, status", [
-    ({}, None, 200),
-    ({'Authorization': ''}, None, 403),
-    ({}, {'username': 'fake', 'password': 'fake'}, 200),
+@mark.parametrize("headers, status", [
+    ({}, 200),
+    ({'Authorization': 'token bad'}, 403),
 ])
-def test_get_new_token(app, headers, data, status):
-    if data:
-        data = json.dumps(data)
+def test_get_new_token(app, headers, status):
     # request a new token
-    r = yield api_request(app, 'authorizations', 'token', method='post', data=data, headers=headers)
+    r = yield api_request(app, 'authorizations', 'token',
+        method='post',
+        headers=headers,
+    )
     assert r.status_code == status
     if status != 200:
         return
@@ -728,7 +772,61 @@ def test_get_new_token(app, headers, data, status):
     assert 'token' in reply
     r = yield api_request(app, 'authorizations', 'token', reply['token'])
     r.raise_for_status()
-    assert 'name' in r.json()
+    reply = r.json()
+    assert reply['name'] == 'admin'
+
+
+@mark.gen_test
+def test_token_formdata(app):
+    """Create a token for a user with formdata and no auth header"""
+    data = {
+        'username': 'fake',
+        'password': 'fake',
+    }
+    r = yield api_request(app, 'authorizations', 'token',
+        method='post',
+        data=json.dumps(data) if data else None,
+        noauth=True,
+    )
+    assert r.status_code == 200
+    reply = r.json()
+    assert 'token' in reply
+    r = yield api_request(app, 'authorizations', 'token', reply['token'])
+    r.raise_for_status()
+    reply = r.json()
+    assert reply['name'] == data['username']
+
+
+@mark.gen_test
+@mark.parametrize("as_user, for_user, status", [
+    ('admin', 'other', 200),
+    ('admin', 'missing', 400),
+    ('user', 'other', 403),
+    ('user', 'user', 200),
+])
+def test_token_as_user(app, as_user, for_user, status):
+    # ensure both users exist
+    u = add_user(app.db, app, name=as_user)
+    if for_user != 'missing':
+        add_user(app.db, app, name=for_user)
+    data = {'username': for_user}
+    headers = {
+        'Authorization': 'token %s' % u.new_api_token(),
+    }
+    r = yield api_request(app, 'authorizations', 'token',
+        method='post',
+        data=json.dumps(data),
+        headers=headers,
+    )
+    assert r.status_code == status
+    reply = r.json()
+    if status != 200:
+        return
+    assert 'token' in reply
+    r = yield api_request(app, 'authorizations', 'token', reply['token'])
+    r.raise_for_status()
+    reply = r.json()
+    assert reply['name'] == data['username']
 
 
 # ---------------

jupyterhub/tests/test_named_servers.py

@@ -38,6 +38,27 @@ def test_create_named_server(app, named_servers):
     assert prefix == user.spawners[servername].server.base_url
     assert prefix.endswith('/user/%s/%s/' % (username, servername))
 
+    r = yield api_request(app, 'users', username)
+    r.raise_for_status()
+
+    user_model = r.json()
+    user_model.pop('last_activity')
+    assert user_model == {
+        'name': username,
+        'groups': [],
+        'kind': 'user',
+        'admin': False,
+        'pending': None,
+        'server': None,
+        'servers': {
+            name: {
+                'name': name,
+                'url': url_path_join(user.url, name, '/'),
+            }
+            for name in ['1', servername]
+        },
+    }
+
 
 @pytest.mark.gen_test
 def test_delete_named_server(app, named_servers):
@@ -69,9 +90,9 @@ def test_delete_named_server(app, named_servers):
         'servers': {
             name: {
                 'name': name,
-                'url': url_path_join(user.url, name),
+                'url': url_path_join(user.url, name, '/'),
             }
-            for name in ['1', servername]
+            for name in ['1']
         },
     }
 

jupyterhub/tests/test_pages.py

@@ -85,11 +85,25 @@ def test_admin_not_admin(app):
 @pytest.mark.gen_test
 def test_admin(app):
     cookies = yield app.login_user('admin')
-    r = yield get_page('admin', app, cookies=cookies)
+    r = yield get_page('admin', app, cookies=cookies, allow_redirects=False)
     r.raise_for_status()
     assert r.url.endswith('/admin')
 
 
+@pytest.mark.parametrize('sort', [
+    'running',
+    'last_activity',
+    'admin',
+    'name',
+])
+@pytest.mark.gen_test
+def test_admin_sort(app, sort):
+    cookies = yield app.login_user('admin')
+    r = yield get_page('admin?sort=%s' % sort, app, cookies=cookies)
+    r.raise_for_status()
+    assert r.status_code == 200
+
+
 @pytest.mark.gen_test
 def test_spawn_redirect(app):
     name = 'wash'
@@ -120,6 +134,12 @@ def test_spawn_redirect(app):
     path = urlparse(r.url).path
     assert path == ujoin(app.base_url, '/user/%s/' % name)
 
+    # test handing of trailing slash on `/user/name`
+    r = yield get_page('user/' + name, app, cookies=cookies)
+    r.raise_for_status()
+    path = urlparse(r.url).path
+    assert path == ujoin(app.base_url, '/user/%s/' % name)
+
 
 @pytest.mark.gen_test
 def test_spawn_page(app):

jupyterhub/tests/test_services_auth.py

@@ -292,6 +292,7 @@ def test_hubauth_service_token(app, mockservice_url):
         'name': name,
         'admin': False,
     }
+    assert not r.cookies
 
     # token in ?token parameter
     r = yield async_requests.get(public_url(app, mockservice_url) + '/whoami/?token=%s' % token)
@@ -319,7 +320,8 @@ def test_oauth_service(app, mockservice_url):
     # first request is only going to set login cookie
     # FIXME: redirect to originating URL (OAuth loses this info)
     s = requests.Session()
-    s.cookies = yield app.login_user('link')
+    name = 'link'
+    s.cookies = yield app.login_user(name)
     # run session.get in async_requests thread
     s_get = lambda *args, **kwargs: async_requests.executor.submit(s.get, *args, **kwargs)
     r = yield s_get(url)
@@ -335,3 +337,23 @@ def test_oauth_service(app, mockservice_url):
         'kind': 'user',
     }
 
+    # token-authenticated request to HubOAuth
+    token = app.users[name].new_api_token()
+    # token in ?token parameter
+    r = yield async_requests.get(public_url(app, mockservice_url) + 'owhoami/?token=%s' % token)
+    r.raise_for_status()
+    reply = r.json()
+    assert reply['name'] == name
+
+    # verify that ?token= requests set a cookie
+    assert len(r.cookies) != 0
+    # ensure cookie works in future requests
+    r = yield async_requests.get(
+        public_url(app, mockservice_url) + 'owhoami/',
+        cookies=r.cookies,
+        allow_redirects=False,
+    )
+    r.raise_for_status()
+    reply = r.json()
+    assert reply['name'] == name
+

jupyterhub/tests/test_spawner.py

@@ -15,11 +15,13 @@ from unittest import mock
 import pytest
 from tornado import gen
 
-from ..user import User
 from ..objects import Hub, Server
+from .. import orm
 from .. import spawner as spawnermod
 from ..spawner import LocalProcessSpawner, Spawner
-from .. import orm
+from ..user import User
+from ..utils import new_token
+from .test_api import add_user
 from .utils import async_requests
 
 _echo_sleep = """
@@ -270,3 +272,77 @@ def test_inherit_ok():
 
         def poll():
             pass
+
+
+@pytest.mark.gen_test
+def test_spawner_reuse_api_token(db, app):
+    # setup: user with no tokens, whose spawner has set the .will_resume flag
+    user = add_user(app.db, app, name='snoopy')
+    spawner = user.spawner
+    assert user.api_tokens == []
+    # will_resume triggers reuse of tokens
+    spawner.will_resume = True
+    # first start: gets a new API token
+    yield user.spawn()
+    api_token = spawner.api_token
+    found = orm.APIToken.find(app.db, api_token)
+    assert found
+    assert found.user.name == user.name
+    assert user.api_tokens == [found]
+    yield user.stop()
+    # second start: should reuse the token
+    yield user.spawn()
+    # verify re-use of API token
+    assert spawner.api_token == api_token
+    # verify that a new token was not created
+    assert user.api_tokens == [found]
+
+
+@pytest.mark.gen_test
+def test_spawner_insert_api_token(db, app):
+    """Token provided by spawner is not in the db
+    
+    Insert token into db as a user-provided token.
+    """
+    # setup: new user, double check that they don't have any tokens registered
+    user = add_user(app.db, app, name='tonkee')
+    spawner = user.spawner
+    assert user.api_tokens == []
+
+    # setup: spawner's going to use a token that's not in the db
+    api_token = new_token()
+    assert not orm.APIToken.find(app.db, api_token)
+    user.spawner.use_this_api_token = api_token
+    # The spawner's provided API token would already be in the db
+    # unless there is a bug somewhere else (in the Spawner),
+    # but handle it anyway.
+    yield user.spawn()
+    assert spawner.api_token == api_token
+    found = orm.APIToken.find(app.db, api_token)
+    assert found
+    assert found.user.name == user.name
+    assert user.api_tokens == [found]
+    yield user.stop()
+
+
+@pytest.mark.gen_test
+def test_spawner_bad_api_token(db, app):
+    """Tokens are revoked when a Spawner gets another user's token"""
+    # we need two users for this one
+    user = add_user(app.db, app, name='antimone')
+    spawner = user.spawner
+    other_user = add_user(app.db, app, name='alabaster')
+    assert user.api_tokens == []
+    assert other_user.api_tokens == []
+
+    # create a token owned by alabaster that antimone's going to try to use
+    other_token = other_user.new_api_token()
+    spawner.use_this_api_token = other_token
+    assert len(other_user.api_tokens) == 1
+
+    # starting a user's server with another user's token
+    # should revoke it
+    with pytest.raises(ValueError):
+        yield user.spawn()
+    assert orm.APIToken.find(app.db, other_token) is None
+    assert other_user.api_tokens == []

jupyterhub/user.py

@@ -201,6 +201,7 @@ class User(HasTraits):
             authenticator=self.authenticator,
             config=self.settings.get('config'),
             proxy_spec=url_path_join(self.proxy_spec, name, '/'),
+            db=self.db,
         )
         # update with kwargs. Mainly for testing.
         spawn_kwargs.update(kwargs)
@@ -317,8 +318,6 @@ class User(HasTraits):
         url of the server will be /user/:name/:server_name
         """
         db = self.db
-        if self.allow_named_servers and not server_name:
-            server_name = default_server_name(self)
 
         base_url = url_path_join(self.base_url, server_name) + '/'
 
@@ -356,11 +355,10 @@ class User(HasTraits):
                 oauth_client = client_store.fetch_by_client_id(client_id)
             except ClientNotFoundError:
                 oauth_client = None
-            # create a new OAuth client + secret on every launch,
-            # except for resuming containers.
-            if oauth_client is None or not spawner.will_resume:
+            # create a new OAuth client + secret on every launch
+            # containers that resume will be updated below
             client_store.add_client(client_id, api_token,
-                                        url_path_join(self.url, 'oauth_callback'),
+                                    url_path_join(self.url, server_name, 'oauth_callback'),
                                     )
         db.commit()
 
@@ -369,7 +367,7 @@ class User(HasTraits):
         if (authenticator):
             yield gen.maybe_future(authenticator.pre_spawn_start(self, spawner))
 
-        spawner._spawn_pending = True
+        spawner._start_pending = True
         # wait for spawner.start to return
         try:
             # run optional preparation work to bootstrap the notebook
@@ -385,12 +383,38 @@ class User(HasTraits):
                 # prior to 0.7, spawners had to store this info in user.server themselves.
                 # Handle < 0.7 behavior with a warning, assuming info was stored in db by the Spawner.
                 self.log.warning("DEPRECATION: Spawner.start should return (ip, port) in JupyterHub >= 0.7")
-            if spawner.api_token != api_token:
+            if spawner.api_token and spawner.api_token != api_token:
                 # Spawner re-used an API token, discard the unused api_token
                 orm_token = orm.APIToken.find(self.db, api_token)
                 if orm_token is not None:
                     self.db.delete(orm_token)
                     self.db.commit()
+                # check if the re-used API token is valid
+                found = orm.APIToken.find(self.db, spawner.api_token)
+                if found:
+                    if found.user is not self.orm_user:
+                        self.log.error("%s's server is using %s's token! Revoking this token.",
+                            self.name, (found.user or found.service).name)
+                        self.db.delete(found)
+                        self.db.commit()
+                        raise ValueError("Invalid token for %s!" % self.name)
+                else:
+                    # Spawner.api_token has changed, but isn't in the db.
+                    # What happened? Maybe something unclean in a resumed container.
+                    self.log.warning("%s's server specified its own API token that's not in the database",
+                        self.name
+                    )
+                    # use generated=False because we don't trust this token
+                    # to have been generated properly
+                    self.new_api_token(spawner.api_token, generated=False)
+                # update OAuth client secret with updated API token
+                if oauth_provider:
+                    client_store = oauth_provider.client_authenticator.client_store
+                    client_store.add_client(client_id, spawner.api_token,
+                                            url_path_join(self.url, server_name, 'oauth_callback'),
+                                            )
+                    db.commit()
+
         except Exception as e:
             if isinstance(e, gen.TimeoutError):
                 self.log.warning("{user}'s server failed to start in {s} seconds, giving up".format(
@@ -409,6 +433,7 @@ class User(HasTraits):
                     user=self.name,
                 ), exc_info=True)
             # raise original exception
+            spawner._start_pending = False
             raise e
         spawner.start_polling()
 
@@ -448,9 +473,12 @@ class User(HasTraits):
         else:
             server_version = resp.headers.get('X-JupyterHub-Version')
             _check_version(__version__, server_version, self.log)
+            # record the Spawner version for better error messages
+            # if it doesn't work
+            spawner._jupyterhub_version = server_version
         finally:
             spawner._waiting_for_response = False
-            spawner._spawn_pending = False
+            spawner._start_pending = False
         return self
 
     @gen.coroutine
@@ -461,6 +489,7 @@ class User(HasTraits):
         """
         spawner = self.spawners[server_name]
         spawner._spawn_pending = False
+        spawner._start_pending = False
         spawner.stop_polling()
         spawner._stop_pending = True
         try:

jupyterhub/utils.py

@@ -142,7 +142,8 @@ def wait_for_server(ip, port, timeout=10):
         ip = '127.0.0.1'
     yield exponential_backoff(
         lambda: can_connect(ip, port),
-        "Server at {ip}:{port} didn't respond in {timeout} seconds".format(ip=ip, port=port, timeout=timeout)
+        "Server at {ip}:{port} didn't respond in {timeout} seconds".format(ip=ip, port=port, timeout=timeout),
+        timeout=timeout
     )
 
 
@@ -175,7 +176,8 @@ def wait_for_http_server(url, timeout=10):
         return False
     re = yield exponential_backoff(
         is_reachable,
-        "Server at {url} didn't respond in {timeout} seconds".format(url=url, timeout=timeout)
+        "Server at {url} didn't respond in {timeout} seconds".format(url=url, timeout=timeout),
+        timeout=timeout
     )
     return re
 

requirements.txt

@@ -4,5 +4,5 @@ tornado>=4.1
 jinja2
 pamela
 python-oauth2>=1.0
-SQLAlchemy>=1.0
+SQLAlchemy>=1.1
 requests

share/jupyter/hub/templates/admin.html

@@ -32,9 +32,9 @@
     <tbody>
       <tr class="user-row add-user-row">
         <td colspan="12">
-          <a id="add-users" class="col-xs-2 btn btn-default">Add Users</a>
-          <a id="stop-all-servers" class="col-xs-2 col-xs-offset-5 btn btn-danger">Stop All</a>
-          <a id="shutdown-hub" class="col-xs-2 col-xs-offset-1 btn btn-danger">Shutdown Hub</a>
+          <a id="add-users" role="button" class="col-xs-2 btn btn-default">Add Users</a>
+          <a id="stop-all-servers" role="button" class="col-xs-2 col-xs-offset-5 btn btn-danger">Stop All</a>
+          <a id="shutdown-hub" role="button" class="col-xs-2 col-xs-offset-1 btn btn-danger">Shutdown Hub</a>
         </td>
       </tr>
   {% for u in users %}
@@ -44,20 +44,20 @@
       <td class="admin-col col-sm-2">{% if u.admin %}admin{% endif %}</td>
       <td class="time-col col-sm-3">{{u.last_activity.isoformat() + 'Z'}}</td>
       <td class="server-col col-sm-2 text-center">
-        <span class="stop-server btn btn-xs btn-danger {% if not u.running %}hidden{% endif %}">stop server</span>
-        <span class="start-server btn btn-xs btn-success {% if u.running %}hidden{% endif %}">start server</span>
+        <span role="button" class="stop-server btn btn-xs btn-danger {% if not u.running %}hidden{% endif %}">stop server</span>
+        <span role="button" class="start-server btn btn-xs btn-success {% if u.running %}hidden{% endif %}">start server</span>
       </td>
       <td class="server-col col-sm-1 text-center">
         {% if admin_access %}
-        <span class="access-server btn btn-xs btn-success {% if not u.running %}hidden{% endif %}">access server</span>
+        <span role="button" class="access-server btn btn-xs btn-success {% if not u.running %}hidden{% endif %}">access server</span>
         {% endif %}
       </td>
       <td class="edit-col col-sm-1 text-center">
-        <span class="edit-user btn btn-xs btn-primary">edit</span>
+        <span role="button" class="edit-user btn btn-xs btn-primary">edit</span>
       </td>
       <td class="edit-col col-sm-1 text-center">
         {% if u.name != user.name %}
-        <span class="delete-user btn btn-xs btn-danger">delete</span>
+        <span role="button" class="delete-user btn btn-xs btn-danger">delete</span>
         {% endif %}
       </td>
       {% endblock user_row %}

share/jupyter/hub/templates/error.html

@@ -22,6 +22,11 @@
     {{message_html | safe}}
   </p>
   {% endif %}
+  {% if extra_error_html %}
+  <p>
+    {{extra_error_html | safe}}
+  </p>
+  {% endif %}
   {% endblock error_detail %}
 </div>
 

share/jupyter/hub/templates/home.html

@@ -6,9 +6,9 @@
   <div class="row">
     <div class="text-center">
       {% if user.running %}
-      <a id="stop" class="btn btn-lg btn-danger">Stop My Server</a>
+      <a id="stop" role="button" class="btn btn-lg btn-danger">Stop My Server</a>
       {% endif %}
-      <a id="start" class="btn btn-lg btn-success" href="{{ url }}">
+      <a id="start"role="button" class="btn btn-lg btn-success" href="{{ url }}">
       {% if not user.running %}
       Start
       {% endif %}

share/jupyter/hub/templates/login.html

@@ -8,10 +8,10 @@
 {% block login %}
 <div id="login-main" class="container">
 {% if custom_html %}
-{{ custom_html }}
+{{ custom_html | safe }}
 {% elif login_service %}
 <div class="service-login">
-  <a class='btn btn-jupyter btn-lg' href='{{authenticator_login_url}}'>
+  <a role="button" class='btn btn-jupyter btn-lg' href='{{authenticator_login_url}}'>
     Sign in with {{login_service}}
   </a>
 </div>

share/jupyter/hub/templates/page.html

@@ -99,9 +99,9 @@
         <ul class="nav navbar-nav">
           <li><a href="{{base_url}}home">Home</a></li>
           <li><a href="{{base_url}}token">Token</a></li>
-          {% endif %}
           {% if user.admin %}
           <li><a href="{{base_url}}admin">Admin</a></li>
+          {% endif %}
         </ul>
         {% endif %}
         <ul class="nav navbar-nav navbar-right">
@@ -109,9 +109,9 @@
             {% block login_widget %}
               <span id="login_widget">
                 {% if user %}
-                  <a id="logout" class="navbar-btn btn-sm btn btn-default" href="{{logout_url}}"> <i aria-hidden="true" class="fa fa-sign-out"></i> Logout</a>
+                  <a id="logout" role="button" class="navbar-btn btn-sm btn btn-default" href="{{logout_url}}"> <i aria-hidden="true" class="fa fa-sign-out"></i> Logout</a>
                 {% else %}
-                  <a id="login" class="btn-sm btn navbar-btn btn-default" href="{{login_url}}">Login</a>
+                  <a id="login" role="button" class="btn-sm btn navbar-btn btn-default" href="{{login_url}}">Login</a>
                 {% endif %}
               </span>
               {% endblock %}

share/jupyter/hub/templates/spawn_pending.html

@@ -8,7 +8,7 @@
       <p>Your server is starting up.</p>
       <p>You will be redirected automatically when it's ready for you.</p>
       <p><i class="fa fa-spinner fa-pulse fa-fw fa-3x" aria-hidden="true"></i></p>
-      <a id="refresh" class="btn btn-lg btn-primary" href="#">refresh</a>
+      <a role="button" id="refresh" class="btn btn-lg btn-primary" href="#">refresh</a>
     </div>
   </div>
 </div>

share/jupyter/hub/templates/token.html

@@ -5,7 +5,7 @@
 <div class="container">
   <div class="row">
     <div class="text-center">
-      <a id="request-token" class="btn btn-lg btn-jupyter" href="#">
+      <a id="request-token" role="button" class="btn btn-lg btn-jupyter" href="#">
         Request new API token
       </a>
     </div>

singleuser/Dockerfile

@@ -0,0 +1,12 @@
+# Build as jupyterhub/singleuser
+# Run with the DockerSpawner in JupyterHub
+
+ARG BASE_IMAGE=jupyter/base-notebook
+FROM $BASE_IMAGE
+MAINTAINER Project Jupyter <jupyter@googlegroups.com>
+
+ADD install_jupyterhub /tmp/install_jupyterhub
+ARG JUPYTERHUB_VERSION=master
+# install pinned jupyterhub and ensure notebook is installed
+RUN python3 /tmp/install_jupyterhub && \
+    python3 -m pip install notebook

singleuser/README.md

@@ -0,0 +1,36 @@
+# jupyterhub/singleuser
+
+Built from the `jupyter/base-notebook` base image.
+
+This image contains a single user notebook server for use with
+[JupyterHub](https://github.com/jupyterhub/jupyterhub). In particular, it is meant
+to be used with the
+[DockerSpawner](https://github.com/jupyterhub/dockerspawner/blob/master/dockerspawner/dockerspawner.py)
+class to launch user notebook servers within docker containers.
+
+The only thing this image accomplishes is pinning the jupyterhub version on top of base-notebook.
+In most cases, one of the Jupyter [docker-stacks](https://github.com/jupyter/docker-stacks) is a better choice.
+You will just have to make sure that you have the right version of JupyterHub installed in your image,
+which can usually be accomplished with one line:
+
+```Dockerfile
+FROM jupyter/base-notebook:5ded1de07260
+RUN pip3 install jupyterhub==0.7.2
+```
+
+The dockerfile that builds this image exposes `BASE_IMAGE` and `JUPYTERHUB_VERSION` as build args, so you can do:
+
+    docker build -t singleuser \
+      --build-arg BASE_IMAGE=jupyter/scipy-notebook \
+      --build-arg JUPYTERHUB_VERSION=0.8.0 \
+      .
+
+in this directory to get a new image `singleuser` that is based on `jupyter/scipy-notebook` with JupyterHub 0.8, for example.
+
+This particular image runs as the `jovyan` user, with home directory at `/home/jovyan`.
+
+## Note on persistence
+
+This home directory, `/home/jovyan`, is *not* persistent by default,
+so some configuration is required unless the directory is to be used
+with temporary or demonstration JupyterHub deployments.

singleuser/hooks/build

@@ -0,0 +1,11 @@
+#!/bin/bash
+set -ex
+
+stable=0.7
+
+for V in master 0.7; do
+    docker build --build-arg JUPYTERHUB_VERSION=$V -t $DOCKER_REPO:$V .
+done
+
+echo "tagging $IMAGE_NAME"
+docker tag $DOCKER_REPO:$stable $IMAGE_NAME

singleuser/hooks/post_push

@@ -0,0 +1,23 @@
+#!/bin/bash
+
+for V in master 0.7; do
+  docker push $DOCKER_REPO:$V
+done
+
+function get_hub_version() {
+  rm -f hub_version
+  V=$1
+  docker run --rm -v $PWD:/version -u $(id -u) -i $DOCKER_REPO:$V sh -c 'jupyterhub --version > /version/hub_version'
+  hub_xyz=$(cat hub_version)
+  split=( ${hub_xyz//./ } )
+  hub_xy="${split[0]}.${split[1]}"
+}
+# tag e.g. 0.7.2 with 0.7
+get_hub_version 0.7
+docker tag $DOCKER_REPO:0.7 $DOCKER_REPO:$hub_xyz
+docker push $DOCKER_REPO:$hub_xyz
+
+# tag e.g. 0.8 with master
+get_hub_version master
+docker tag $DOCKER_REPO:master $DOCKER_REPO:$hub_xy
+docker push $DOCKER_REPO:$hub_xy

singleuser/install_jupyterhub

@@ -0,0 +1,21 @@
+#!/usr/bin/env python
+import os
+from subprocess import check_call
+import sys
+
+V = os.environ['JUPYTERHUB_VERSION']
+
+pip_install = [
+    sys.executable, '-m', 'pip', 'install', '--no-cache', '--upgrade',
+    '--upgrade-strategy', 'only-if-needed',
+]
+if V == 'master':
+    req = 'https://github.com/jupyterhub/jupyterhub/archive/master.tar.gz'
+else:
+    version_info = [ int(part) for part in V.split('.') ]
+    version_info[-1] += 1
+    upper_bound = '.'.join(map(str, version_info))
+    vs = '>=%s,<%s' % (V, upper_bound)
+    req = 'jupyterhub%s' % vs
+
+check_call(pip_install + [req])