Updated capitalisation of names. Addressed revisions.
Fleshed out the prerequists and explanation of access control.
Added part of configuration section to set JupyterLab as the default interface.
corrected need for sudo
Added warning to reverse-proxy section to recommend use of HTTPS and firewall.
- A trivial bug caused by my last change to #2397 - made possible by
the fact we didn't have a way to reliable test PAM stuff.
- Thanks to @narnish for noticing.
- Closes: #2875
- We now default to ubuntu bionic (18.04) and try once with ubuntu xenial
(16.04).
- We now always test Python 3.8 but allow it to fail, as compared to not
allowing it to fail and only testing it on tagged commits. This is a
bugfix I'd say.
- We now no longer test Python 3.5 and Python 3.6 dedicatedly without
any custom configuration like usage of subdomain, which allows us to
reduce the number of build jobs in a way I think makes a great sense to
compromise.
Some notes:
- Added a conda-forge and DockerHub badge
- Added logo's and made us conform with the team-compass badges section
as can be found here:
https://jupyterhub-team-compass.readthedocs.io/en/latest/building-blocks/readme-badges.html
- Concluded that our CircleCI badge is good because it let's us overview
the repo's build systems, but that it is bad because it is only is about
documentation preview in PRs which isn't useful in a README's header in
a way.
- Noted there was a CircleCI token in the badge, that I believe is meant
to be used with private repo access rather than public repo access. I'm
not sure we need that but I made it a markdown/html comment for now.
- Decided to not manually add a line break between badges. I figured it
could make sense to break manually before the social badges instead of
automatically letting it wrap at some point, but we don't really know
the size of the window viewing so it felt like a bad idea to hardcode
that.
- When the Dockerfile was turned into a multi-stage build, it seems
the share/ directory was not copied to the final image. This
resulted in certain components (static/components/, static/css/)
being missing, which resulted in the JupyterHub share directory not
being findable (in jupyterhub/_data.py). This led to all kinds of
weird havoc, like templates not being findable (#2852).
- I am still unsure if this is the right fix, please check this well.
- Closes: #2852
- While debugging another problem, I noticed some failures to build
the C extensions in the logs. Adding build-essential should fix
that (also as mentioned in the logs themselves).
- Extensions failed for tornado, sqlalchemy, and pyrsistent(pvectorc)
and can be found by searching the previous output for "fail".
Closes#2819 by exiting JupyterHub directly with an error if a config
file has been specified for the config_file traitlet, for example
through the -f or --config flag, but isn't available on the file
system.
- In the cull script, the max_age and inactive_limit are used from the
outer scope. In the case that you add extra logic, one may want to
modify these values.
- In that case, you either have to rename them locally, or access the
outer scope with "nonlocal", the first of which is too much work,
the second of which has a high chance of introducing bugs (as it did
for me).
- This change introduces a fix for everyone. It doesn't change basic
functionality, but makes local modifications simpler.
- Pass in user object & request object only explicitly.
Much better interface that is harder to break by internal
refactoring. We can always add more parameters if needed?
/user-redirect/ is used to help link to a particular url
in the logged in user's authenticated notebook. For example,
if I'm logged in as user 'yuvipanda' and hit the URL
/hub/user-redirect/git-pull, it'll redirect me to
/user/yuvipanda/git-pull. This is extremely useful in
connecting hub links to notebook server extensions, such
as nbgitpuller.
Admins might want to customize how this redirection is done -
for example, redirect users to different running servers
based on the nbgitpuller repository they are linking from.
Adding a hook here helps accomplish that.
allows services to be explicitly blessed to skip the extra oauth confirmation page
added in 1.0
This confirmation page is unhelpful for many admin-managed services,
and is mainly intended for cross-user access.
The default behavior is unchanged, but services can now opt-out of confirmation
(as is done already for the user's own servers).
Use with caution, as this eliminates users' ability to confirm that a service
should be able to authenticate them.
- API requests to non-running servers are not uncommon when you cull
servers and people leave tabs open and active. It returns with 503
and logs all headers, which can take up half of our total log lines
- This avoids logging headers for all 502 and 503 return statuses.
#2747 presented an alternative (more complex) implementation, but this
turned out to be appropriate.
- Closes: #2747
In current versions of MySQL and MariaDB `innodb_file_format`
and `innodb_large_prefix` have been removed. This allows them to not
exist and makes sure the format for the rows are `Dynamic` (default
for current versions).
If init_spawners takes too long (default: 10 seconds) to complete,
app start will be allowed to continue while finishing in the background.
Adds new `check` pending state for the initial check.
Checking lots of spawners can take a long time,
so allowing this to be async limits the impact on startup time
at the expense of starting the Hub in a not-quite-fully-ready state.
- Introduce the EventLog class from BinderHub for emitting
structured event data
- Instrument server starts and stops to emit events
- Defaults to not saving any events anywhere
The flask example in the documentation was still using the
input argument `cookie_cache_max_age` when instantiating
`HubAuth` object. `cookie_cache_max_age` is deprecated since
JupyterHub 0.8 and should be replaced by `cache_max_age`.
- Install pip in the docs conda env (or conda complains).
- Do not override page.html, the next/previous buttons are now handled by
alabaster_jupyterhub (this actually remove the duplicated next/prev
buttons)
- use alabaster_jupyterhub when building locally, this make it easy for
new contributor to get the _exact_ same appearance than on
readthedocs.
- cull_idle_servers.py gets the full server state, so is capable of
doing any kind of arbitrary logic on the profile in order to be more
flexible in culling.
- This patch does not change anything, but gives an embedded
(commented out) example of how you can easily add custom logic to
the script.
- This was added as a tempate/demo for #2598.
* Add missing responses (doesn't include all possible responses yet)
* Refactor invalid multi in body parameters into a single parameter
* Change form type into valid formData
* Fix use of required fields
* Apply a few other minor fixes
Fixes https://github.com/jupyterhub/jupyterhub/issues/2566 to some
degree by making the announcement stand out using twitter-bootstrap
classes `alert` and `alert-warning`. Perhaps we could theme twitter
bootstrap or this alert specifically with jupyter related colors as well
though?
Windows doesn't have support for signal handling so it can't use the
signal handling capabilities of asyncio. Use the previous atexit
strategy on the Windows case instead.
Signed-off-by: Alejandro Del Castillo <alejandro.delcastillo@ni.com>
Big thanks to Erik, Tim, and Min for the great comments!
Change names to be more clear, add function doc comments,
change scoping on some functions, add handle_logout to let
people take custom logout actions, extract
render_logout_page from get method, add TODO.
AS A developer of a Logout handler
I WANT to be able to call a function to kill spawners and
do other backend logout stuff and a separate function to
forward the user along the logout chain.
I believe this PR adds (moderately private) methods to the
Logout Handler to do just that.
update several links (html targets don't work anymore)
had to add rest-api redirect so link would resolve,
since there isn't a ref for files in _static
- /user/:name no longer triggers implicit spawn at any point
- add /spawn-pending/:user/:server handler for pending page. This page has no side effects.
- spawn links point to /spawn/:user/:server to finish hooking up links for named servers and options_form handling
- It took me a bit longer than I would have liked for me to figure out
how to run the proxy separate from the hub. When I had to do this a
second time for a different hub, it also took me too long.
- This adds a page dedicated to running the proxy separate from the
hub, since it is relatively easy and has a high usability
improvement.
- Currently work in progress.
TEXT is wrong on Oracle, LargeBinary is wrong everywhere else.
Text seems to be the high-level type that maps to the right thing both places.
This results in no change on supported implementations, as Text == TEXT there.
- We don't need the extra normalization of that function.
- Also add in username_map support here. It probably isn't needed
most of the time with PAM, but it keeps things consistent and is
easier than documenting an exception.
Traitlets require quotes around literals, to avoid interpreting them as
as datatypes other than string. However, quotes are problematic on the
notebook_dir case. On Windows, Popen will mis-interpret the quotes and
escape them, which trips the process spawn. To avoid problems, only
quote if necessary.
Signed-off-by: Alejandro del Castillo <alejandro.delcastillo@ni.com>
adds Authenticator.auth_refresh_age and Authenticator.refresh_pre_spawn config
- auth_refresh_age allows auth to expire (default: 5 minutes) before calling Authenticator.refresh_user.
- refresh_pre_spawn forces refresh prior to spawn (in case of auth tokens, etc.)
this introduces a race between the early RuntimeError being tested
and the no_patience causing handlers to return early if async start isn’t complete.
With tornado coroutines, an early RuntimeError could be guaranteed to resolve promptly, but asyncio isn’t as consistent,
possibly causing some of the recent flaky tests.
Windows doesn't have a pwd module. To avoid an import error on Windows,
move import statement inside functions that use pwd.
Signed-off-by: Alejandro del Castillo <alejandro.delcastillo@ni.com>
The current request handler might be needed to determine if the auth
data needs to be refreshed.
Signed-off-by: Alejandro del Castillo <alejandro.delcastillo@ni.com>
Use setuptools console_scripts functionality to create top level jupyter
& jupyterhub-single user entry point scripts on *nix, and executables on
Windows.
Signed-off-by: Alejandro del Castillo <alejandro.delcastillo@ni.com>
when a token doesn't identify a user, the response is None.
These results are cached, but the cache checked for `is None`,
causing failed-auth responses to effectively not be cached.
Hoist admin status determination from authentication to a secondary function called by get_authenticated_user
Create mock objects for struct_group and struct_passwd, migrate existing mock group objects to it
Remove old admin mock stuff for authenticate
- trust subdomain_host by default
- JupyterHub.trusted_alt_names is inherited by Spawners by default. Do we need Spawner.ssl_alt_names to be separately configurable?
One of the example was using quotes instead of backticks.
Backticks are the "older" way of doing things, which has a number of
disadvantes:
http://mywiki.wooledge.org/BashFAQ/082
Here I'm more worried about readability as depending on font and "smart"
editor helping on the web, many people may confuse ` with ', it could
end up modifying formatting on makrdown powered website... etc...
jupyterhub.authenticators for authenticators, jupyterhub.spawners for spawners
This has the effect that authenticators and spawners can be selected by name instead of full import string (e.g. 'github' or 'dummy' or 'kubernetes')
and, perhaps more importantly, the autogenerated configuration file will include a section for each installed and registered class.
- Expands the previous documentation on upgrading JupyterHub
to include more information.
- Remove specific documentation on 0.7 -> 0.8 upgrade, since
this seems to be a straight copy of the markdown version of
upgrading docs. The important thing about the 0.7 -> 0.8 upgrade
(requiring versions of JupyterHub to match) is now in the
main document.
- Move from markdown to rst
Info on upgrading is important & relevant. This consolidates
the index to be a bit better. Next step is to consolidate the
documentation into one page.
Removes the 'tutorials' index page as well, since that only
had a reference to z2jh (which is now referenced from the
'distribution' section). The distribution section has
better visibility too
Currently, the sections in index.rst are using ** for bold,
rather than true section headers. This prevents them from being
linkable. Since we'd like to link to the 'contributing' section
from CONTRIBUTING.md, we change this by moving everything to
section headers. We also move to the toctree directive, since
it keeps the bullets aligned properly (they were hanging if
we used simple * markers)
This also replaces CONTRIBUTING.md content with a link to
the docs.
- Move from CONTRIBUTING.md to a subdirectory in docs, so
we can expand and add more documentation.
- Move from markdown to reStructuredTest
- Add a direct blurb in the JupyterHub docs index page on
how contribution.
- More prominent link to the Code of Conduct
- Add section on getting in touch with the JupyterHub community
define some pending/ready helpers as static constants on orm.Spawner
allows treating orm.Spawner the same as Spawner wrappers,
as long as `.active` etc. checks are performed first
and generate no events if not pending
Reason: race condition is unavoidable between first pending check and check inside _generate_progress.
In this event, return immediately.
The current list in the docs is out of date. The list
in the wiki is more up-to-date, and easier for folks
to change over time. In the long run, we should decide
where lists like this belong.
- delete oauth clients for servers when they shutdown
- avoid deleting oauth clients for servers still running across an 0.8 -> 0.9 upgrade, when the oauth client ids changed from `user-NAME` to `jupyterhub-user-NAME`
- refresh_user may return True in the common case, identifying that everything is up-to-date
- return False for "needs login"
- return auth_data dict when an update can be performed without logging in again
- `.get_current_user` is called in the `prepare` stage for all handlers
- use `.current_user` to access current user in methods
- adds Authenticator.refresh_user for refreshing user auth (unused at this point)
With changes to CHP requiring a second, different
authority, the complexity of managing trust within
JupyterHub has risen. To solve this, Certipy now
has a feature to specify what components should
trust what and builds trust bundles accordingly.
Mainly small fixes, but the token page could be completely broken
This release will include the spawner.handler addition,
but not the oauthlib change currently in master
typos in token expiry:
- omitted from token model (it's in the spec in docs, but wasn't in the model)
- wrong type when sorting oauth tokens on token page could cause token page to not render
Windows doesn't support signal.SIGKILL, which is used by
_check_previous_process to kill the CHP if still running. Use existing
implementation to kill the CHP and children processes on Windows
instead.
Signed-off-by: Alejandro del Castillo <alejandro.delcastillo@ni.com>
Previously, signal.SIGTERM was using 3 times, instead of using it 2
times, then signal.SIGKILL.
Signed-off-by: Alejandro del Castillo <alejandro.delcastillo@ni.com>
To better accommodate external certificate management
as well as building of trust, Certipy was refactored.
This included general improvements to file and
record handling. In the process, some of Certipy's
APIs changed slightly, but should be more stable now
going forward.
avoids issues with proxies dropping connections when no data passes through
Progress behavior should already be resilient to dropped connections,
as the progress ought to just resume anew.
When request uri matching with base_url in PrefixRedirectHandler,
it's better to ensure uri with tariling slash. That's will avoid
redirecting /foobar to /foobar/hub/foobar.
Currently, to check if the proxy is running, os.kill(pid,0) is used,
which doesn't work on Windows. Wrapped call into a new function that
adds a Windows case.
Signed-off-by: Alejandro del Castillo <alejandro.delcastillo@ni.com>
Setup general ssl request, not just to api
Basic tests comprised of non-ssl test copies
Create the context only when request is http
Refactor ssl key, cert, ca names
Configure the AsyncHTTPClient at app start
Change tests to import existing ones with ssl on
Override __new__ in MockHub to turn on SSL
Add Localhost to trusted alt names
Update to match refactored certipy names
Add the FQDN to cert alt names for hub
Ensure notebooks do not trust each other
Drop certs in user's home directory
Refactor cert creation and movement
Make alt names configurable
Make attaching alt names more generic
Setup ssl_context for the singleuser hub check
The Hub will exit if consecutive failure count reaches this threshold
Any successful spawn will reset the count to 0
useful for auto-restarting / self-healing deployments such as kubernetes/systemd/docker where restarting the Hub
default is disabled, since it would bring down the Hub if it’s not an auto-restarting deployment
raise SystemExit on sigterm instead of calling atexit directly
- ensure fresh asyncio eventloop is created (not just IOLoop)
- makes cleanup more likely to run (one source of orphaned proxies)
allows Spawners to implement logic such as processing GET params to select inputs
USE WITH CARE because this gives authors of links the ability to pass parameters to spawn without user knowledge or input.
This should only be used for things like selecting from a list of all known-good choices, e.g. a profile list.
this is the routespec for sending requests to the hub
It is [host]/prefix/ (not /hub/) so it receives all
requests, not just those destined for the hub
When the Hub listens on all ips by default, the connection ip is the hostname.
in some cases (e.g. certain kubernetes deployments) the hub’s container’s hostname is not connectable from itself, preventing managed services from connecting to the hub.
This ensures that managed service processes talk to the hub over localhost in this case, rather than via the hostname.
JupyterHub packages are in the 'conda-forge' channel of Anaconda packages; if the Anaconda installation doesn't already have 'conda-forge' enabled, `conda install jupyterhub` fails.
Rather than adding instructions to enable 'conda-forge' in Anaconda, this patch modifies the installation command to specify that channel.
Added a command that worked for me to fix the situation that localhost:8000 is unable to reach the hub even though the published command for Docker exposes the correct port.
- Using the new template_vars setting (#1872), allow the variable
`announcement` to create a header message on all the pages in the
title, or the variables `announcement_{home,login,logout,spawn}` to
set variables on these single pages.
- This is not the most powerful method of putting an announcement into
the templates, because it requires a server restart to change. But
the invasiveness is very low, and allows minimal message
without having to touch the templates themselves.
- Closes: #1836
escape with `_` instead of `%`.
This is not technically rigorous, as collisions are possible (users foo_40 and foo@ have the same domain)
and other domain restrictions are not applied (length, starting characters, etc.).
Username normalization can be used to apply stricter, more rigorous structure.
Addresses issue #1747.
These additions aren't perfect -- it's unfortunate that I've added
mention of reverse proxies on two separate pages. I don't _think_
these can reasonably be put on the same page -- perhaps a cross
reference?
- This message is presented when the last spawn failed, along with a
HTTP 500. The current text is quite confusing, especially when the
problem may just be solvable by trying to respawn again.
On the Windows case, the configurable-http-proxy is spwaned using a
shell. To stop the proxy, we need to terminate both the main process
(shell) and its child (proxy).
Signed-off-by: Alejandro del Castillo <alejandro.delcastillo@ni.com>
from the sqlalchemy docs
checks if a connection is valid via `SELECT 1` prior to using it.
Since we have long-running connections, this helps us survive database restarts, disconnects, etc.
- update config to single tuple instead of two integers
- call it spawn_throttle_retry_range
- fix setting Retry-After header without disabling error pages
enables `c.JupyterHub.bind_url = 'unix+http://%2Fsrv%2Fjupyterhub%2Fjupyterhub.sock'`
for listening on a bsd socket.
Similarly, bind_url and connect_url work as overrides everywhere
Add the hub_socket option to the JupyterHub class, which takes
precedence over the hub_ip and hub_port setting. It does not forward
this setting to the Hub class though, and a few log messages still say
the hub is listening on `http://:8000` that works fine when testing with
netcat:
```
$ nc -U /tmp/jhub.sock
GET /login HTTP/1.1
HTTP/1.1 302 Found
Server: TornadoServer/4.5.1
Content-Type: text/html; charset=UTF-8
Date: Fri, 28 Jul 2017 02:05:36 GMT
X-Jupyterhub-Version: 0.8.0.dev
Content-Security-Policy: frame-ancestors 'self'; report-uri /hub/security/csp-report
Location: /hub/login
Content-Length: 0
```
Should still be better documented I guess.
- Instead of creating many options for different timeouts of users and
servers, just add a note that the whole culler can be run multiple
times with different options. See discussion in #1834.
- Closes: #1834
- This will allow, for example, cull_idle_servers to be more
intelligent when culling servers.
- This is only given to admin API users, because we don't know if all
spawners expect their state to be made available to users.
in order to use sqlalchemy's expire_on_commit=False optimization,
we need to make sure that objects are kept up to date.
This means we cannot rely on ForeignKey ondelete/onupdate behavior,
we must use sqlalchemy's local relationship cascades
The main key here is that we must use relationships to set foreign-key relations,
e.g. APIToken.user = user instead of APIToken.user_id = user.id.
It also means that we cannot use passive_deletes,
which allows sqlalchemy to defer to the database's more efficient ON DELETE behavior.
This makes deletions more expensive in particular,
but should improve db performance overall.
allows opening an IPython shell with a connection to your database
alembic moved from `python -m jupyterhub.dbutil` to `python -m jupyterhub.dbutil alembic` subcommand
function-scoped fixture for shutting down servers
avoids servers leaking into neighbor tests without having to teardown the app itself after every test
allows iterating through an async generator, yielding items until another Future resolves
if/when that deadline Future resolves, ready items will continue to be yielded until there is one that actually needs to wait
at which point the iteration will halt
- add Spawner.progress method. Must be an async generator of JSON-able progress events
- add /api/users/:user/server-progress eventstream endpoint
- use eventstream to fill progress bar on the spawn pending page
use iso8601 Z suffix for UTC timestamps
use dateutil to parse dates from proxy, as well
even though CHP uses iso8601 UTC timestamps, we no longer assume CHP, so use more general parsing
in our db we are stuck with naïve datetime objects, so use those internally.
But ensure we put 'Z' on timestamps we ship externally
Now that it's unambiguous whether the proxy should start or not,
we don't need a warning about generating tokens causing issues for hub restart.
We can raise a strict, early error if proxy s external and token is still unspecified,
rather than running into a 403 error due to a generated token
JH can now differentiate between authenticated and authorized users via PAM
This allows JH to respect PAM-accessible user access controls.
This also fixes missing PAMAuthenticator.encoding usages.
- ignore spawner state when determining redirect destination
- remove implicit spawn from login handler (rely on redirect to user.url for spawn)
- settings.redirect_to_server determines if login sends users to /user/:name vs /hub/home
- visiting `/hub/` should result in the same destination regardless of login state or spawner state
asyncio has different coroutine start mechanics than tornado
tornado starts coroutines immediately,
whereas asyncio doesn't until they are scheduled with either ensure_future or waited upon.
In part to cleanup a few remnants of early design where jupyterhub was ‘jupyter-hub’ instead of ‘jupyterhub’.
Should also clarify to some degree what the cookies are for.
- hub login cookie is now ‘jupyterhub-hub-login’ instead of ‘jupyter-hub-token’
- user server cookie is now ‘jupyterhub-user-<name>’ instead of ‘user-name’ to keep jupyterhub prefix on all cookies
All cookies at this point:
- jupyterhub-session-id on /
- jupyterhub-hub-login on /hub/ (the main login cookie)
- jupyterhub-services on /services/
- jupyterhub-user-<name> on /user/:name
- jupyterhub-user-<name>-oauth-state on /user/:name during oauth
send SIGINFO (ctrl-T) to jupyterhub and it will dump
process info
- if psutil is available, show cpu, memory, FD counts
- always show stacks of non-idle threads
avoids filling up hidden .Trash directory when files are deleted
since there's no UI for trash in a jupyterhub deployment, this mainly results in files never being deleted and possibly filling up disks
put the Spawner instance back so it can reuse the token
'real' reuse cases don't need this because the info is stored in their own storage,
e.g. a stopped container.
HasTraits are expensive to instantiate, so make Users as light as possible
Removes immediate instantiation of Spawners during User init. Spawners will only be instantiated while running
Avoids instantiating too many objects before they are used
- deletes Spawner instances after they stop to avoid lingering instances
- use user_dict cache more often instead of db queries
- check for empty spawners dict to avoid a few Spawner instantiations
puts each check for a running spawner in a coroutine and runs them all concurrently.
Note: this will only improve performance when a large number of Spawners are running and `yield spawner.poll()` takes a nontrivial amount of time.
This is because these are coroutines, not threads. If instantiating Spawners themselves takes a long time, performance will not be affected.
This is still causing problems in all fresh deployments we are doing. I am fine with another solution, but at least wanted to proposed this as a fix for now.
If true, the user can have custom templates (specified in
template_paths) that extend the default templates, by referencing them
as "BASE:filename.html". This makes it easier to add information to
exising templates.
- ._running private attribute is removed. We don't need it anymore,
since we were only using it while the application was run in a background thread.
- call blocking cleanup in a thread because asyncio doesn't allow multiple loops in one thread.
Try to hit every possible exit point from the spawn_single_server
method, with an appropriate status code.
The default histogram buckets are also meant for request latencies,
but spawning usually takes longer so we use custom buckets
This patch introduces Prometheus for exposing metrics
about JupyterHub's operation. We expose a standard /metrics
endpoint that can be queried without authentication. We
take on prometheus_client as an unconditional dependency
to both simplify code & because it is a pure python package
with no dependencies itself.
The first pass adds 'RED' style metrics for all HTTP requests.
http://rancher.com/red-method-for-prometheus-3-key-metrics-for-monitoring/
has some info on the RED method, but to summarize:
For each request type, record at least the following metrics
Rate – the number of requests, per second, your services are serving.
Errors – the number of failed requests per second.
Duration – The amount of time each request takes expressed as a time interval.
This instantly gives us a lot of useful metrics in a very
compact form.
.terminate() only sends the signal,
it doesn't wait for the process to exit.
If the process doesn't exit promptly,
the next instance may try to grab the port before the previous process has released it,
causing failure with EADDRINUSE.
and add loud warning about discarding information
this has been the cause of many debugging difficulties,
when redirecting output seems to be a better option in ~all cases.
- add `authorization` to default Access-Control-Allow-Headers
- allow overriding `Access-Control-Allow-Headers` just like everything else in case default is inappropriate
- ensure case-insensitive comparison for proper header checks
Upgrades the database (if needed) on start.
This is opt-in, for uses like the helm chart where explicit 'upgrade-db' steps are hard to insert.
This ought to be safe for sqlite users, where an automatic backup file is created *if an upgrade will occur*.
In 0.8, the jupyterhub api token can also be used to make requests to
hte jupyter notebook given some conditions. This commit updates that
documentation
Installing the loop instructs the tornado loop to point to the ayncio loop and use
that. IOLoop.configure told the tornado loop to create a new ioloop when
a loop was needed, which is not what we want.
in case of multiple simultaneous
- state arg is strictly required now
- default cookie name in case of no collision is unchanged
- in case of collision, randomize cookie name with a suffix and store cookie_name in state
- expire state cookies after 10 minutes, not 1 day
- set ONDELETE='set null' on spawner->server relation (fixes error when deleting servers that stopped)
- set `spawner.server = None`, which is not triggered when deleting orm_spawner.server
proxies may not route `/user/name` correctly, only `/user/name/...`, so make sure that `/user/name` is redirected to `/user/name/`
this manifests as a redirect loop between /user/name and /hub/user/name when a route exists but /user/name is still
being routed to the Hub
so the error message is the same, however it was arrived at.
potential downside: it could look like the current request is spawning and failing,
rather than the reality that a previous spawn failed and we are just re-presenting the earlier error.
It's possible for there to have been a long time in between spawn and error.
require users to visit /hub/home and click 'Start My Server' to get a new server
Visits to /hub/user/:name will get an error if the previous spawn failed,
rather than triggering a new spawn.
This should guarantee that a user sees an error if their spawn failed,
regardless of when the failure occurred and how long it took.
Some cases of slow errors could result in triggering a new spawn indefinitely without
the user seeing an error message.
/hub/spawn was a simple redirect to /user/:name in the absence of a spawn form,
but now clears the `_spawn_future` prior to redirect
to signal that a new spawn has been explicitly requested in the case of a prior failure.
this is the most likely cause of redirect loops when using docker,
so record the spawner version and check it when a redirect is detected.
In the event of a redirect and mismatch, fail with a message explaining the version mismatch and how to fix it.
1. set _proxy_pending before first wait to ensure that there is never a gap between setting spawn flags
2. always call `finish_user_spawn` to reduce the number of finalization cases
3. wait for proxy to finish on the slow_spawn timeout, not just start, because we are only interested in the total duration for page responsiveness
handle will_resume case correctly, where an API token *may* be re-used.
Previously, we only did it right if the token was *always* reused,
but clearing out a container would get it into a bad state.
ensures that singleuser cookies do not persist across single-user instances
relaunching a singleuser instance invalidates all cookies used with that instance
- test that .will_resume preserves tokens (worked, but wasn't tested)
If a Spawner reuses a token, validate it in the db:
- verify that it's in the db
- if it doesn't map onto the right user, revoke the token
- if it's not in the db, insert it as a user-provided token
The most likely case is prior unclean shutdown of something like DockerSpawner,
where a spawn failed and thus the token was revoked,
but the container was in fact created.
If you are reporting an issue with JupyterHub, please use the GitHub search feature to check if your issue has been asked already. If it has, please add your comments to the existing issue.
Some tips:
-Running `jupyter troubleshoot` from the command line, if possible, and posting
its output would also be helpful.
-Running JupyterHub in `--debug` mode (`jupyterhub --debug`) can also be helpful for troubleshooting.
--->
**Describe the bug**
A clear and concise description of what the bug is.
<!---Add description here--->
**To Reproduce**
<!---
Please share the steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error
--->
**Expected behavior**
<!---
A clear and concise description of what you expected to happen.
Welcome! As a [Jupyter](https://jupyter.org) project, we follow the [Jupyter contributor guide](https://jupyter.readthedocs.io/en/latest/contributor/content-contributor.html).
Welcome! As a [Jupyter](https://jupyter.org) project,
you can follow the [Jupyter contributor guide](https://jupyter.readthedocs.io/en/latest/contributor/content-contributor.html).
Make sure to also follow [Project Jupyter's Code of Conduct](https://github.com/jupyter/governance/blob/master/conduct/code_of_conduct.md)
for a friendly and welcoming collaborative environment.
## Setting up a development environment
JupyterHub requires Python >= 3.5 and nodejs.
As a Python project, a development install of JupyterHub follows standard practices for the basics (steps 1-2).
# see me at: http://petstore.swagger.io/?url=https://raw.githubusercontent.com/jupyter/jupyterhub/master/docs/rest-api.yml#/default
# see me at: http://petstore.swagger.io/?url=https://raw.githubusercontent.com/jupyterhub/jupyterhub/master/docs/rest-api.yml#/default
swagger:'2.0'
info:
title:JupyterHub
description:The REST API for JupyterHub
version:0.8.0dev
version:0.9.0dev
license:
name:BSD-3-Clause
schemes:
- [http, https]
[http, https]
securityDefinitions:
token:
type:apiKey
@@ -89,7 +89,7 @@ paths:
post:
summary:Create multiple users
parameters:
- name:data
- name:body
in:body
required:true
schema:
@@ -147,7 +147,7 @@ paths:
in:path
required:true
type:string
- name:data
- name:body
in:body
required:true
description:Updated user info. At least one key to be updated (name or admin) is required.
@@ -176,6 +176,63 @@ paths:
responses:
'204':
description:The user has been deleted
/users/{name}/activity:
post:
summary:
Notify Hub of activity for a given user.
description:
Notify the Hub of activity by the user,
e.g. accessing a service or (more likely)
actively using a server.
parameters:
- name:name
description:username
in:path
required:true
type:string
- name:body
in:body
schema:
type:object
properties:
last_activity:
type:string
format:date-time
description:|
Timestamp of last-seen activity for this user.
Only needed if this is not activity associated
with using a given server.
servers:
description:|
Register activity for specific servers by name.
The keys of this dict are the names of servers.
The default server has an empty name ('').
type:object
properties:
'<server name>':
description:|
Activity for a single server.
type:object
required:
- last_activity
properties:
last_activity:
type:string
format:date-time
description:|
Timestamp of last-seen activity on this server.
example:
last_activity:'2019-02-06T12:54:14Z'
servers:
'':
last_activity:'2019-02-06T12:54:14Z'
gpu:
last_activity:'2019-02-06T12:54:14Z'
responses:
'401':
$ref:'#/responses/Unauthorized'
'404':
description:Nosuch user
/users/{name}/server:
post:
summary:Start a user's single-user notebook server
@@ -185,6 +242,16 @@ paths:
in:path
required:true
type:string
- name:options
description:|
Spawn options can be passed as a JSON body
when spawning via the API instead of spawn form.
The structure of the options
will depend on the Spawner's configuration.
in:body
required:false
schema:
type:object
responses:
'201':
description:The user's notebook server has started
@@ -203,25 +270,134 @@ paths:
description:The user's notebook server has stopped
'202':
description:The user's notebook server has not yet stopped as it is taking a while to stop
/users/{name}/admin-access:
/users/{name}/servers/{server_name}:
post:
summary:Grant admin access to this user's notebook server
summary:Start a user's single-user named-server notebook server
parameters:
- name:name
description:username
in:path
required:true
type:string
- name:server_name
description:name given to a named-server
in:path
required:true
type:string
- name:options
description:|
Spawn options can be passed as a JSON body
when spawning via the API instead of spawn form.
The structure of the options
will depend on the Spawner's configuration.
in:body
required:false
schema:
type:object
responses:
'201':
description:The user's notebook named-server has started
'202':
description:The user's notebook named-server has not yet started, but has been requested
delete:
summary:Stop a user's named-server
parameters:
- name:name
description:username
in:path
required:true
type:string
- name:server_name
description:name given to a named-server
in:path
required:true
type:string
- name:remove
description:|
Whether to fully remove the server, rather than just stop it.
Removing a server deletes things like the state of the stopped server.
in:body
required:false
schema:
type:boolean
responses:
'204':
description:The user's notebook named-server has stopped
'202':
description:The user's notebook named-server has not yet stopped as it is taking a while to stop
/users/{name}/tokens:
parameters:
- name:name
description:username
in:path
required:true
type:string
get:
summary:List tokens for the user
responses:
'200':
description:Sets a cookie granting the requesting administrator access to the user's notebook server
/user:
summary:Return authenticated user's model
description:
description:The list of tokens
schema:
type:array
items:
$ref:'#/definitions/Token'
'401':
$ref:'#/responses/Unauthorized'
'404':
description:Nosuch user
post:
summary:Create a new token for the user
parameters:
- name:token_params
in:body
required:false
schema:
type:object
properties:
expires_in:
type:number
description:lifetime (in seconds) after which the requested token will expire.
note:
type:string
description:A note attached to the token for future bookkeeping
responses:
'201':
description:The newly created token
schema:
$ref:'#/definitions/Token'
'400':
description:Body must be a JSON dict or empty
/users/{name}/tokens/{token_id}:
parameters:
responses:
'200':
description:The authenticated user's model is returned.
- name:name
description:username
in:path
required:true
type:string
- name:token_id
in:path
required:true
type:string
get:
summary:Get the model for a token by id
responses:
'200':
description:The info for the new token
schema:
$ref:'#/definitions/Token'
delete:
summary:Delete (revoke) a token by id
responses:
'204':
description:The token has been deleted
/user:
get:
summary:Return authenticated user's model
responses:
'200':
description:The authenticated user's model is returned.
schema:
$ref:'#/definitions/User'
/groups:
get:
summary:List groups
@@ -279,7 +455,7 @@ paths:
in:path
required:true
type:string
- name:data
- name:body
in:body
required:true
description:The users to add to the group
@@ -304,7 +480,7 @@ paths:
in:path
required:true
type:string
- name:data
- name:body
in:body
required:true
description:The users to remove from the group
@@ -362,7 +538,7 @@ paths:
summary:Notify the Hub about a new proxy
description:Notifies the Hub of a new proxy to use.
parameters:
- name:data
- name:body
in:body
required:true
description:Any values that have changed for the new proxy. All keys are optional.
@@ -394,14 +570,15 @@ paths:
Logging in via this method is only available when the active Authenticator
accepts passwords (e.g. not OAuth).
parameters:
- name:username
- name:credentials
in:body
required:false
type:string
- name:password
in:body
required:false
type:string
schema:
type:object
properties:
username:
type:string
password:
type:string
responses:
'200':
description:The new API token
@@ -417,10 +594,10 @@ paths:
get:
summary:Identify a user or service from an API token
parameters:
- name:token
in:path
required:true
type:string
- name:token
in:path
required:true
type:string
responses:
'200':
description:The user or service identified by the API token
@@ -431,14 +608,14 @@ paths:
summary:Identify a user from a cookie
description:Used by single-user notebook servers to hand off cookie authentication to the Hub
parameters:
- name:cookie_name
in:path
required:true
type:string
- name:cookie_value
in:path
required:true
type:string
- name:cookie_name
in:path
required:true
type:string
- name:cookie_value
in:path
required:true
type:string
responses:
'200':
description:The user identified by the cookie
@@ -473,6 +650,11 @@ paths:
in:query
required:true
type:string
responses:
'200':
description:Success
'400':
description:OAuth2Error
/oauth2/token:
post:
summary:Request an OAuth2 token
@@ -484,27 +666,27 @@ paths:
parameters:
- name:client_id
description:The client id
in:form
in:formData
required:true
type:string
- name:client_secret
description:The client secret
in:form
in:formData
required:true
type:string
- name:grant_type
description:The grant type (always 'authorization_code')
in:form
in:formData
required:true
type:string
- name:code
description:The code provided by the authorization redirect
in:form
in:formData
required:true
type:string
- name:redirect_uri
description:The redirect url
in:form
in:formData
required:true
type:string
responses:
@@ -523,14 +705,28 @@ paths:
post:
summary:Shutdown the Hub
parameters:
- name:proxy
- name:body
in:body
type:boolean
description:Whether the proxy should be shutdown as well (default from Hub config)
- name:servers
in:body
type:boolean
description:Whether users' notebook servers should be shutdown as well (default from Hub config)
schema:
type:object
properties:
proxy:
type:boolean
description:Whether the proxy should be shutdown as well (default from Hub config)
servers:
type:boolean
description:Whether users' notebook servers should be shutdown as well (default from Hub config)
responses:
'202':
description:Shutdown successful
'400':
description:Unexpeced value for proxy or servers
# Descriptions of common responses
responses:
NotFound:
description:The specified resource was not found
Unauthorized:
description:Authentication/Authorization error
definitions:
User:
type:object
@@ -551,12 +747,54 @@ definitions:
description:The user's notebook server's base URL, if running; null if not.
pending:
type:string
enum:["spawn","stop"]
enum:["spawn","stop",null]
description:The currently pending action, if any
last_activity:
type:string
format:date-time
description:Timestamp of last-seen activity from the user
servers:
type:array
description:The active servers for this user.
items:
$ref:'#/definitions/Server'
Server:
type:object
properties:
name:
type:string
description:The server's name. The user's default server has an empty name ('')
ready:
type:boolean
description:|
Whether the server is ready for traffic.
Will always be false when any transition is pending.
pending:
type:string
enum:["spawn","stop",null]
description:|
The currently pending action, if any.
A server is not ready if an action is pending.
url:
type:string
description:|
The URL where the server can be accessed
(typically /user/:name/:server.name/).
progress_url:
type:string
description:|
The URL for an event-stream to retrieve events during a spawn.
started:
type:string
format:date-time
description:UTC timestamp when the server was last started.
last_activity:
type:string
format:date-time
description:UTC timestamp last-seen activity on this server.
state:
type:object
description:Arbitrary internal state from this server's spawner. Only available on the hub's users list or get-user-by-name method, and only if a hub admin. None otherwise.
Group:
type:object
properties:
@@ -591,3 +829,40 @@ definitions:
description:The command used to start the service (if managed)
items:
type:string
info:
type:object
description:|
Additional information a deployment can attach to a service.
JupyterHub does not use this field.
Token:
type:object
properties:
token:
type:string
description:The token itself. Only present in responses to requests for a new token.
id:
type:string
description:The id of the API token. Used for modifying or deleting the token.
user:
type:string
description:The user that owns a token (undefined if owned by a service)
service:
type:string
description:The service that owns the token (undefined of owned by a user)
note:
type:string
description:A note about the token, typically describing what it was created for.
created:
type:string
format:date-time
description:Timestamp when this token was created
expires_at:
type:string
format:date-time
description:Timestamp when this token expires. Null if there is no expiry.
@@ -5,14 +5,577 @@ its link will bring up a GitHub listing of changes. Use `git log` on the
command line for details.
## [Unreleased] 0.8
## [Unreleased]
## 1.1
### [1.1.0] 2020-01-17
1.1 is a release with lots of accumulated fixes and improvements,
especially in performance, metrics, and customization.
There are no database changes in 1.1, so no database upgrade is required
when upgrading from 1.0 to 1.1.
Of particular interest to deployments with automatic health checking and/or large numbers of users is that the slow startup time
introduced in 1.0 by additional spawner validation can now be mitigated by `JupyterHub.init_spawners_timeout`,
allowing the Hub to become responsive before the spawners may have finished validating.
Several new Prometheus metrics are added (and others fixed!)
to measure sources of common performance issues,
such as proxy interactions and startup.
1.1 also begins adoption of the Jupyter telemetry project in JupyterHub,
See [The Jupyter Telemetry docs](https://jupyter-telemetry.readthedocs.io)
for more info. The only events so far are starting and stopping servers,
but more will be added in future releases.
There are many more fixes and improvements listed below.
Thanks to everyone who has contributed to this release!
#### New
- LocalProcessSpawner should work on windows by using psutil.pid_exists [#2882](https://github.com/jupyterhub/jupyterhub/pull/2882) ([@ociule](https://github.com/ociule))
- trigger auth_state_hook prior to options form, add auth_state to template namespace [#2881](https://github.com/jupyterhub/jupyterhub/pull/2881) ([@minrk](https://github.com/minrk))
- Added guide 'install jupyterlab the hard way' #2110 [#2842](https://github.com/jupyterhub/jupyterhub/pull/2842) ([@mangecoeur](https://github.com/mangecoeur))
- Add prometheus metric to measure hub startup time [#2799](https://github.com/jupyterhub/jupyterhub/pull/2799) ([@rajat404](https://github.com/rajat404))
-`JupyterHub.user_redirect_hook` is added to allow admins to customize /user-redirect/ behavior [#2790](https://github.com/jupyterhub/jupyterhub/pull/2790) ([@yuvipanda](https://github.com/yuvipanda))
- Add prometheus metric to measure hub startup time [#2799](https://github.com/jupyterhub/jupyterhub/pull/2799) ([@rajat404](https://github.com/rajat404))
- Add prometheus metric to measure proxy route poll times [#2798](https://github.com/jupyterhub/jupyterhub/pull/2798) ([@rajat404](https://github.com/rajat404))
-`PROXY_DELETE_DURATION_SECONDS` prometheus metric is added, to measure proxy route deletion times [#2788](https://github.com/jupyterhub/jupyterhub/pull/2788) ([@rajat404](https://github.com/rajat404))
-`Service.oauth_no_confirm` is added, it is useful for admin-managed services that are considered part of the Hub and shouldn't need to prompt the user for access [#2767](https://github.com/jupyterhub/jupyterhub/pull/2767) ([@minrk](https://github.com/minrk))
-`JupyterHub.default_server_name` is added to make the default server be a named server with provided name [#2735](https://github.com/jupyterhub/jupyterhub/pull/2735) ([@krinsman](https://github.com/krinsman))
-`JupyterHub.init_spawners_timeout` is introduced to combat slow startups on large JupyterHub deployments [#2721](https://github.com/jupyterhub/jupyterhub/pull/2721) ([@minrk](https://github.com/minrk))
- The configuration `uids` for local authenticators is added to consistently assign users UNIX id's between installations [#2687](https://github.com/jupyterhub/jupyterhub/pull/2687) ([@rgerkin](https://github.com/rgerkin))
-`JupyterHub.activity_resolution` is introduced with a default value of 30s improving performance by not updating the database with user activity too often [#2605](https://github.com/jupyterhub/jupyterhub/pull/2605) ([@minrk](https://github.com/minrk))
- [HubAuth](https://jupyterhub.readthedocs.io/en/stable/api/services.auth.html#jupyterhub.services.auth.HubAuth)'s SSL configuration can now be set through environment variables [#2588](https://github.com/jupyterhub/jupyterhub/pull/2588) ([@cmd-ntrf](https://github.com/cmd-ntrf))
- Expose spawner.user_options in REST API. [#2755](https://github.com/jupyterhub/jupyterhub/pull/2755) ([@danielballan](https://github.com/danielballan))
- add block for scripts included in head [#2828](https://github.com/jupyterhub/jupyterhub/pull/2828) ([@bitnik](https://github.com/bitnik))
- Instrument JupyterHub to record events with jupyter_telemetry [Part II] [#2698](https://github.com/jupyterhub/jupyterhub/pull/2698) ([@Zsailer](https://github.com/Zsailer))
- Make announcements visible without custom HTML [#2570](https://github.com/jupyterhub/jupyterhub/pull/2570) ([@consideRatio](https://github.com/consideRatio))
- Display server version on admin page [#2776](https://github.com/jupyterhub/jupyterhub/pull/2776) ([@vilhelmen](https://github.com/vilhelmen))
- Cleanup if spawner stop fails [#2849](https://github.com/jupyterhub/jupyterhub/pull/2849) ([@gabber12](https://github.com/gabber12))
- Fix an issue occurring with the default spawner and `internal_ssl` enabled [#2785](https://github.com/jupyterhub/jupyterhub/pull/2785) ([@rpwagner](https://github.com/rpwagner))
- Fix named servers to not be spawnable unless activated [#2772](https://github.com/jupyterhub/jupyterhub/pull/2772) ([@bitnik](https://github.com/bitnik))
- JupyterHub now awaits proxy availability before accepting web requests [#2750](https://github.com/jupyterhub/jupyterhub/pull/2750) ([@minrk](https://github.com/minrk))
- Fix a no longer valid assumption that MySQL and MariaDB need to have `innodb_file_format` and `innodb_large_prefix` configured [#2712](https://github.com/jupyterhub/jupyterhub/pull/2712) ([@chicocvenancio](https://github.com/chicocvenancio))
- Login/Logout button now updates to Login on logout [#2705](https://github.com/jupyterhub/jupyterhub/pull/2705) ([@aar0nTw](https://github.com/aar0nTw))
- Fix handling of exceptions within `pre_spawn_start` hooks [#2684](https://github.com/jupyterhub/jupyterhub/pull/2684) ([@GeorgianaElena](https://github.com/GeorgianaElena))
- Fix an issue where a user could end up spawning a default server instead of a named server as intended [#2682](https://github.com/jupyterhub/jupyterhub/pull/2682) ([@rcthomas](https://github.com/rcthomas))
- /hub/admin now redirects to login if unauthenticated [#2670](https://github.com/jupyterhub/jupyterhub/pull/2670) ([@GeorgianaElena](https://github.com/GeorgianaElena))
- Fix spawning of users with names containing characters that needs to be escaped [#2648](https://github.com/jupyterhub/jupyterhub/pull/2648) ([@nicorikken](https://github.com/nicorikken))
- Fix faulty redirects to 404 that could occur with the use of named servers [#2594](https://github.com/jupyterhub/jupyterhub/pull/2594) ([@vilhelmen](https://github.com/vilhelmen))
- JupyterHub API spec is now a valid OpenAPI spec [#2590](https://github.com/jupyterhub/jupyterhub/pull/2590) ([@sbrunk](https://github.com/sbrunk))
- Use of `--help` or `--version` previously could output unrelated errors [#2584](https://github.com/jupyterhub/jupyterhub/pull/2584) ([@minrk](https://github.com/minrk))
- No longer crash on startup in Windows [#2560](https://github.com/jupyterhub/jupyterhub/pull/2560) ([@adelcast](https://github.com/adelcast))
- Escape usernames in the frontend [#2640](https://github.com/jupyterhub/jupyterhub/pull/2640) ([@nicorikken](https://github.com/nicorikken))
#### Maintenance
- Optimize CI jobs and default to bionic [#2897](https://github.com/jupyterhub/jupyterhub/pull/2897) ([@consideRatio](https://github.com/consideRatio))
- catch connection error for ssl failures [#2889](https://github.com/jupyterhub/jupyterhub/pull/2889) ([@minrk](https://github.com/minrk))
- Fix implementation of default server name [#2887](https://github.com/jupyterhub/jupyterhub/pull/2887) ([@krinsman](https://github.com/krinsman))
- Log JupyterHub version on startup [#2752](https://github.com/jupyterhub/jupyterhub/pull/2752) ([@consideRatio](https://github.com/consideRatio))
- Reduce verbosity for "Failing suspected API request to not-running server" (new) [#2751](https://github.com/jupyterhub/jupyterhub/pull/2751) ([@rkdarst](https://github.com/rkdarst))
- Use autodoc-traits sphinx extension [#2723](https://github.com/jupyterhub/jupyterhub/pull/2723) ([@willingc](https://github.com/willingc))
- Add New Server: change redirecting to relative to home page in js [#2714](https://github.com/jupyterhub/jupyterhub/pull/2714) ([@bitnik](https://github.com/bitnik))
- Create a warning when creating a service implicitly from service_tokens [#2704](https://github.com/jupyterhub/jupyterhub/pull/2704) ([@katsar0v](https://github.com/katsar0v))
- Documentation update: hint for using services instead of service tokens. [#2679](https://github.com/jupyterhub/jupyterhub/pull/2679) ([@katsar0v](https://github.com/katsar0v))
- Update spawn-form example [#2662](https://github.com/jupyterhub/jupyterhub/pull/2662) ([@kinow](https://github.com/kinow))
- Update flask hub authentication services example in doc [#2658](https://github.com/jupyterhub/jupyterhub/pull/2658) ([@cmd-ntrf](https://github.com/cmd-ntrf))
- close `<div class="container">` tag in home.html [#2649](https://github.com/jupyterhub/jupyterhub/pull/2649) ([@bitnik](https://github.com/bitnik))
- Some theme updates; no double NEXT/PREV buttons. [#2647](https://github.com/jupyterhub/jupyterhub/pull/2647) ([@Carreau](https://github.com/Carreau))
- fix typos on technical reference documentation [#2646](https://github.com/jupyterhub/jupyterhub/pull/2646) ([@ilee38](https://github.com/ilee38))
- Update links for Hadoop-related subprojects [#2645](https://github.com/jupyterhub/jupyterhub/pull/2645) ([@jcrist](https://github.com/jcrist))
- Fixed docs and testing code to use refactored SimpleLocalProcessSpawner [#2631](https://github.com/jupyterhub/jupyterhub/pull/2631) ([@danlester](https://github.com/danlester))
- Update the config used for testing [#2628](https://github.com/jupyterhub/jupyterhub/pull/2628) ([@jtpio](https://github.com/jtpio))
- Update doc: do not suggest depricated config key [#2626](https://github.com/jupyterhub/jupyterhub/pull/2626) ([@lumbric](https://github.com/lumbric))
- Add missing words [#2625](https://github.com/jupyterhub/jupyterhub/pull/2625) ([@remram44](https://github.com/remram44))
- cull-idle: Include a hint on how to add custom culling logic [#2613](https://github.com/jupyterhub/jupyterhub/pull/2613) ([@rkdarst](https://github.com/rkdarst))
- Hide Stop My Server red button after server stopped. [#2577](https://github.com/jupyterhub/jupyterhub/pull/2577) ([@aar0nTw](https://github.com/aar0nTw))
- Update link of `changelog` [#2565](https://github.com/jupyterhub/jupyterhub/pull/2565) ([@iblis17](https://github.com/iblis17))
- Update to simplify the language related to spawner options [#2558](https://github.com/jupyterhub/jupyterhub/pull/2558) ([@NikeNano](https://github.com/NikeNano))
- Adding the use case of the Elucidata: How Jupyter Notebook is used in… [#2548](https://github.com/jupyterhub/jupyterhub/pull/2548) ([@IamViditAgarwal](https://github.com/IamViditAgarwal))
- Dict rewritten as literal [#2546](https://github.com/jupyterhub/jupyterhub/pull/2546) ([@remyleone](https://github.com/remyleone))
## 1.0
### [1.0.0] 2019-05-03
JupyterHub 1.0 is a major milestone for JupyterHub.
Huge thanks to the many people who have contributed to this release,
whether it was through discussion, testing, documentation, or development.
#### Major new features
- Support TLS encryption and authentication of all internal communication.
Spawners must implement `.move_certs` method to make certificates available
to the notebook server if it is not local to the Hub.
- There is now full UI support for managing named servers.
With named servers, each jupyterhub user may have access to more than one named server. For example, a professor may access a server named `research` and another named `teaching`.
- Authenticators can now expire and refresh authentication data by implementing
`Authenticator.refresh_user(user)`.
This allows things like OAuth data and access tokens to be refreshed.
When used together with `Authenticator.refresh_pre_spawn = True`,
auth refresh can be forced prior to Spawn,
allowing the Authenticator to *require* that authentication data is fresh
immediately before the user's server is launched.
```eval_rst
.. seealso::
- :meth:`.Authenticator.refresh_user`
- :meth:`.Spawner.create_certs`
- :meth:`.Spawner.move_certs`
```
#### New features
- allow custom spawners, authenticators, and proxies to register themselves via 'entry points', enabling more convenient configuration such as:
```python
c.JupyterHub.authenticator_class = 'github'
c.JupyterHub.spawner_class = 'docker'
c.JupyterHub.proxy_class = 'traefik_etcd'
```
- Spawners are passed the tornado Handler object that requested their spawn (as `self.handler`),
so they can do things like make decisions based on query arguments in the request.
- SimpleSpawner and DummyAuthenticator, which are useful for testing, have been merged into JupyterHub itself:
```python
# For testing purposes only. Should not be used in production.
c.JupyterHub.authenticator_class = 'dummy'
c.JupyterHub.spawner_class = 'simple'
```
These classes are **not** appropriate for production use. Only testing.
- Add health check endpoint at `/hub/health`
- Several prometheus metrics have been added (thanks to [Outreachy](https://www.outreachy.org/) applicants!)
- A new API for registering user activity.
To prepare for the addition of [alternate proxy implementations](https://github.com/jupyterhub/traefik-proxy),
responsibility for tracking activity is taken away from the proxy
and moved to the notebook server (which already has activity tracking features).
Activity is now tracked by pushing it to the Hub from user servers instead of polling the
proxy API.
- Dynamic `options_form` callables may now return an empty string
which will result in no options form being rendered.
- `Spawner.user_options` is persisted to the database to be re-used,
so that a server spawned once via the form can be re-spawned via the API
with the same options.
- Added `c.PAMAuthenticator.pam_normalize_username` option for round-tripping
usernames through PAM to retrieve the normalized form.
- Added `c.JupyterHub.named_server_limit_per_user` configuration to limit
the number of named servers each user can have.
The default is 0, for no limit.
- API requests to HubAuthenticated services (e.g. single-user servers)
`authentication` should have a default value of None
for backward-compatibility with jupyterhub < 1.0.
- Prometheus metrics page is now authenticated.
Any authenticated user may see the prometheus metrics.
To disable prometheus authentication,
set `JupyterHub.authenticate_prometheus = False`.
- Visits to `/user/:name` no longer trigger an implicit launch of the user's server.
Instead, a page is shown indicating that the server is not running
with a link to request the spawn.
- API requests to `/user/:name` for a not-running server will have status 503 instead of 404.
- OAuth includes a confirmation page when attempting to visit another user's server,
so that users can choose to cancel authentication with the single-user server.
Confirmation is still skipped when accessing your own server.
#### Fixed
- Various fixes to improve Windows compatibility
(default Authenticator and Spawner still do not support Windows, but other Spawners may)
- Fixed compatibility with Oracle db
- Fewer redirects following a visit to the default `/` url
- Error when progress is requested before progress is ready
- Error when API requests are made to a not-running server without authentication
- Avoid logging database password on connect if password is specified in `JupyterHub.db_url`.
#### Development changes
There have been several changes to the development process that shouldn't
generally affect users of JupyterHub, but may affect contributors.
In general, see `CONTRIBUTING.md` for contribution info or ask if you have questions.
- JupyterHub has adopted `black` as a code autoformatter and `pre-commit`
as a tool for automatically running code formatting on commit.
This is meant to make it *easier* to contribute to JupyterHub,
so let us know if it's having the opposite effect.
- JupyterHub has switched its test suite to using `pytest-asyncio` from `pytest-tornado`.
- OAuth is now implemented internally using `oauthlib` instead of `python-oauth2`. This should have no effect on behavior.
## 0.9
### [0.9.6] 2019-04-01
JupyterHub 0.9.6 is a security release.
- Fixes an Open Redirect vulnerability (CVE-2019-10255).
JupyterHub 0.9.5 included a partial fix for this issue.
### [0.9.4] 2018-09-24
JupyterHub 0.9.4 is a small bugfix release.
- Fixes an issue that required all running user servers to be restarted
when performing an upgrade from 0.8 to 0.9.
- Fixes content-type for API endpoints back to `application/json`.
It was `text/html` in 0.9.0-0.9.3.
### [0.9.3] 2018-09-12
JupyterHub 0.9.3 contains small bugfixes and improvements
- Fix token page and model handling of `expires_at`.
This field was missing from the REST API model for tokens
and could cause the token page to not render
- Add keep-alive to progress event stream to avoid proxies dropping
the connection due to inactivity
- Documentation and example improvements
- Disable quit button when using notebook 5.6
- Prototype new feature (may change prior to 1.0):
pass requesting Handler to Spawners during start,
accessible as `self.handler`
### [0.9.2] 2018-08-10
JupyterHub 0.9.2 contains small bugfixes and improvements.
- Documentation and example improvements
- Add `Spawner.consecutive_failure_limit` config for aborting the Hub if too many spawns fail in a row.
- Fix for handling SIGTERM when run with asyncio (tornado 5)
- Windows compatibility fixes
### [0.9.1] 2018-07-04
JupyterHub 0.9.1 contains a number of small bugfixes on top of 0.9.
- Use a PID file for the proxy to decrease the likelihood that a leftover proxy process will prevent JupyterHub from restarting
- `c.LocalProcessSpawner.shell_cmd` is now configurable
- API requests to stopped servers (requests to the hub for `/user/:name/api/...`) fail with 404 rather than triggering a restart of the server
- Compatibility fix for notebook 5.6.0 which will introduce further
security checks for local connections
- Managed services always use localhost to talk to the Hub if the Hub listening on all interfaces
- When using a URL prefix, the Hub route will be `JupyterHub.base_url` instead of unconditionally `/`
- additional fixes and improvements
### [0.9.0] 2018-06-15
JupyterHub 0.9 is a major upgrade of JupyterHub.
There are several changes to the database schema,
so make sure to backup your database and run:
jupyterhub upgrade-db
after upgrading jupyterhub.
The biggest change for 0.9 is the switch to asyncio coroutines everywhere
instead of tornado coroutines. Custom Spawners and Authenticators are still
free to use tornado coroutines for async methods, as they will continue to
work. As part of this upgrade, JupyterHub 0.9 drops support for Python < 3.5
and tornado < 5.0.
#### Added
#### Changed
- Require Python >= 3.5
- Require tornado >= 5.0
- Use asyncio coroutines throughout
- Set status 409 for conflicting actions instead of 400,
e.g. creating users or groups that already exist.
- timestamps in REST API continue to be UTC, but now include 'Z' suffix
to identify them as such.
- REST API User model always includes `servers` dict,
not just when named servers are enabled.
- `server` info is no longer available to oauth identification endpoints,
only user info and group membership.
- `User.last_activity` may be None if a user has not been seen,
rather than starting with the user creation time
which is now separately stored as `User.created`.
- static resources are now found in `$PREFIX/share/jupyterhub` instead of `share/jupyter/hub` for improved consistency.
- Deprecate `.extra_log_file` config. Use pipe redirection instead:
jupyterhub &>> /var/log/jupyterhub.log
- Add `JupyterHub.bind_url` config for setting the full bind URL of the proxy.
Sets ip, port, base_url all at once.
- Add `JupyterHub.hub_bind_url` for setting the full host+port of the Hub.
`hub_bind_url` supports unix domain sockets, e.g.
`unix+http://%2Fsrv%2Fjupyterhub.sock`
- Deprecate `JupyterHub.hub_connect_port` config in favor of `JupyterHub.hub_connect_url`. `hub_connect_ip` is not deprecated
and can still be used in the common case where only the ip address of the hub differs from the bind ip.
#### Added
- Spawners can define a `.progress` method which should be an async generator.
The generator should yield events of the form:
```python
{
"message": "some-state-message",
"progress": 50,
}
```
These messages will be shown with a progress bar on the spawn-pending page.
The `async_generator` package can be used to make async generators
compatible with Python 3.5.
- track activity of individual API tokens
- new REST API for managing API tokens at `/hub/api/user/tokens[/token-id]`
- allow viewing/revoking tokens via token page
- User creation time is available in the REST API as `User.created`
- Server start time is stored as `Server.started`
- `Spawner.start` may return a URL for connecting to a notebook instead of `(ip, port)`. This enables Spawners to launch servers that setup their own HTTPS.
- Optimize database performance by disabling sqlalchemy expire_on_commit by default.
- Add `python -m jupyterhub.dbutil shell` entrypoint for quickly
launching an IPython session connected to your JupyterHub database.
- Include `User.auth_state` in user model on single-user REST endpoints for admins only.
- Include `Server.state` in server model on REST endpoints for admins only.
- Add `Authenticator.blacklist` for blacklisting users instead of whitelisting.
- Pass `c.JupyterHub.tornado_settings['cookie_options']` down to Spawners
so that cookie options (e.g. `expires_days`) can be set globally for the whole application.
- SIGINFO (`ctrl-t`) handler showing the current status of all running threads,
coroutines, and CPU/memory/FD consumption.
- Add async `Spawner.get_options_form` alternative to `.options_form`, so it can be a coroutine.
- Add `JupyterHub.redirect_to_server` config to govern whether
users should be sent to their server on login or the JupyterHub home page.
- html page templates can be more easily customized and extended.
- Allow registering external OAuth clients for using the Hub as an OAuth provider.
- Add basic prometheus metrics at `/hub/metrics` endpoint.
- Add session-id cookie, enabling immediate revocation of login tokens.
- Authenticators may specify that users are admins by specifying the `admin` key when return the user model as a dict.
- Added "Start All" button to admin page for launching all user servers at once.
- Services have an `info` field which is a dictionary.
This is accessible via the REST API.
- `JupyterHub.extra_handlers` allows defining additional tornado RequestHandlers attached to the Hub.
- API tokens may now expire.
Expiry is available in the REST model as `expires_at`,
and settable when creating API tokens by specifying `expires_in`.
#### Fixed
- Remove green from theme to improve accessibility
- Fix error when proxy deletion fails due to route already being deleted
- clear `?redirects` from URL on successful launch
- disable send2trash by default, which is rarely desirable for jupyterhub
- Put PAM calls in a thread so they don't block the main application
in cases where PAM is slow (e.g. LDAP).
- Remove implicit spawn from login handler,
instead relying on subsequent request for `/user/:name` to trigger spawn.
- Fixed several inconsistencies for initial redirects,
depending on whether server is running or not and whether the user is logged in or not.
- Admin requests for `/user/:name` (when admin-access is enabled) launch the right server if it's not running instead of redirecting to their own.
- Major performance improvement starting up JupyterHub with many users,
especially when most are inactive.
- Various fixes in race conditions and performance improvements with the default proxy.
- Fixes for CORS headers
- Stop setting `.form-control` on spawner form inputs unconditionally.
- Better recovery from database errors and database connection issues
without having to restart the Hub.
- Fix handling of `~` character in usernames.
- Fix jupyterhub startup when `getpass.getuser()` would fail,
e.g. due to missing entry in passwd file in containers.
## 0.8
### [0.8.1] 2017-11-07
JupyterHub 0.8.1 is a collection of bugfixes and small improvements on 0.8.
#### Added
- Run tornado with AsyncIO by default
- Add `jupyterhub --upgrade-db` flag for automatically upgrading the database as part of startup.
This is useful for cases where manually running `jupyterhub upgrade-db`
as a separate step is unwieldy.
- Avoid creating backups of the database when no changes are to be made by
`jupyterhub upgrade-db`.
#### Fixed
- Add some further validation to usernames - `/` is not allowed in usernames.
- Fix empty logout page when using auto_login
- Fix autofill of username field in default login form.
- Fix listing of users on the admin page who have not yet started their server.
- Fix ever-growing traceback when re-raising Exceptions from spawn failures.
- Remove use of deprecated `bower` for javascript client dependencies.
### [0.8.0] 2017-10-03
JupyterHub 0.8 is a big release!
Perhaps the biggest change is the use of OAuth to negotiate authentication
between the Hub and single-user services.
Due to this change, it is important that the single-user server
and Hub are both running the same version of JupyterHub.
If you are using containers (e.g. via DockerSpawner or KubeSpawner),
this means upgrading jupyterhub in your user images at the same time as the Hub.
In most cases, a
pip install jupyterhub==version
in your Dockerfile is sufficient.
#### Added
- JupyterHub now defined a `Proxy` API for custom
proxy implementations other than the default.
The defaults are unchanged,
but configuration of the proxy is now done on the `ConfigurableHTTPProxy` class instead of the top-level JupyterHub.
TODO: docs for writing a custom proxy.
- Single-user servers and services
(anything that uses HubAuth)
can now accept token-authenticated requests via the Authentication header.
- Authenticators can now store state in the Hub's database.
To do so, the `authenticate` method should return a dict of the form
```python
{
'username': 'name',
'state': {}
}
```
This data will be encrypted and requires `JUPYTERHUB_CRYPT_KEY` environment variable to be set
and the `Authenticator.enable_auth_state` flag to be True.
If these are not set, auth_state returned by the Authenticator will not be stored.
- There is preliminary support for multiple (named) servers per user in the REST API.
Named servers can be created via API requests, but there is currently no UI for managing them.
- Add `LocalProcessSpawner.popen_kwargs` and `LocalProcessSpawner.shell_cmd`
for customizing how user server processes are launched.
- Add `Authenticator.auto_login` flag for skipping the "Login with..." page explicitly.
- Add `JupyterHub.hub_connect_ip` configuration
for the ip that should be used when connecting to the Hub.
This is promoting (and deprecating) `DockerSpawner.hub_ip_connect`
for use by all Spawners.
- Add `Spawner.pre_spawn_hook(spawner)` hook for customizing
pre-spawn events.
- Add `JupyterHub.active_server_limit` and `JupyterHub.concurrent_spawn_limit`
for limiting the total number of running user servers and the number of pending spawns, respectively.
#### Changed
- more arguments to spawners are now passed via environment variables (`.get_env()`)
rather than CLI arguments (`.get_args()`)
- internally generated tokens no longer get extra hash rounds,
significantly speeding up authentication.
The hash rounds were deemed unnecessary because the tokens were already
generated with high entropy.
- `JUPYTERHUB_API_TOKEN` env is available at all times,
rather than being removed during single-user start.
The token is now accessible to kernel processes,
enabling user kernels to make authenticated API requests to Hub-authenticated services.
- Cookie secrets should be 32B hex instead of large base64 secrets.
- pycurl is used by default, if available.
#### Fixed
So many things fixed!
- Collisions are checked when users are renamed
- Fix bug where OAuth authenticators could not logout users
due to being redirected right back through the login process.
- If there are errors loading your config files,
JupyterHub will refuse to start with an informative error.
Previously, the bad config would be ignored and JupyterHub would launch with default configuration.
- Raise 403 error on unauthorized user rather than redirect to login,
which could cause redirect loop.
- Set `httponly` on cookies because it's prudent.
- Improve support for MySQL as the database backend
- Many race conditions and performance problems under heavy load have been fixed.
- Fix alembic tagging of database schema versions.
#### Removed
- End support for Python 3.3
@@ -155,7 +718,17 @@ Fix removal of `/login` page in 0.4.0, breaking some OAuth providers.
JupyterHub can be configured to record structured events from a running server using Jupyter's `Telemetry System`_. The types of events that JupyterHub emits are defined by `JSON schemas`_ listed below_
emitted as JSON data, defined and validated by the JSON schemas listed below.
- [JupyterHub User Guide](https://www.rc.colorado.edu/support/user-guide/jupyterhub.html)
- Slurm job dispatched on Crestone compute cluster
- log troubleshooting
@@ -77,16 +77,25 @@ easy to do with RStudio too.
- Earth Lab at CU
- [Tutorial on Parallel R on JupyterHub](https://earthdatascience.org/tutorials/parallel-r-on-jupyterhub/)
### George Washington University
- [Jupyter Hub](http://go.gwu.edu/jupyter) with university single-sign-on. Deployed early 2017.
### HTCondor
- [HTCondor Python Bindings Tutorial from HTCondor Week 2017 includes information on their JupyterHub tutorials](https://research.cs.wisc.edu/htcondor/HTCondorWeek2017/presentations/TueBockelman_Python.pdf)
- [nbgraderutils](https://github.com/dice-group/nbgraderutils): Use JupyterHub + nbgrader + iJava kernel for online Java exercises. Used in lecture Statistical Natural Language Processing.
### Penn State University
- [Press release](https://news.psu.edu/story/523093/2018/05/24/new-open-source-web-apps-available-students-and-faculty): "New open-source web apps available for students and faculty" (but Hub is currently down; checked 04/26/19)
[Everware](https://github.com/everware) Reproducible and reusable science powered by jupyterhub and docker. Like nbviewer, but executable. CERN, Geneva [website](http://everware.xyz/)
2. If you need to allow for even more users, a dynamic amount of servers can be used on a cloud,
take a look at the `Zero to JupyterHub with Kubernetes <https://github.com/jupyterhub/zero-to-jupyterhub-k8s>`__ .
Four subsystems make up JupyterHub:
* a **Hub** (tornado process) that is the heart of JupyterHub
* a **configurable http proxy** (node-http-proxy) that receives the requests from the client's browser
* multiple **single-user Jupyter notebook servers** (Python/IPython/tornado) that are monitored by Spawners
* an **authentication class** that manages how users can access the system
Besides these central pieces, you can add optional configurations through a `config.py` file and manage users kernels on an admin panel. A simplification of the whole system can be seen in the figure below:
### Install a default conda environment for all users
First create a folder for conda envs (might exist already):
```sh
sudo mkdir /opt/conda/envs/
```
Then create a conda environment to your liking within that folder. Here we have called it 'python' because it will
be the obvious default - call it whatever you like. You can install whatever you like into this environment, but you MUST at least install `ipykernel`.
There is relatively little for the administrator to do here, as users will have to set up their own environments using the shell.
On login they should run `conda init` or `/opt/conda/bin/conda`. The can then use conda to set up their environment,
although they must also install `ipykernel`. Once done, they can enable their kernel using:
```sh
/path/to/kernel/env/bin/python -m ipykernel install --name 'python-my-env' --display-name "Python My Env"
```
This will place the kernel spec into their home folder, where Jupyter will look for it on startup.
## Setting up a reverse proxy
The guide so far results in JupyterHub running on port 8000. It is not generally advisable to run open web services in
this way - instead, use a reverse proxy running on standard HTTP/HTTPS ports.
> **Important**: Be aware of the security implications especially if you are running a server that is accessible from the open internet
> i.e. not protected within an institutional intranet or private home/office network. You should set up a firewall and
> HTTPS encryption, which is outside of the scope of this guide. For HTTPS consider using [LetsEncrypt](https://letsencrypt.org/)
> or setting up a [self-signed certificate](https://www.digitalocean.com/community/tutorials/how-to-create-a-self-signed-ssl-certificate-for-nginx-in-ubuntu-18-04).
> Firewalls may be set up using `ufs` or `firewalld` and combined with `fail2ban`.
### Using Nginx
Nginx is a mature and established web server and reverse proxy and is easy to install using `sudo apt install nginx`.
Details on using Nginx as a reverse proxy can be found elsewhere. Here, we will only outline the additional steps needed
to setup JupyterHub with Nginx and host it at a given URL e.g. `<your-server-ip-or-url>/jupyter`.
This could be useful for example if you are running several services or web pages on the same server.
To achieve this needs a few tweaks to both the JupyterHub configuration and the Nginx config. First, edit the
configuration file `/opt/jupyterhub/etc/jupyterhub/jupyterhub_config.py` and add the line:
```python
c.JupyterHub.bind_url='http://:8000/jupyter'
```
where `/jupyter` will be the relative URL of the JupyterHub.
Now Nginx must be configured with a to pass all traffic from `/jupyter` to the the local address `127.0.0.1:8000`.
Add the following snippet to your nginx configuration file (e.g. `/etc/nginx/sites-available/default`).
```
location /jupyter/ {
# NOTE important to also set base url of jupyterhub to /jupyter in its config
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
