hazza/docker-stacks
1
0
Fork 0
mirror of https://github.com/jupyter/docker-stacks.git synced 2025-10-17 15:02:57 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #1158 from romainx/spark_3_0_1

Browse Source 
Spark 3.0.1 -> fixes #1156
This commit is contained in:
Peter Parente
2020-09-18 15:00:44 -04:00
committed by GitHub
parent 6d42503c68 67e4600b09
commit 4b26cce185
3 changed files with 9 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
base-notebook/test/test_container_options.py
View File

@@ -11,11 +11,13 @@ LOGGER = logging.getLogger(__name__)
def test_cli_args(container, http_client):
"""Container should respect notebook server command line args
(e.g., disabling token security)"""
container.run(
command=['start-notebook.sh', '--NotebookApp.token=""']
c = container.run(
command=["start-notebook.sh", "--NotebookApp.token=''"]
)
resp = http_client.get('http://localhost:8888')
resp.raise_for_status()
logs = c.logs(stdout=True).decode('utf-8')
LOGGER.debug(logs)
assert 'login_submit' not in resp.text

6
docs/using/specifics.md
View File

@@ -12,7 +12,7 @@ This page provides details about features specific to one or more images.
You can build a `pyspark-notebook` image (and also the downstream `all-spark-notebook` image) with a different version of Spark by overriding the default value of the following arguments at build time.
* Spark distribution is defined by the combination of the Spark and the Hadoop version and verified by the package checksum, see [Download Apache Spark](https://spark.apache.org/downloads.html) for more information.
* Spark distribution is defined by the combination of the Spark and the Hadoop version and verified by the package checksum, see [Download Apache Spark](https://spark.apache.org/downloads.html) for more information. At this time the build will only work with the set of versions available on the Apache Spark download page, so it will not work with the archived versions.
* `spark_version`: The Spark version to install (`3.0.0`).
* `hadoop_version`: The Hadoop version (`3.2`).
* `spark_checksum`: The package checksum (`BFE4540...`).
@@ -59,8 +59,6 @@ docker run -it --rm jupyter/pyspark-notebook:spark-2.4.6 pyspark --version
* get the version (`ls /usr/local/spark/python/lib/`),
* set the version `--build-arg py4j_version=0.10.7`.
*Note: At the time of writing there is an issue preventing to use Spark `2.4.6` with Python `3.8`, see [this answer on SO](https://stackoverflow.com/a/62173969/4413446) for more information.*
```bash
docker run -it --rm jupyter/pyspark-notebook:spark-2.4.6 ls /usr/local/spark/python/lib/
# py4j-0.10.7-src.zip PY4J_LICENSE.txt pyspark.zip
@@ -68,6 +66,8 @@ docker run -it --rm jupyter/pyspark-notebook:spark-2.4.6 ls /usr/local/spark/pyt
# --build-arg py4j_version=
```
*Note: At the time of writing there is an issue preventing to use Spark `2.4.6` with Python `3.8`, see [this answer on SO](https://stackoverflow.com/a/62173969/4413446) for more information.*
### Usage Examples
The `jupyter/pyspark-notebook` and `jupyter/all-spark-notebook` images support the use of [Apache Spark](https://spark.apache.org/) in Python, R, and Scala notebooks. The following sections provide some examples of how to get started using them.

4
pyspark-notebook/Dockerfile
View File

@@ -13,9 +13,9 @@ USER root
# Spark dependencies
# Default values can be overridden at build time
# (ARGS are in lower case to distinguish them from ENV)
ARG spark_version="3.0.0"
ARG spark_version="3.0.1"
ARG hadoop_version="3.2"
ARG spark_checksum="BFE45406C67CC4AE00411AD18CC438F51E7D4B6F14EB61E7BF6B5450897C2E8D3AB020152657C0239F253735C263512FFABF538AC5B9FFFA38B8295736A9C387"
ARG spark_checksum="E8B47C5B658E0FBC1E57EEA06262649D8418AE2B2765E44DA53AAF50094877D17297CC5F0B9B35DF2CEEF830F19AA31D7E56EAD950BBE7F8830D6874F88CFC3C"
ARG py4j_version="0.10.9"
ARG openjdk_version="11"