diff --git a/all-spark-notebook/Dockerfile b/all-spark-notebook/Dockerfile index 684c9ddb..f2847248 100644 --- a/all-spark-notebook/Dockerfile +++ b/all-spark-notebook/Dockerfile @@ -42,11 +42,3 @@ RUN arch=$(uname -m) && \ mamba clean --all -f -y && \ fix-permissions "${CONDA_DIR}" && \ fix-permissions "/home/${NB_USER}" - -# Spylon-kernel -RUN mamba install --quiet --yes 'spylon-kernel' && \ - mamba clean --all -f -y && \ - python -m spylon_kernel install --sys-prefix && \ - rm -rf "/home/${NB_USER}/.local" && \ - fix-permissions "${CONDA_DIR}" && \ - fix-permissions "/home/${NB_USER}" diff --git a/all-spark-notebook/README.md b/all-spark-notebook/README.md index a614343d..898aca6a 100644 --- a/all-spark-notebook/README.md +++ b/all-spark-notebook/README.md @@ -1,4 +1,4 @@ -# Jupyter Notebook Python, Scala, R, Spark Stack +# Jupyter Notebook Python, R, Spark Stack [![docker pulls](https://img.shields.io/docker/pulls/jupyter/all-spark-notebook.svg)](https://hub.docker.com/r/jupyter/all-spark-notebook/) [![docker stars](https://img.shields.io/docker/stars/jupyter/all-spark-notebook.svg)](https://hub.docker.com/r/jupyter/all-spark-notebook/) diff --git a/docs/using/selecting.md b/docs/using/selecting.md index b6ce0939..b003a7c3 100644 --- a/docs/using/selecting.md +++ b/docs/using/selecting.md @@ -175,7 +175,7 @@ communities. [Dockerfile commit history](https://github.com/jupyter/docker-stacks/commits/master/all-spark-notebook/Dockerfile) | [Docker Hub image tags](https://hub.docker.com/r/jupyter/all-spark-notebook/tags/) -`jupyter/all-spark-notebook` includes Python, R, and Scala support for Apache Spark. +`jupyter/all-spark-notebook` includes Python and R support for Apache Spark. - Everything in `jupyter/pyspark-notebook` and its ancestor images - [IRKernel](https://irkernel.github.io/) to support R code in Jupyter notebooks @@ -183,7 +183,6 @@ communities. [sparklyr](https://spark.rstudio.com), [ggplot2](https://ggplot2.tidyverse.org) packages -- [spylon-kernel](https://github.com/vericast/spylon-kernel) to support Scala code in Jupyter notebooks ### Image Relationships diff --git a/docs/using/specifics.md b/docs/using/specifics.md index adabf7c2..04eef9cf 100644 --- a/docs/using/specifics.md +++ b/docs/using/specifics.md @@ -76,7 +76,7 @@ docker run -it --rm jupyter/pyspark-notebook:spark-2.4.7 pyspark --version ### Usage Examples -The `jupyter/pyspark-notebook` and `jupyter/all-spark-notebook` images support the use of [Apache Spark](https://spark.apache.org/) in Python, R, and Scala notebooks. +The `jupyter/pyspark-notebook` and `jupyter/all-spark-notebook` images support the use of [Apache Spark](https://spark.apache.org/) in Python and R notebooks. The following sections provide some examples of how to get started using them. #### Using Spark Local Mode @@ -144,24 +144,6 @@ sdf_len(sc, 100, repartition = 1) %>% # 5050 ``` -##### Local Mode in Scala - -Spylon kernel instantiates a `SparkContext` for you in variable `sc` after you configure Spark -options in a `%%init_spark` magic cell. - -```python -%%init_spark -# Configure Spark to use a local master -launcher.master = "local" -``` - -```scala -// Sum of the first 100 whole numbers -val rdd = sc.parallelize(0 to 100) -rdd.sum() -// 5050 -``` - #### Connecting to a Spark Cluster in Standalone Mode Connection to Spark Cluster on **[Standalone Mode](https://spark.apache.org/docs/latest/spark-standalone.html)** requires the following set of steps: @@ -235,24 +217,6 @@ sdf_len(sc, 100, repartition = 1) %>% # 5050 ``` -##### Standalone Mode in Scala - -Spylon kernel instantiates a `SparkContext` for you in variable `sc` after you configure Spark -options in a `%%init_spark` magic cell. - -```python -%%init_spark -# Configure Spark to use a local master -launcher.master = "spark://master:7077" -``` - -```scala -// Sum of the first 100 whole numbers -val rdd = sc.parallelize(0 to 100) -rdd.sum() -// 5050 -``` - ### Define Spark Dependencies ```{note} diff --git a/tests/all-spark-notebook/data/local_spylon.ipynb b/tests/all-spark-notebook/data/local_spylon.ipynb deleted file mode 100644 index 94f3c5b5..00000000 --- a/tests/all-spark-notebook/data/local_spylon.ipynb +++ /dev/null @@ -1,51 +0,0 @@ -{ - "cells": [ - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "%%init_spark\n", - "# Spark session & context\n", - "launcher.master = \"local\"\n", - "launcher.conf.spark.executor.cores = 1" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "// Sum of the first 100 whole numbers\n", - "val rdd = sc.parallelize(0 to 100)\n", - "rdd.sum()\n", - "// 5050" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "spylon-kernel", - "language": "scala", - "name": "spylon-kernel" - }, - "language_info": { - "codemirror_mode": "text/x-scala", - "file_extension": ".scala", - "help_links": [ - { - "text": "MetaKernel Magics", - "url": "https://metakernel.readthedocs.io/en/latest/source/README.html" - } - ], - "mimetype": "text/x-scala", - "name": "scala", - "pygments_lexer": "scala", - "version": "0.4.1" - } - }, - "nbformat": 4, - "nbformat_minor": 4 -} diff --git a/tests/all-spark-notebook/test_spark_notebooks.py b/tests/all-spark-notebook/test_spark_notebooks.py index fb4a6cbc..34f8d565 100644 --- a/tests/all-spark-notebook/test_spark_notebooks.py +++ b/tests/all-spark-notebook/test_spark_notebooks.py @@ -15,7 +15,7 @@ THIS_DIR = Path(__file__).parent.resolve() @pytest.mark.parametrize( "test_file", # TODO: add local_sparklyr - ["local_pyspark", "local_spylon", "local_sparkR", "issue_1168"], + ["local_pyspark", "local_sparkR", "issue_1168"], ) def test_nbconvert(container: TrackedContainer, test_file: str) -> None: """Check if Spark notebooks can be executed""" diff --git a/tests/base-notebook/test_packages.py b/tests/base-notebook/test_packages.py index a5d279fb..a8f3dca9 100644 --- a/tests/base-notebook/test_packages.py +++ b/tests/base-notebook/test_packages.py @@ -55,7 +55,6 @@ PACKAGE_MAPPING = { "pytables": "tables", "scikit-image": "skimage", "scikit-learn": "sklearn", - "spylon-kernel": "spylon_kernel", # R "randomforest": "randomForest", "rcurl": "RCurl",