hazza/docker-stacks
1
0
Fork 0
mirror of https://github.com/jupyter/docker-stacks.git synced 2025-10-18 15:32:56 +00:00
Code Issues Packages Projects Releases Wiki Activity

Spark version incompatibility with Delta

Browse Source 
Calling the "latest" tag in "FROM jupyter/pyspark-notebook:latest" installs the Spark 3.2 version which is incompatible with delta lake ( https://docs.delta.io/latest/releases.html#-compatibility-with-as ). Installing the last version of Spark 3.1.x resolves the issue. In the jupyter docker repo that would be version 3.1.18
This commit is contained in:
agustin genoud
2022-01-12 20:24:39 -03:00
committed by GitHub
parent f51aa480b7
commit 4579e5acf3
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docs/using/recipes.md
View File

@@ -528,7 +528,7 @@ Please note that the [Delta Lake](https://delta.io/) packages are only available
By adding the properties to `spark-defaults.conf`, the user no longer needs to enable Delta support in each notebook.
```dockerfile
FROM jupyter/pyspark-notebook:latest
FROM jupyter/pyspark-notebook:lab-3.1.18
ARG DELTA_CORE_VERSION="1.0.0"
RUN pip install --quiet --no-cache-dir delta-spark==${DELTA_CORE_VERSION} && \