* Recipe for Microsoft SQL Server
Dockerfile with Microsoft SQL Server ODBC driver and pyodbc.
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
* Get rid of apt-key, only install what's needed and clean up properly
* Replace apt-get with apt
* Switch from apt to apt-get
* Use --no-install-recommends in the right place
* Use simple pip install
* Install pip package under jovyan
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Ayaz Salikhov <mathbunnyru@users.noreply.github.com>
Delta 1.2.1:
Fix an issue with loading error messages in --packages mode. Previous release had a bug that resulted in user getting NullPointerException instead of proper error message when using Delta Lake with --packages mode either in pyspark or spark-shell (Fix, Test)
Fix incorrect exception type thrown in some Python APIs. A bug caused pyspark to throw incorrect type of exceptions instead of expected AnalysisException. This issue is fixed. See issue #1086 for more details.
Fix for S3 multi-cluster mode configuration. A bug in the S3 multi-cluster mode caused --conf to not work for certain configuration parameters. This issue is fixed by having these configuration parameters begin with spark. See the updated documentation.
Make the GCS LogStore configuration simpler by automatically deriving the LogStore implementation class config spark.delta.logStore.gs.impl from the scheme in the table path. See the updated documentation.
Make SetAccumulator thread safe. SetAccumulator used by Merge was not thread safe and might cause executor heartbeat failures in rare cases. This was fixed by using a synchronized set.
Calling the "latest" tag in "FROM jupyter/pyspark-notebook:latest" installs the Spark 3.2 version which is incompatible with delta lake ( https://docs.delta.io/latest/releases.html#-compatibility-with-as ). Installing the last version of Spark 3.1.x resolves the issue. In the jupyter docker repo that would be version 3.1.18
