Pin pandas to version 1.5.3 to all spark images (#1925)

* 1.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add note

* typo

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove test

* >=1.5.3 and <2.0.0

* update test

* Update pyspark-notebook/Dockerfile

Co-authored-by: Ayaz Salikhov <mathbunnyru@users.noreply.github.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update pyspark-notebook/Dockerfile

Co-authored-by: Ayaz Salikhov <mathbunnyru@users.noreply.github.com>

* move test to file

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add pandas to EXCLUDED_PACKAGES

* add 1.5.3,<2.0.0 and sort list

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add '

* "pandas[version='>"

* Rename test_pandas_version.py to unit_pandas_version.py

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Ayaz Salikhov <mathbunnyru@users.noreply.github.com>

pyspark-notebook/Dockerfile

@@ -65,7 +65,9 @@ RUN fix-permissions "/etc/ipython/"
 USER ${NB_UID}
 
 # Install pyarrow
+# Temporarily pin pandas to version 1.5.3, see: https://github.com/jupyter/docker-stacks/issues/1924
 RUN mamba install --yes \
+    'pandas>=1.5.3,<2.0.0' \
     'pyarrow' && \
     mamba clean --all -f -y && \
     fix-permissions "${CONDA_DIR}" && \

tests/base-notebook/test_packages.py

@@ -71,6 +71,7 @@ EXCLUDED_PACKAGES = [
     "hdf5",
     "jupyterlab-git",
     "openssl",
+    "pandas[version='>",
     "protobuf",
     "python",
     "r-irkernel",

tests/pyspark-notebook/units/unit_pandas_version.py

@@ -0,0 +1,6 @@
+# Copyright (c) Jupyter Development Team.
+# Distributed under the terms of the Modified BSD License.
+
+import pandas
+
+assert pandas.__version__ == "1.5.3"