Commit Graph

239 Commits

Author SHA1 Message Date
Ayaz Salikhov
e1bd309263 Use curl instead of wget for Spark and Julia downloads (#1950)
Co-authored-by: Bjørn Jørgensen <bjornjorgensen@users.noreply.github.com>
2023-08-03 13:18:01 +04:00
Bjørn Jørgensen
df5d516e34 Pin pandas to version 1.5.3 to all spark images (#1925)
* 1.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add note

* typo

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* remove test

* >=1.5.3 and <2.0.0

* update test

* Update pyspark-notebook/Dockerfile

Co-authored-by: Ayaz Salikhov <mathbunnyru@users.noreply.github.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update pyspark-notebook/Dockerfile

Co-authored-by: Ayaz Salikhov <mathbunnyru@users.noreply.github.com>

* move test to file

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add pandas to EXCLUDED_PACKAGES

* add 1.5.3,<2.0.0 and sort list

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add '

* "pandas[version='>"

* Rename test_pandas_version.py to unit_pandas_version.py

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Ayaz Salikhov <mathbunnyru@users.noreply.github.com>
2023-06-27 15:42:18 +04:00
Bjørn Jørgensen
846f4cd319 Upgrade Apache Spark to 3.4.1 (#1922)
* rc1

* 3.4.1
2023-06-23 20:40:28 +04:00
Ayaz Salikhov
799403c4c6 Do not use wget --quiet, as it makes debugging much harder 2023-06-19 17:57:19 +04:00
Bjørn Jørgensen
2a32afabd8 Upgrade Apache Spark to 3.4.0 (#1880)
* 1.

* rc2

* rc3

* RC4

* fix checksum

* rc5

* rc6

* rc7

* 3.4.0

---------

Co-authored-by: Ayaz Salikhov <mathbunnyru@users.noreply.github.com>
2023-04-14 20:02:06 +04:00
Ayaz Salikhov
8dbeaa5401 Remove --quiet flag when not needed (#1887) 2023-03-09 15:52:20 +04:00
Bjørn Jørgensen
0baeda6253 Upgrade Apache Spark to 3.3.2 2023-02-17 19:39:54 +04:00
Ayaz Salikhov
55e224e35d Change spark download url to https://dlcdn.apache.org (#1876) 2023-02-16 11:36:15 +04:00
Bjørn Jørgensen
bada6c21e9 Upgrade spark to 3.3.1 (#1814) 2022-10-26 17:33:08 +04:00
Bjørn Jørgensen
bbe3bbcfa4 Add EXPOSE port 4040 for pyspark image (#1783)
* Add EXPOSE

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2022-09-09 00:37:40 +04:00
Bjørn Jørgensen
1533087aaf Add Scala as --build-arg (#1757)
* add scala version choise

* add ; \ fi

* change checksum and removed default scala version

* remove RUN

* add { } and remove old code

* remove 3 duplicated lines.

* Add the commint as a comment

* Add back #Fix

* Rename downloadeds as spark.tgz

* Fix doc

* Update specifics.md

* New fix

* Fix wget

* Remove make link to spark

* Set full path to /usr/local/spark

* Change /usr/local/spark to ${SPARK_HOME}

* fix RUN with if

* Remove empty lines

* Update Dockerfile

* Update Dockerfile

* Update Dockerfile

Co-authored-by: Ayaz Salikhov <mathbunnyru@users.noreply.github.com>
2022-07-27 22:51:05 +04:00
Darek
4449d13ba7 Merge branch 'jupyter:master' into master 2022-07-05 14:04:08 -04:00
Darek
b319f66084 Removing Spark config specific to Apache Arrow and Java 11 2022-07-05 10:13:18 -04:00
Ayaz Salikhov
44d3104b42 Remove export G_SLICE=always-malloc 2022-07-04 23:38:36 +04:00
Darek
aeb940220f Spark->3.3,Hadoop->3,Scala->2.13,Java->17 2022-06-16 14:53:37 -04:00
Ben Mares
476829f0f4 Proposed improvements for G_SLICE (#1697)
* Try to fix mamba under aarch64 QEMU by setting ENV G_SLICE=always-malloc

* Set G_SLICE inside if statement

* Write "extract_threads: 1" to /opt/.condarc

Co-authored-by: Ayaz Salikhov <mathbunnyru@gmail.com>
2022-05-03 16:51:38 +03:00
Ayaz Salikhov
bcf5118cb7 Fix minor PyCharm issues 2022-03-19 00:15:27 +02:00
Ayaz Salikhov
d35992de68 Add explicit pipefail and bash shell to all Dockerfiles 2022-03-17 13:07:47 +02:00
Ayaz Salikhov
6d04f39de6 Run tests on all children images 2022-02-17 16:34:15 +00:00
Brad Solomon
7507a5d22a Bump spark from 3.2.0 -> 3.2.1
Release notes:
https://spark.apache.org/releases/spark-release-3-2-1.html

Signed-off-by: Brad Solomon <81818815+brsolomon-deloitte@users.noreply.github.com>
2022-02-07 07:59:33 -05:00
Ayaz Salikhov
37c510fc8e Fix all typing issues 2022-01-23 12:44:16 +03:00
Ayaz Salikhov
3f6d683f76 Fix 2022-01-22 20:16:42 +03:00
Ayaz Salikhov
897b6e0544 Better warnings check 2022-01-22 19:43:47 +03:00
Ayaz Salikhov
968a9aedde Allow warning for pyspark 2022-01-22 13:18:20 +02:00
Ayaz Salikhov
12b618c5da Unify docker usage in tests 2022-01-22 00:46:20 +02:00
Ayaz Salikhov
2a1316c9ac Add some typing to tests 2022-01-18 19:13:17 +03:00
Ayaz Salikhov
c81a0eae4a Update pyspark-notebook/ipython_kernel_config.py 2022-01-09 11:54:24 +03:00
romainx
9529e3dffe Turn off ipython low-level output capture and forward 2022-01-07 17:53:27 +01:00
Darek
df72530d28 Upgrading Spark to 3.2.0 2021-10-20 13:42:25 +00:00
Ayaz Salikhov
e89b8a1475 Make all READMEs look the same 2021-10-17 01:10:40 +03:00
Ayaz Salikhov
a6d0ed456e Merge branch 'master' into asalikhov/automatic_conda_versioning 2021-07-09 16:52:31 +03:00
Ayaz Salikhov
3c9e62efce Introduce owner to Dockerfiles to make it easy to test locally 2021-07-08 17:26:56 +03:00
Ayaz Salikhov
38fa41fb24 Merge branch 'master' into asalikhov/automatic_conda_versioning 2021-07-02 21:19:32 +03:00
Ayaz Salikhov
ace42f37d7 Remove microbadge usage 2021-06-29 12:50:35 +03:00
Ayaz Salikhov
3c5e7c9cb4 Merge branch 'master' into asalikhov/automatic_conda_versioning 2021-06-29 02:28:53 +03:00
Erik Sundell
411663b5d4 black style tweaks: add trailing commas 2021-06-28 22:07:26 +02:00
Erik Sundell
fe3968efe0 pre-commit: run black autoformatter on all files 2021-06-28 22:07:23 +02:00
Ayaz Salikhov
44979c015d Merge branch 'master' into asalikhov/automatic_conda_versioning 2021-06-27 14:25:23 +03:00
Ayaz Salikhov
5c860df8c4 Move assert after logs output to make debug easier 2021-06-27 13:19:05 +03:00
Ayaz Salikhov
30a1d17085 Add a way to easily test units 2021-06-26 18:40:50 +03:00
Ayaz Salikhov
bfb8cc7d50 Allow conda to automatically deduce package versions 2021-06-20 22:13:15 +03:00
Ayaz Salikhov
4befdd5948 Merge branch 'master' into asalikhov/use_mamba 2021-06-07 23:05:26 +03:00
Ayaz Salikhov
396024a4dd Merge pull request #1335 from mathbunnyru/asalikhov/unify_bash_variables
Unify bash variables usage and add quotes where needed
2021-06-07 23:00:11 +03:00
Darek
3b5b735bb6 Upgrading spark_version->3.1.2 2021-06-04 15:02:52 +00:00
Ayaz Salikhov
6ab40f0002 Use mamba instead of conda in spark images 2021-06-01 20:30:30 +03:00
Ayaz Salikhov
0999f1a36f Unify bash variables usage and add quotes where needed 2021-05-26 13:06:10 +03:00
Ayaz Salikhov
fbceaa9892 Fixes 2021-05-22 14:01:01 +03:00
Ayaz Salikhov
66aaa99176 Update links 2021-05-21 14:17:18 +03:00
Ayaz Salikhov
0d08cc0efb Remove hooks from .dockerignore 2021-05-07 14:17:40 +03:00
Ayaz Salikhov
a3d9bb8f61 Merge branch 'master' into asalikhov/fix_markdown 2021-05-06 19:59:49 +03:00