Bjørn Jørgensen
bada6c21e9
Upgrade spark to 3.3.1 ( #1814 )
2022-10-26 17:33:08 +04:00
Bjørn Jørgensen
bbe3bbcfa4
Add EXPOSE port 4040 for pyspark image ( #1783 )
...
* Add EXPOSE
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
2022-09-09 00:37:40 +04:00
Bjørn Jørgensen
1533087aaf
Add Scala as --build-arg
( #1757 )
...
* add scala version choise
* add ; \ fi
* change checksum and removed default scala version
* remove RUN
* add { } and remove old code
* remove 3 duplicated lines.
* Add the commint as a comment
* Add back #Fix
* Rename downloadeds as spark.tgz
* Fix doc
* Update specifics.md
* New fix
* Fix wget
* Remove make link to spark
* Set full path to /usr/local/spark
* Change /usr/local/spark to ${SPARK_HOME}
* fix RUN with if
* Remove empty lines
* Update Dockerfile
* Update Dockerfile
* Update Dockerfile
Co-authored-by: Ayaz Salikhov <mathbunnyru@users.noreply.github.com >
2022-07-27 22:51:05 +04:00
Darek
4449d13ba7
Merge branch 'jupyter:master' into master
2022-07-05 14:04:08 -04:00
Darek
b319f66084
Removing Spark config specific to Apache Arrow and Java 11
2022-07-05 10:13:18 -04:00
Ayaz Salikhov
44d3104b42
Remove export G_SLICE=always-malloc
2022-07-04 23:38:36 +04:00
Darek
aeb940220f
Spark->3.3,Hadoop->3,Scala->2.13,Java->17
2022-06-16 14:53:37 -04:00
Ben Mares
476829f0f4
Proposed improvements for G_SLICE ( #1697 )
...
* Try to fix mamba under aarch64 QEMU by setting ENV G_SLICE=always-malloc
* Set G_SLICE inside if statement
* Write "extract_threads: 1" to /opt/.condarc
Co-authored-by: Ayaz Salikhov <mathbunnyru@gmail.com >
2022-05-03 16:51:38 +03:00
Ayaz Salikhov
bcf5118cb7
Fix minor PyCharm issues
2022-03-19 00:15:27 +02:00
Ayaz Salikhov
d35992de68
Add explicit pipefail and bash shell to all Dockerfiles
2022-03-17 13:07:47 +02:00
Ayaz Salikhov
6d04f39de6
Run tests on all children images
2022-02-17 16:34:15 +00:00
Brad Solomon
7507a5d22a
Bump spark from 3.2.0 -> 3.2.1
...
Release notes:
https://spark.apache.org/releases/spark-release-3-2-1.html
Signed-off-by: Brad Solomon <81818815+brsolomon-deloitte@users.noreply.github.com >
2022-02-07 07:59:33 -05:00
Ayaz Salikhov
37c510fc8e
Fix all typing issues
2022-01-23 12:44:16 +03:00
Ayaz Salikhov
3f6d683f76
Fix
2022-01-22 20:16:42 +03:00
Ayaz Salikhov
897b6e0544
Better warnings check
2022-01-22 19:43:47 +03:00
Ayaz Salikhov
968a9aedde
Allow warning for pyspark
2022-01-22 13:18:20 +02:00
Ayaz Salikhov
12b618c5da
Unify docker usage in tests
2022-01-22 00:46:20 +02:00
Ayaz Salikhov
2a1316c9ac
Add some typing to tests
2022-01-18 19:13:17 +03:00
Ayaz Salikhov
c81a0eae4a
Update pyspark-notebook/ipython_kernel_config.py
2022-01-09 11:54:24 +03:00
romainx
9529e3dffe
Turn off ipython low-level output capture and forward
2022-01-07 17:53:27 +01:00
Darek
df72530d28
Upgrading Spark to 3.2.0
2021-10-20 13:42:25 +00:00
Ayaz Salikhov
e89b8a1475
Make all READMEs look the same
2021-10-17 01:10:40 +03:00
Ayaz Salikhov
a6d0ed456e
Merge branch 'master' into asalikhov/automatic_conda_versioning
2021-07-09 16:52:31 +03:00
Ayaz Salikhov
3c9e62efce
Introduce owner to Dockerfiles to make it easy to test locally
2021-07-08 17:26:56 +03:00
Ayaz Salikhov
38fa41fb24
Merge branch 'master' into asalikhov/automatic_conda_versioning
2021-07-02 21:19:32 +03:00
Ayaz Salikhov
ace42f37d7
Remove microbadge usage
2021-06-29 12:50:35 +03:00
Ayaz Salikhov
3c5e7c9cb4
Merge branch 'master' into asalikhov/automatic_conda_versioning
2021-06-29 02:28:53 +03:00
Erik Sundell
411663b5d4
black style tweaks: add trailing commas
2021-06-28 22:07:26 +02:00
Erik Sundell
fe3968efe0
pre-commit: run black autoformatter on all files
2021-06-28 22:07:23 +02:00
Ayaz Salikhov
44979c015d
Merge branch 'master' into asalikhov/automatic_conda_versioning
2021-06-27 14:25:23 +03:00
Ayaz Salikhov
5c860df8c4
Move assert after logs output to make debug easier
2021-06-27 13:19:05 +03:00
Ayaz Salikhov
30a1d17085
Add a way to easily test units
2021-06-26 18:40:50 +03:00
Ayaz Salikhov
bfb8cc7d50
Allow conda to automatically deduce package versions
2021-06-20 22:13:15 +03:00
Ayaz Salikhov
4befdd5948
Merge branch 'master' into asalikhov/use_mamba
2021-06-07 23:05:26 +03:00
Ayaz Salikhov
396024a4dd
Merge pull request #1335 from mathbunnyru/asalikhov/unify_bash_variables
...
Unify bash variables usage and add quotes where needed
2021-06-07 23:00:11 +03:00
Darek
3b5b735bb6
Upgrading spark_version->3.1.2
2021-06-04 15:02:52 +00:00
Ayaz Salikhov
6ab40f0002
Use mamba instead of conda in spark images
2021-06-01 20:30:30 +03:00
Ayaz Salikhov
0999f1a36f
Unify bash variables usage and add quotes where needed
2021-05-26 13:06:10 +03:00
Ayaz Salikhov
fbceaa9892
Fixes
2021-05-22 14:01:01 +03:00
Ayaz Salikhov
66aaa99176
Update links
2021-05-21 14:17:18 +03:00
Ayaz Salikhov
0d08cc0efb
Remove hooks from .dockerignore
2021-05-07 14:17:40 +03:00
Ayaz Salikhov
a3d9bb8f61
Merge branch 'master' into asalikhov/fix_markdown
2021-05-06 19:59:49 +03:00
romainx
25d4876efe
Regular update
2021-05-04 18:36:03 +02:00
Romain
f46d59f4b6
Merge pull request #1279 from mathbunnyru/asalikhov/archive_spark
...
Install spark from archive.apache.org to be able to use old versions
2021-05-04 08:28:08 +02:00
Ayaz Salikhov
16ec0852c5
Merge branch 'master' into asalikhov/fix_markdown
2021-05-04 01:08:53 +03:00
Ayaz Salikhov
b5d6f04e43
Delete unused hadolint ignore
2021-05-02 23:28:53 +03:00
Ayaz Salikhov
0fff54f3f2
Install spark from archive.apache.org to be able to use old versions
2021-05-02 23:28:32 +03:00
Ayaz Salikhov
c2031e2a4a
Fix more errors
2021-05-01 13:07:40 +03:00
Ayaz Salikhov
bf5ec600d1
Use mdlint rules [WIP]
2021-05-01 12:52:41 +03:00
Ayaz Salikhov
833e235281
Implement SparkInfoManifest
2021-04-25 12:40:48 +03:00