scikit-hep/scikit-hep

Interest in scikit-hep jupyter docker stack?

tomeichlersmith opened this issue · 7 comments

tomeichlersmith commented

I've found this meta-package (and a lot of its sub-packages) extremely helpful for my work in HEP. One issue I noticed was that a lot of the computers I work on are on old/stale Python versions (e.g. CentOS7 is still Python 3.6.8) and building a new Python or getting IT to install an updated Python is time consuming. I also noticed that Project Jupyter has containerized images of Jupyter Lab (what they call "Docker Stacks"), so I quickly made a simple image that adds scikit-hep to the standard scipy stack. Is there any interest in incorporating this with this package or the scikit-hep organization?

https://github.com/tomeichlersmith/jupyter-scikit-hep

This is incredibly useful because many of our clusters have a container runner on them (e.g. singularity or apptainer) that can handle running the images built by this repo, so I can run the newest version of scikit-hep with a newer version of Python by simply downloading a ~1G image from the GitHub container registry.

eduardo-rodrigues commented

Dear @tomeichlersmith, thank you for your thumbs up on Scikit-HEP and this discussion. Very much appreciated.

Let me ping @matthewfeickert for a start, as he is a big consumer of Docker and will have a more educated opinion. Cc also @henryiii and @jpivarski. I might in fact email the admins mailing list so that everyone is aware ... Or you do it, since the list is open.

Again, many thanks.

eduardo-rodrigues commented

Further to this - do you @tomeichlersmith actually fancy emailing the admins list directly (see https://scikit-hep.org/getting-in-touch#mailing-lists) or shall I do it?

tomeichlersmith commented

I can do it just to introduce myself.

  • 👍1
matthewfeickert commented

Hi @tomeichlersmith 👋. I'm glad to hear that you're finding the scikit-hep metapackage useful in your work and that you've built a Docker image that is useful for you. A few of the Scikit-HEP Docker images provide their own minimal Docker images (pyhf does at least) and so I'm not sure I see the need to make a whole Scikit-HEP project for a Dockerfile. I'm also hesitant to start providing larger and larger Docker images (unless the broader community is really asking for them a lot) as it seems that end users are in a better place to make their own that provide exactly the environment that they need (your Dockerfile is an example of this in my mind).

Can you elaborate a bit more on what you think the value add is here? Obviously for you this is super useful as it is allowing you to get your work done with the Python version and tools that you want to use, which is great! But what value do other users get from it? If the only value is a more modern CPython runtime there are other ways to do this and I don't think that Scikit-HEP needs to be responsible for providing CPython.

tomeichlersmith commented

I guess, from my point of view, the value is a more modern CPython runtime and a fixed CPython runtime between all of the different workstations that could be collaborating on the same Python analysis code. This insures developers against complex version differences between different people while also providing users with a "portable" environment.

I hear what you're saying about having to maintain a potentially bloating Dockerfile and its related images and I don't feel too strongly about incorporating it - just wanted to offer.

matthewfeickert commented

I guess, from my point of view, the value is a more modern CPython runtime and a fixed CPython runtime between all of the different workstations that could be collaborating on the same Python analysis code. This insures developers against complex version differences between different people while also providing users with a "portable" environment.

This (Python runtimes) is something that I think end users need to work out for themselves. Scikit-HEP as an org provides libraries that are tested against a variety of Pythons that are supported (we try to follow a modified version of NEP 29). Scikit-HEP shouldn't be mandating what version of Python people use (the Docker images that we do provide for individual projects are also meant to be viewed as applications or application base environments and not primarily development environments).

One issue I noticed was that a lot of the computers I work on are on old/stale Python versions (e.g. CentOS7 is still Python 3.6.8) and building a new Python or getting IT to install an updated Python is time consuming.

I very much appreciate that this is a real pain. I would recommend that you try to have serious discussions with people that control the machines that you work on. Ignoring all the problems with making people tied to EOL Python releases by default, CentOS 7 is EOL next year

$ python -m pip install --upgrade norwegianblue
$ eol python
┌───────┬────────────┬─────────┬────────────────┬────────────┐
│ cycle │  release   │  latest │ latest release │    eol     │
├───────┼────────────┼─────────┼────────────────┼────────────┤
│  3.11 │ 2022-10-24 │  3.11.2 │   2023-02-07   │ 2027-10-24 │
│  3.10 │ 2021-10-04 │ 3.10.10 │   2023-02-07   │ 2026-10-04 │
│  3.9  │ 2020-10-05 │  3.9.16 │   2022-12-06   │ 2025-10-05 │
│  3.8  │ 2019-10-14 │  3.8.16 │   2022-12-06   │ 2024-10-14 │
│  3.7  │ 2018-06-26 │  3.7.16 │   2022-12-06   │ 2023-06-27 │
│  3.6  │ 2016-12-22 │  3.6.15 │   2021-09-03   │ 2021-12-23 │
│  3.5  │ 2015-09-12 │  3.5.10 │   2020-09-05   │ 2020-09-13 │
│  3.4  │ 2014-03-15 │  3.4.10 │   2019-03-18   │ 2019-03-18 │
│  3.3  │ 2012-09-29 │  3.3.7  │   2017-09-19   │ 2017-09-29 │
│  2.7  │ 2010-07-03 │  2.7.18 │   2020-04-19   │ 2020-01-01 │
│  2.6  │ 2008-10-01 │  2.6.9  │   2013-10-29   │ 2013-10-29 │
└───────┴────────────┴─────────┴────────────────┴────────────┘
$ eol centos
┌──────────┬────────────┬──────────┬────────────┬────────────┬──────────────────────────────────────────────────────────────┐
│  cycle   │  release   │  latest  │  support   │    eol     │                             link                             │
├──────────┼────────────┼──────────┼────────────┼────────────┼──────────────────────────────────────────────────────────────┤
│    9     │ 2021-09-15 │    9     │ 2027-05-31 │ 2027-05-31 │ https://blog.centos.org/2021/12/introducing-centos-stream-9/ │
│ stream-8 │ 2019-09-24 │    8     │ 2024-05-31 │ 2024-05-31 │  https://wiki.centos.org/Manuals/ReleaseNotes/CentOSStream   │
│    8     │ 2019-09-24 │ 8 (2111) │ 2021-12-31 │ 2021-12-31 │  https://wiki.centos.org/Manuals/ReleaseNotes/CentOS8.2111   │
│    7     │ 2014-07-07 │ 7 (2009) │ 2020-08-06 │ 2024-06-30 │  https://wiki.centos.org/Manuals/ReleaseNotes/CentOS7.2009   │
│    6     │ 2011-07-10 │   6.10   │ 2017-05-10 │ 2020-11-30 │   https://wiki.centos.org/Manuals/ReleaseNotes/CentOS6.10    │
└──────────┴────────────┴──────────┴────────────┴────────────┴──────────────────────────────────────────────────────────────┘

and with CERN and other organizations moving to AlmaLinux in response to CentOS8 getting killed people should be thinking about moving off CentOS in the very near future.

  • 👍1
tomeichlersmith commented

With this more personal project not within the scope of the Scikit-HEP project, I am closing this comment. Thank you for the discussion and extra information!

  • 👍1