Build the minimum amount of atoms a package depends on
msdobrescu opened this issue · 8 comments
Disclaimer: I am just beginning this kind of task, so, although a solution may be possible, I may not be able to find it yet.
Can't cleanly build media-sound/clementine
. By "cleanly" I mean building only the atoms not yet present in some existing layer under mOS
. I use the community repository
of mOS
.
For a start, I've made myself a diagram of layers dependency.
@startuml
scale 0.8
'skinparam linetype ortho
'skinparam linetype polyline
'skinparam nodesep 10
skinparam ranksep 100
skinparam artifact {
backgroundColor #FFD700
}
skinparam package<<repo>> {
backgroundColor #00FFFF
}
skinparam package<<collection>> {
backgroundColor #DAA520
}
skinparam package<<layer>> {
/'
borderColor Transparent
backgroundColor Transparent
fontColor Transparent
stereotypeFontColor Transparent
'/
backgroundColor #FFFFE0
}
skinparam package<<meta>> {
backgroundColor #EE82EE
}
allow_mixing
package "os-commons" <<repo>> {
collections "entities"
}
package "desktop" <<repo>> {
package "apps" <<collection>> {
artifact "vlc"
artifact "pavucontrol-qt"
}
package "meta" <<collection>> {
package "systemd-groups" <<meta>> {
}
}
package "cinnamon" <<layer>> {
}
package "codecs" <<layer>> {
}
package "enlightenment" <<layer>> {
}
package "firmware" <<layer>> {
}
package "gnome" <<layer>> {
}
package "gnome-common" <<layer>> {
}
package "java" <<layer>> {
}
package "kde-apps-minimal" <<layer>> {
}
package "kde-common" <<layer>> {
}
package "kde-pim" <<layer>> {
}
package "lxqt" <<layer>> {
}
package "mate" <<layer>> {
}
package "net-tools" <<layer>> {
}
package "plasma" <<layer>> {
}
package "qt" <<layer>> {
}
package "sys-fs" <<layer>> {
}
package "system-x" <<collection>> as sxc {
artifact "gcc-base"
package "system-x" <<layer>> {
}
}
package "wayfire" <<layer>> {
}
package "X" <<layer>> {
}
package "xfce" <<layer>> {
}
}
"gnome-common" <-- "cinnamon"
"system-x" <-- "codecs"
"entities" <-- "codecs"
"X" <-- "enlightenment"
"X" <-- "gnome-common"
"entities" <-- "gnome-common"
"gnome-common" <-- "gnome"
"X" <-- "gnome"
"entities" <-- "gnome"
"X" <-- "java"
"plasma" <-- "kde-apps-minimal"
"qt" <-- "kde-common"
"entities" <-- "kde-common"
"plasma" <-- "kde-pim"
"entities" <-- "kde-pim"
"kde-common" <-- "lxqt"
"pavucontrol-qt" <-- "lxqt"
"X" <-- "mate"
"gnome-common" <-- "mate"
"system-x" <-- "net-tools"
"entities" <-- "net-tools"
"kde-common" <-- "plasma"
"vlc" <-- "plasma"
"entities" <-- "plasma"
"X" <-- "qt"
"system-x" <-- "sys-fs"
"entities" <-- "systemd-groups"
"gcc-base" <-- "system-x"
"systemd-groups" <-- "system-x"
"entities" <-- "system-x"
"system-x" <-- "X"
"sys-fs" <-- "X"
"codecs" <-- "X"
"net-tools" <-- "X"
"gnome-common" <-- "wayfire"
"gnome-common" <-- "xfce"
@enduml
This brings the question:
Why gnome layer requires gnome-common
and X
layers while gnome-common requires X
already?
In other words why requiring gnome-common
is not enough (by bringing X
in its requirements)?
Or is it?
So, analyzing media-sound/clementine
's deps, I have found that building it is depending on:
plasma
layer primarlygnome-commons
layer due tomtp
implemented bygnome-base/gvfs
And here we go...
Describe alternatives you've considered
Incrementally, I've added the needed deps one by one to the community repository
collection:
- &libs
category: "libs"
name: "chromaprint"
version: "1.5.0"
labels:
emerge.jobs: "1"
emerge.packages: >-
media-libs/chromaprint
atom: "media-libs/chromaprint"
provides:
- name: "chromaprint"
category: "media-libs"
version: ">=0"
pre_emerge:
- dev-cpp/gtest
requires:
- &layers-X
category: "layers"
name: "X"
version: ">=0"
build_requires:
- <<: *layers-X
- <<: *libs
name: "libmygpo-qt"
version: "1.1.0"
labels:
emerge.jobs: "1"
emerge.packages: >-
media-libs/libmygpo-qt
atom: "media-libs/libmygpo-qt"
provides:
- name: "libmygpo-qt"
category: "media-libs"
version: ">=0"
pre_emerge:
- dev-cpp/gtest
requires:
- &layers-qt
category: "layers"
name: "qt"
version: ">=0"
build_requires:
- <<: *layers-qt
The trouble begins with gnome-base/gvfs
.
In order to satisfy it, to avoid emerge stopping due to errors like these below, must refer gnome-commons
layer.
Source: (here).
Step 9/9 : RUN emerge -j ${JOBS} media-sound/clementine
INFO ---> Running in 212eab660414
INFO
* IMPORTANT: 11 news items need reading for repository 'gentoo'.
* Use eselect news read to view new items.
INFO [ebuild N ] gnome-base/gvfs-1.48.1-r1
[ebuild N ] media-sound/clementine-1.4.0_rc1
INFO
The following USE changes are necessary to proceed:
(see "package.use" in the portage(5) man page for more details)
# required by media-sound/clementine-1.4.0_rc1::gentoo[mtp]
# required by media-sound/clementine (argument)
>=gnome-base/gvfs-1.48.1-r1 mtp
Use --autounmask-write to write changes to config files (honoring
CONFIG_PROTECT). Carefully examine the list of proposed changes,
paying special attention to mask or keyword changes that may expose
experimental or unstable packages.
INFO The command '/bin/sh -c emerge -j ${JOBS} media-sound/clementine' returned a non-zero code: 1
ERROR Error: failed building package image: Could not push image: quay.io/mocaccinocache/desktop:f92d013b7212732aa056a8b3cbd7317bfe3beaa2b90614fafd1e2a9d7b8453ea clementine-apps-1.4.0.dockerfile: Could not build image: quay.io/mocaccinocache/desktop:f92d013b7212732aa056a8b3cbd7317bfe3beaa2b90614fafd1e2a9d7b8453ea clementine-apps-1.4.0.dockerfile: Failed running command: : exit status 1
ERROR Bailing out
exit status 1
Error: Process completed with exit code 1.
Clementine config has become this:
- <<: *apps
name: "clementine"
version: "1.4.0"
labels:
emerge.jobs: "1"
emerge.packages: >-
media-sound/clementine
atom: "media-sound/clementine"
provides:
- name: "clementine"
category: "media-sound"
version: ">=0"
pre_emerge:
- dev-cpp/gtest
- media-libs/chromaprint
- media-libs/libmygpo-qt
requires:
- <<: *layers-plasma
- <<: *layers-gnome-common
- &libs-chromaprint
category: "libs"
name: "chromaprint"
version: ">=0"
- &libs-libmygpo-qt
category: "libs"
name: "libmygpo-qt"
version: ">=0"
build_requires:
- <<: *layers-plasma
- <<: *layers-gnome-common
The outcome is starting emerge qt
layer, as seen here, but that is expected to be provided by the plasma
layer inherently, isn't it?
INFO Building image quay.io/mocaccino/mocaccino-community:40dabfe5bb779e8925176e2c5cfab2060d862cdb9b2b0f338d57f95eef0d779e
INFO Sending build context to Docker daemon 14.85kB
INFO
INFO Step 1/9 : FROM quay.io/mocaccino/mocaccino-community:builder-ae8ba98acdf11e8208c134ec199901cd
INFO ---> 47494cd2783a
Step 2/9 : COPY . /luetbuild
INFO ---> 2757dc43e60d
Step 3/9 : WORKDIR /luetbuild
INFO ---> Running in f772eee9e5ce
INFO Removing intermediate container f772eee9e5ce
---> c99af5f46951
Step 4/9 : ENV PACKAGE_NAME=qt
INFO ---> Running in 0a03324e9384
INFO Removing intermediate container 0a03324e9384
---> cbeccd427b2c
Step 5/9 : ENV PACKAGE_VERSION=5.15.2+85
INFO ---> Running in d1e2a9d35b50
INFO Removing intermediate container d1e2a9d35b50
---> 14475feb83a6
Step 6/9 : ENV PACKAGE_CATEGORY=layers
INFO ---> Running in 9b1b92aa8db9
INFO Removing intermediate container 9b1b92aa8db9
---> 0ccb3b79a2fb
Step 7/9 : ENV JOBS=1
INFO ---> Running in 3c989ca53739
INFO Removing intermediate container 3c989ca53739
---> 993ab3f4b980
Step 8/9 : RUN emerge -j ${JOBS} dev-qt/qdbus dev-qt/qtbluetooth dev-qt/qtcharts dev-qt/qtchooser dev-qt/qtconcurrent dev-qt/qtcore dev-qt/qtdbus dev-qt/qtdeclarative dev-qt/qtgraphicaleffects dev-qt/qtgui dev-qt/qthelp dev-qt/qtimageformats dev-qt/qtlockedfile dev-qt/qtmultimedia dev-qt/qtnetwork dev-qt/qtnetworkauth dev-qt/qtopengl dev-qt/qtpaths dev-qt/qtpositioning dev-qt/qtprintsupport dev-qt/qtquickcontrols dev-qt/qtquickcontrols2 dev-qt/qtscript dev-qt/qtscxml dev-qt/qtsensors dev-qt/qtserialport dev-qt/qtsingleapplication dev-qt/qtspeech dev-qt/qtsql dev-qt/qtsvg dev-qt/qttest dev-qt/qttranslations dev-qt/qtvirtualkeyboard dev-qt/qtwayland dev-qt/qtwebchannel dev-qt/qtwebsockets dev-qt/qtwidgets dev-qt/qtx11extras dev-qt/qtxml dev-qt/qtxmlpatterns dev-qt/qtwebengine
INFO ---> Running in 0c665a9cbcc0
INFO
* IMPORTANT: 11 news items need reading for repository 'gentoo'.
* Use eselect news read to view new items.
INFO >>> Verifying ebuild manifests
INFO >>> Running pre-merge checks for net-libs/nodejs-14.17.6
INFO >>> Running pre-merge checks for dev-qt/qtcore-5.15.2-r10
INFO >>> Emerging (1 of 48) dev-qt/qtchooser-66-r1::gentoo
INFO >>> Installing (1 of 48) dev-qt/qtchooser-66-r1::gentoo
INFO >>> Recording dev-qt/qtchooser in "world" favorites file...
INFO >>> Emerging (2 of 48) dev-libs/re2-0.2021.06.01::gentoo
INFO >>> Installing (2 of 48) dev-libs/re2-0.2021.06.01::gentoo
INFO >>> Emerging (3 of 48) app-arch/snappy-1.1.9::gentoo
INFO >>> Installing (3 of 48) app-arch/snappy-1.1.9::gentoo
INFO >>> Emerging (4 of 48) dev-lang/python-2.7.18_p13::gentoo
...
Describe the solution you'd like
I would expect to pull all the necessary layers, specifying the top ones only, probably build the container in order to build only the remaining atoms.
In the case above, those atoms would be:
dev-cpp/gtest
media-libs/chromaprint
media-libs/libmygpo-qt
media-sound/clementine
Additional context
I am aware sometimes the layers structure is not ideal, things can be moved around, but it's not always possible (not saying here it's the case for media-sound/clementine
).
That's more an issue on how a repository was designed, than rather an issue with luet itself. Or I am missing something?
Actually, I thought this myself. But, seems some KDE apps need Gnome libs sometimes, due to missing those implementation in a KDE/Qt lib. This would need, for instance, to build some intermediary layer with the common stuff. Eventually, due to deep deps, some may end needing merging some layers, like qt
or plasma
and gonme-common
. On the user side the result is the same, indeed, but on repo builder side does not make sense, probably.
My question above is still in place:
Why gnome layer requires gnome-common and X layers while gnome-common requires X already?
In other words why requiring gnome-common is not enough (by bringing X in its requirements)?
Or is it?
I mean, why would need that (apparently) redundancy?
Then, why qt
is emerged for the presented setup? What layer is brought of the required ones then? Why gnome-common
and not plasma
? I presume it's gnome-common
, because it seems to start emerging qt
- and I expect to continue up to plasma
.
For such a case, seems necessary to merge the containers somehow, to end up with plasma
+ gnome-common
and build. Otherwise takes forever and we shoot flies with a big cannon...
I would analyze this particular case if it's possible to move atoms in a way that it makes it build with a smaller print, but this is just a case.
Actually, I thought this myself. But, seems some KDE apps need Gnome libs sometimes, due to missing those implementation in a KDE/Qt lib. This would need, for instance, to build some intermediary layer with the common stuff. Eventually, due to deep deps, some may end needing merging some layers, like
qt
orplasma
andgonme-common
. On the user side the result is the same, indeed, but on repo builder side does not make sense, probably.
Yes, you can definitely have a building layer that contains all the deps, and have smaller packages extracting just the required from it - there is no need to make that layer installable for the user
Yes, you can definitely have a building layer that contains all the deps, and have smaller packages extracting just the required from it - there is no need to make that layer installable for the user
That seems out of my reach as mOS
user. Or I don't have the knowledge how, given the current layers definition.
Yes, you can definitely have a building layer that contains all the deps, and have smaller packages extracting just the required from it - there is no need to make that layer installable for the user
That seems out of my reach as
mOS
user. Or I don't have the knowledge how, given the current layers definition.
That's why I think this is a mOS issue rather then a luet one... Nevertheless https://luet-lab.github.io/docs/docs/concepts/packages/specfile/#building-strategies should give you some indications: you can hust select the parts you are interested to from a layer and create packages as necessary
Closing the issue then 👍