How to integrate plaso/log2timeline into an airflow container?
arnvtn opened this issue · 1 comments
arnvtn commented
Hi.
I'm using an Airflow 2.9.1 container with docker compose. For various reasons I decided to link my docker-compose.yaml file with a Dockerfile in order to be able to add a certain number of tools at container startup. In particular, I want to use plaso/log2timeline to automate a number of tasks. However, when I try to install plaso-tools with apt, I get the following error:
=> ERROR [airflow-init 7/12] RUN apt update && apt-get -y upgrade && apt-get -y install locales plaso-tools 22.5s
------
> [airflow-init 7/12] RUN apt update && apt-get -y upgrade && apt-get -y install locales plaso-tools:
0.296
0.296 WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
0.296
0.473 Get:1 http://deb.debian.org/debian bookworm InRelease [151 kB]
0.527 Hit:2 https://ppa.launchpadcontent.net/gift/stable/ubuntu jammy InRelease
0.527 Get:3 http://deb.debian.org/debian bookworm-updates InRelease [55.4 kB]
0.528 Get:4 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
0.528 Hit:5 https://download.docker.com/linux/debian bookworm InRelease
0.528 Get:6 https://packages.microsoft.com/debian/12/prod bookworm InRelease [3618 B]
0.548 Hit:7 https://archive.mariadb.org/mariadb-10.11/repo/debian bookworm InRelease
0.576 Hit:8 https://apt.postgresql.org/pub/repos/apt bookworm-pgdg InRelease
0.630 Get:9 http://archive.ubuntu.com/ubuntu jammy InRelease [270 kB]
3.175 Fetched 528 kB in 3s (191 kB/s)
3.175 Reading package lists...
5.074 Building dependency tree...
5.641 Reading state information...
5.730 23 packages can be upgraded. Run 'apt list --upgradable' to see them.
5.730 W: http://archive.ubuntu.com/ubuntu/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
5.781 Reading package lists...
7.125 Building dependency tree...
7.612 Reading state information...
7.713 Calculating upgrade...
8.309 The following packages will be upgraded:
8.309 build-essential db-util dumb-init hicolor-icon-theme libalgorithm-diff-perl
8.309 libepoxy0 libev4 libgraphite2-3 libnpth0 libnsl-dev libnsl2 libthai-data
8.310 libthai0 libxau6 libxcomposite1 libxdmcp6 libxinerama1 libxxhash0 make patch
8.310 python3-jaraco.classes python3-pyparsing x11-common
8.476 23 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
8.476 Need to get 1173 kB of archives.
8.476 After this operation, 1380 kB disk space will be freed.
8.476 Get:1 https://ppa.launchpadcontent.net/gift/stable/ubuntu jammy/main amd64 python3-pyparsing all 3.0.9-1ppa1~jammy [80.2 kB]
8.554 Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 libxxhash0 amd64 0.8.1-1 [27.3 kB]
8.644 Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 libnsl-dev amd64 1.3.0-2build2 [71.3 kB]
8.730 Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 libnsl2 amd64 1.3.0-2build2 [42.3 kB]
8.817 Get:5 http://archive.ubuntu.com/ubuntu jammy/main amd64 libxau6 amd64 1:1.0.9-1build5 [7634 B]
8.905 Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 libxdmcp6 amd64 1:1.1.3-0ubuntu5 [10.9 kB]
8.992 Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 make amd64 4.3-4.1build1 [180 kB]
9.083 Get:8 http://archive.ubuntu.com/ubuntu jammy/main amd64 build-essential amd64 12.9ubuntu3 [4744 B]
9.189 Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 hicolor-icon-theme all 0.17-2 [9976 B]
9.297 Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libalgorithm-diff-perl all 1.201-1 [41.8 kB]
9.382 Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libepoxy0 amd64 1.5.10-1 [237 kB]
9.550 Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libgraphite2-3 amd64 1.3.14-1build2 [71.3 kB]
9.585 Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libnpth0 amd64 1.6-3build2 [8664 B]
9.650 Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 libthai-data all 0.1.29-1build1 [162 kB]
9.737 Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 libthai0 amd64 0.1.29-1build1 [19.2 kB]
9.825 Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 libxcomposite1 amd64 1:0.4.5-1build2 [7192 B]
9.911 Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 libxinerama1 amd64 2:1.1.4-3 [7382 B]
10.000 Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 patch amd64 2.7.6-7build2 [109 kB]
10.09 Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6452 B]
10.18 Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 x11-common all 1:7.7+23ubuntu2 [23.4 kB]
10.27 Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 db-util all 1:5.3.21~exp1ubuntu4 [2714 B]
10.37 Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dumb-init amd64 1.2.5-2 [14.1 kB]
10.53 Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libev4 amd64 1:4.33-1 [29.4 kB]
12.67 Fetched 1173 kB in 2s (533 kB/s)
(Reading database ... 32182 files and directories currently installed.)
12.81 Preparing to unpack .../libxxhash0_0.8.1-1_amd64.deb ...
12.85 Unpacking libxxhash0:amd64 (0.8.1-1) over (0.8.1-1) ...
12.99 Setting up libxxhash0:amd64 (0.8.1-1) ...
(Reading database ... 32181 files and directories currently installed.)
13.18 Preparing to unpack .../00-libnsl-dev_1.3.0-2build2_amd64.deb ...
13.18 Unpacking libnsl-dev:amd64 (1.3.0-2build2) over (1.3.0-2) ...
13.47 Preparing to unpack .../01-libnsl2_1.3.0-2build2_amd64.deb ...
13.48 Unpacking libnsl2:amd64 (1.3.0-2build2) over (1.3.0-2) ...
13.55 Preparing to unpack .../02-libxau6_1%3a1.0.9-1build5_amd64.deb ...
13.55 Unpacking libxau6:amd64 (1:1.0.9-1build5) over (1:1.0.9-1) ...
13.64 Preparing to unpack .../03-libxdmcp6_1%3a1.1.3-0ubuntu5_amd64.deb ...
13.69 Unpacking libxdmcp6:amd64 (1:1.1.3-0ubuntu5) over (1:1.1.2-3) ...
13.75 Preparing to unpack .../04-make_4.3-4.1build1_amd64.deb ...
13.78 Unpacking make (4.3-4.1build1) over (4.3-4.1) ...
13.91 Preparing to unpack .../05-build-essential_12.9ubuntu3_amd64.deb ...
13.91 Unpacking build-essential (12.9ubuntu3) over (12.9) ...
14.05 Preparing to unpack .../06-hicolor-icon-theme_0.17-2_all.deb ...
14.10 Unpacking hicolor-icon-theme (0.17-2) over (0.17-2) ...
14.23 Preparing to unpack .../07-libalgorithm-diff-perl_1.201-1_all.deb ...
14.26 Unpacking libalgorithm-diff-perl (1.201-1) over (1.201-1) ...
14.60 Preparing to unpack .../08-libepoxy0_1.5.10-1_amd64.deb ...
14.61 Unpacking libepoxy0:amd64 (1.5.10-1) over (1.5.10-1) ...
14.83 Preparing to unpack .../09-libgraphite2-3_1.3.14-1build2_amd64.deb ...
14.84 Unpacking libgraphite2-3:amd64 (1.3.14-1build2) over (1.3.14-1) ...
14.93 Preparing to unpack .../10-libnpth0_1.6-3build2_amd64.deb ...
14.94 Unpacking libnpth0:amd64 (1.6-3build2) over (1.6-3) ...
15.04 Preparing to unpack .../11-libthai-data_0.1.29-1build1_all.deb ...
15.08 Unpacking libthai-data (0.1.29-1build1) over (0.1.29-1) ...
15.34 Preparing to unpack .../12-libthai0_0.1.29-1build1_amd64.deb ...
15.36 Unpacking libthai0:amd64 (0.1.29-1build1) over (0.1.29-1) ...
15.43 Preparing to unpack .../13-libxcomposite1_1%3a0.4.5-1build2_amd64.deb ...
15.48 Unpacking libxcomposite1:amd64 (1:0.4.5-1build2) over (1:0.4.5-1) ...
15.58 Preparing to unpack .../14-libxinerama1_2%3a1.1.4-3_amd64.deb ...
15.60 Unpacking libxinerama1:amd64 (2:1.1.4-3) over (2:1.1.4-3) ...
15.74 Preparing to unpack .../15-patch_2.7.6-7build2_amd64.deb ...
15.77 Unpacking patch (2.7.6-7build2) over (2.7.6-7) ...
15.86 Preparing to unpack .../16-python3-jaraco.classes_3.2.1-3_all.deb ...
15.98 Unpacking python3-jaraco.classes (3.2.1-3) over (3.2.1-3) ...
16.18 Preparing to unpack .../17-x11-common_1%3a7.7+23ubuntu2_all.deb ...
16.22 Unpacking x11-common (1:7.7+23ubuntu2) over (1:7.7+23) ...
16.36 Preparing to unpack .../18-db-util_1%3a5.3.21~exp1ubuntu4_all.deb ...
16.40 Unpacking db-util (1:5.3.21~exp1ubuntu4) over (5.3.2) ...
16.53 Preparing to unpack .../19-dumb-init_1.2.5-2_amd64.deb ...
16.61 Unpacking dumb-init (1.2.5-2) over (1.2.5-2) ...
16.91 Preparing to unpack .../20-libev4_1%3a4.33-1_amd64.deb ...
16.93 Unpacking libev4:amd64 (1:4.33-1) over (1:4.33-1) ...
17.09 Preparing to unpack .../21-python3-pyparsing_3.0.9-1ppa1~jammy_all.deb ...
17.20 Unpacking python3-pyparsing (3.0.9-1ppa1~jammy) over (3.0.9-1) ...
17.53 Setting up dumb-init (1.2.5-2) ...
17.56 Setting up libgraphite2-3:amd64 (1.3.14-1build2) ...
17.60 Setting up libxau6:amd64 (1:1.0.9-1build5) ...
17.61 Setting up libxdmcp6:amd64 (1:1.1.3-0ubuntu5) ...
17.80 Setting up python3-jaraco.classes (3.2.1-3) ...
18.01 Setting up hicolor-icon-theme (0.17-2) ...
18.14 Setting up db-util (1:5.3.21~exp1ubuntu4) ...
18.15 Setting up libalgorithm-diff-perl (1.201-1) ...
18.17 Setting up libev4:amd64 (1:4.33-1) ...
18.19 Setting up x11-common (1:7.7+23ubuntu2) ...
18.24 Installing new version of config file /etc/X11/Xsession ...
18.42 invoke-rc.d: could not determine current runlevel
18.43 invoke-rc.d: policy-rc.d denied execution of restart.
18.44 Setting up libnpth0:amd64 (1.6-3build2) ...
18.47 Setting up python3-pyparsing (3.0.9-1ppa1~jammy) ...
18.70 Setting up make (4.3-4.1build1) ...
18.73 Setting up libepoxy0:amd64 (1.5.10-1) ...
18.77 Setting up libxinerama1:amd64 (2:1.1.4-3) ...
18.78 Setting up patch (2.7.6-7build2) ...
18.87 Setting up libnsl2:amd64 (1.3.0-2build2) ...
18.94 Setting up build-essential (12.9ubuntu3) ...
18.98 Setting up libthai-data (0.1.29-1build1) ...
19.07 Setting up libxcomposite1:amd64 (1:0.4.5-1build2) ...
19.08 Setting up libthai0:amd64 (0.1.29-1build1) ...
19.15 Setting up libnsl-dev:amd64 (1.3.0-2build2) ...
19.21 Processing triggers for libc-bin (2.36-9+deb12u7) ...
19.47 Reading package lists...
21.36 Building dependency tree...
21.88 Reading state information...
22.19 locales is already the newest version (2.36-9+deb12u7).
22.19 Some packages could not be installed. This may mean that you have
22.19 requested an impossible situation or if you are using the unstable
22.19 distribution that some required packages have not yet been created
22.19 or been moved out of Incoming.
22.19 The following information may help to resolve the situation:
22.19
22.19 The following packages have unmet dependencies:
22.42 python3-lz4 : Depends: python3 (< 3.11) but 3.11.2-1+b1 is to be installed
22.42 python3-psutil : Depends: python3 (< 3.11) but 3.11.2-1+b1 is to be installed
22.42 python3-yara : Depends: python3 (< 3.11) but 3.11.2-1+b1 is to be installed
22.42 python3-zmq : Depends: python3 (< 3.11) but 3.11.2-1+b1 is to be installed
22.42 python3-zstd : Depends: python3 (< 3.11) but 3.11.2-1+b1 is to be installed
22.43 E: Unable to correct problems, you have held broken packages.
------
failed to solve: process "/bin/bash -o pipefail -o errexit -o nounset -o nolog -c apt update && apt-get -y upgrade && apt-get -y install locales plaso-tools" did not complete successfully: exit code: 100
To get rid of this error, I tried installing an earlier version of python (python3.10) but it didn't work. I've also tried installing the dependencies that raise the error directly with apt, but when I do so, I get the following error:
=> ERROR [airflow-init 7/13] RUN apt-get -y install python3-lz4 python3-psutil python3-yara python3-zmq python3-zstd 4.9s
------
> [airflow-init 7/13] RUN apt-get -y install python3-lz4 python3-psutil python3-yara python3-zmq python3-zstd:
0.346 Reading package lists...
2.747 Building dependency tree...
3.420 Reading state information...
3.847 Some packages could not be installed. This may mean that you have
3.847 requested an impossible situation or if you are using the unstable
3.847 distribution that some required packages have not yet been created
3.847 or been moved out of Incoming.
3.847 The following information may help to resolve the situation:
3.847
3.847 The following packages have unmet dependencies:
4.762 python3-lz4 : Depends: python3 (< 3.11) but 3.11.2-1+b1 is to be installed
4.762 python3-psutil : Depends: python3 (< 3.11) but 3.11.2-1+b1 is to be installed
4.762 python3-yara : Depends: python3 (< 3.11) but 3.11.2-1+b1 is to be installed
4.762 python3-zmq : Depends: python3 (< 3.11) but 3.11.2-1+b1 is to be installed
4.762 python3-zstd : Depends: python3 (< 3.11) but 3.11.2-1+b1 is to be installed
4.780 E: Unable to correct problems, you have held broken packages.
------
failed to solve: process "/bin/bash -o pipefail -o errexit -o nounset -o nolog -c apt-get -y install python3-lz4 python3-psutil python3-yara python3-zmq python3-zstd" did not complete successfully: exit code: 100
If anyone has a solution, I'm very interested.
There is my Dockerfile :
FROM apache/airflow:2.9.1
USER root
RUN apt update && apt install -y build-essential libzstd-dev wget software-properties-common python3-launchpadlib apt-transport-https apt-utils libterm-readline-gnu-perl && apt-get -y update && apt-get -y upgra>
RUN sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 871920D1991BC93C
RUN add-apt-repository "deb http://archive.ubuntu.com/ubuntu jammy main universe"
RUN add-apt-repository "deb https://ppa.launchpadcontent.net/gift/stable/ubuntu jammy main"
RUN apt-get install -f
RUN apt-get -y install python3-lz4 python3-psutil python3-yara python3-zmq python3-zstd
RUN apt update && apt-get -y upgrade && apt-get -y install locales plaso-tools
RUN apt-get clean && rm -rf /var/cache/apt/* /var/lib/apt/lists/*
USER airflow
ADD requirements.txt .
RUN rm /home/airflow/.local/lib/python3.12/site-packages/airflow/example_dags/tutorial*
RUN rm /home/airflow/.local/lib/python3.12/site-packages/airflow/example_dags/example*
RUN pip install apache-airflow==${AIRFLOW_VERSION} -r requirements.txt
The container is running on Ubuntu 22.04 Jammy Jellyfish.
joachimmetz commented
Next version of Plaso will use Ubuntu 24.04
If anyone has a solution, I'm very interested.
Unable to correct problems, you have held broken packages.
This states it very clearly, you have conflicting packages. This is not a Plaso problem but a problem of your build configuration. Hence closing.
Note that this project is not resourced to support every exotic build configuration out there.