[Ubuntu] Stenotype keeps crashing

Question

[Ubuntu] Stenotype keeps crashing

FabFaeb opened this issue 6 years ago · 9 comments

Hi,

I am currently playing around with Stenographer on several Ubuntu 16.04/18.04 VMs. Unfortunately on all of these machines, I see recurring crashes of the stenotype component after 2 to 5 minutes (usually around the 2-minute-mark). The error looks like this (from /var/log/syslog):

Apr 2 07:51:48 nsm-sensor stenographer[699]:
2019-04-02T05:51:48.716656Z T:537fe7 [util.cc:117] WATCHDOG FAILURE: Thread 0
012ABORTABORTABORT#012/usr/bin/stenotype(+0x70a1) [0x5604943e40a1]
012/usr/bin/stenotype(+0x5240) [0x5604943e2240]
012/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xbd57f) [0x7effda83b57f]
012/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7effdad536db]
012/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7effda29688f]
Apr 2 07:51:48 nsm-sensor stenographer[699]:
2019/04/02 07:51:48 Stenotype stopped after 2m52.433619838s: stenotype wait failed: signal: aborted (core dumped)
Apr 2 07:51:48 nsm-sensor stenographer[699]:
2019/04/02 07:51:48 Deleted stale output file "/etc/stenographer/packets/.1554184258882421"
Apr 2 07:56:43 nsm-sensor stenographer[699]:
2019-04-02T05:56:43.235697Z T:8cdec7 [util.cc:117] WATCHDOG FAILURE: Thread 0
012ABORTABORTABORT#012/usr/bin/stenotype(+0x70a1) [0x55bdabf000a1]
012/usr/bin/stenotype(+0x5240) [0x55bdabefe240]
012/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xbd57f) [0x7f8b0fa5d57f]
012/lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f8b0ff756db]
012/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f8b0f4b888f]
Apr 2 07:56:43 nsm-sensor stenographer[699]:
2019/04/02 07:56:43 Stenotype stopped after 4m54.482629796s: stenotype wait failed: signal: aborted (core dumped)
Apr 2 07:56:43 nsm-sensor stenographer[699]:
2019/04/02 07:56:43 Deleted stale output file "/etc/stenographer/packets/.1554184554061973"

Additionally, the written files in the packets directory are all empty. Essentially I am having the same issues as described in this reddit post: https://www.reddit.com/r/AskNetsec/comments/9181kc/anyone_run_stenographer/

Has anyone else experienced similar issues? Are there any ideas what I could do about it?
What I have already tried:

Change hardware specs of VM (currently 2 Cores and 8 GB of RAM)
Different seccomp flags
Reinstalling the system
Ensured enough disk space is available

I'll be thankful for any hints!

Answer 1 · 2019-04-07T11:34:09.000Z

If nothing gets written to disk this might have to do with file permissions. Try running stenotype manually mimicking the options you see the service using, plus the -v FLAG to get more insight as of the crash

Answer 2 · 2019-04-08T13:24:41.000Z

Stenotype simply stops processing immediately, never throwing any errors or logging any packets:

2019-04-08T13:23:55.240558Z T:9479a7 [stenotype.cc:545] Stenotype running with these arguments:
2019-04-08T13:23:55.240610Z T:9479a7 [stenotype.cc:547] 0:	"stenotype"
2019-04-08T13:23:55.240615Z T:9479a7 [stenotype.cc:547] 1:	"--dir=/data/stenographer/packets"
2019-04-08T13:23:55.240620Z T:9479a7 [stenotype.cc:547] 2:	"--iface=enp2s0f1"
2019-04-08T13:23:55.240624Z T:9479a7 [stenotype.cc:547] 3:	"-vvvvvvvvvvv"
2019-04-08T13:23:55.240628Z T:9479a7 [stenotype.cc:549] Starting, page size is 4096
2019-04-08T13:23:55.240634Z T:9479a7 [stenotype.cc:576] Setting up AF_PACKET sockets for packet reading
2019-04-08T13:23:55.596129Z T:9479a7 [packets.cc:229] Setting promiscuous mode for enp2s0f1
2019-04-08T13:23:55.596169Z T:9479a7 [packets.cc:237] Interface enp2s0f1 already in promisc mode
2019-04-08T13:23:55.596234Z T:9479a7 [stenotype.cc:248] Dropping privileges
2019-04-08T13:23:55.596246Z T:9479a7 [stenotype.cc:253] Dropping priviledges from 0 to GID nogroup
2019-04-08T13:23:55.596503Z T:9479a7 [stenotype.cc:265] Dropping priviledges from 0 to UID nobody
2019-04-08T13:23:55.598026Z T:9479a7 [stenotype.cc:652] Starting writing threads
2019-04-08T13:23:55.598050Z T:9479a7 [stenotype.cc:655] Starting thread 0
2019-04-08T13:23:55.598074Z T:1473d7 [stenotype.cc:427] Handling signals
2019-04-08T13:23:55.598105Z T:9479a7 [stenotype.cc:666] Starting indexing threads
2019-04-08T13:23:55.598200Z T:1373b7 [stenotype.cc:395] Starting WriteIndexes thread 0
2019-04-08T13:23:55.598858Z T:9479a7 [stenotype.cc:680] ===============Waiting for thread==============
2019-04-08T13:23:55.598859Z T:1373b7 [stenotype.cc:403] Waiting for index
2019-04-08T13:23:55.598859Z T:13f3c7 [stenotype.cc:450] Thread 0 starting to process packets
2019-04-08T13:23:55.599117Z T:9479a7 [stenotype.cc:683] Thread finished
2019-04-08T13:23:55.599144Z T:9479a7 [stenotype.cc:686] Finished all threads
2019-04-08T13:23:55.599169Z T:9479a7 [stenotype.cc:689] Closing write index queue 0, waiting for thread
2019-04-08T13:23:55.599245Z T:1373b7 [stenotype.cc:405] Got index 0
2019-04-08T13:23:55.599259Z T:1373b7 [stenotype.cc:414] Exiting write index thread
2019-04-08T13:23:56.598772Z T:9479a7 [stenotype.cc:693] Index thread finished
2019-04-08T13:23:56.598803Z T:9479a7 [stenotype.cc:698] Process exiting successfully
2019-04-08T13:23:56.598952Z T:1473d7 [stenotype.cc:436] Signal handling done

Answer 3 · 2019-04-08T13:42:29.000Z

This won't solve your issue but I would try running it manually to make sure data gets stored at all, ie:

stenotype --preallocate_file_mb=20 --blocks=128 --seccomp=none -v --threads=1 --iface=enp2s0f1 --dir=/data/stenographer --fileage_sec=1200 --blockage_sec=30

Answer 4 · 2019-04-08T15:01:02.000Z

This won't solve your issue but I would try running it manually to make sure data gets stored at all, ie:
stenotype --preallocate_file_mb=20 --blocks=128 --seccomp=none -v --threads=1 --iface=enp2s0f1 --dir=/data/stenographer --fileage_sec=1200 --blockage_sec=30

That doesn't fix it but better reveals the problem. Seeing that it is failing to create directories got me to look at the ID that it's dropping to. Contrary to the documentation, it is dropping to nobody and nogroup rather than stenographer, which is what I would expect since the INSTALL.MD directs you to chown -R stenographer:stenographer the path for the repository.

Answer 5 · 2019-04-11T12:58:51.000Z

Sorry for my delayed answer:
In my case it seemed to be a combination of a file permission problem and seccomp. When deactivating seccomp and setting the appropriate permissions, stenographer works as intended. When I keep seccomp activated I get the following lines in the syslog and stenotype just dies.

kernel: [  452.433303] audit: type=1326 audit(1554283284.929:18): auid=4294967295 uid=111 gid=117 ses=4294967295 pid=23371 comm="stenotype" exe="/usr/bin/stenotype" sig=31 arch=c000003e syscall=2 compat=0 ip=0x7fc10ab3aa5f code=0x0
 
kernel: [  453.434763] audit: type=1326 audit(1554283285.933:19): auid=4294967295 uid=111 gid=117 ses=4294967295 pid=23369 comm="stenotype" exe="/usr/bin/stenotype" sig=31 arch=c000003e syscall=231 compat=0 ip=0x7fc10ab8d748 code=0x0

Is my understanding, that these two syscalls (231 and 2) need to be added to stenotype.cc via SECCOMP_RULE_ADD, correct?

Answer 6 · 2020-06-24T06:12:53.000Z

I am getting this error while trying to run. any help much appreciated. This is with default options also. Do we need to run from specific user. I am running from root:root

stenotype --preallocate_file_mb=20 --blocks=128 --seccomp=none -v --threads=1 --iface=ens224 --dir=/data/stenographer --fileage_sec=1200 --blockage_sec=30
2020-06-24T06:07:42.421984Z T:653ba7 [stenotype.cc:554] Starting, page size is 4096
2020-06-24T06:07:42.422128Z T:653ba7 [stenotype.cc:581] Setting up AF_PACKET sockets for packet reading
2020-06-24T06:07:42.447663Z T:653ba7 [stenotype.cc:252] Dropping privileges
2020-06-24T06:07:42.447695Z T:653ba7 [stenotype.cc:257] Dropping priviledges from 0 to GID nogroup
2020-06-24T06:07:42.448809Z T:653ba7 [stenotype.cc:260] CHECK(group != NULL) Unable to get info for group nogroup
ABORTABORTABORT
stenotype() [0x4081f8]
stenotype() [0x42bb14]
stenotype() [0x42e1bc]
/lib64/libc.so.6(__libc_start_main+0xf5) [0x7fc86373d545]
stenotype() [0x40412b]

Aborted (core dumped)

Answer 7 · 2020-07-20T10:19:56.000Z

For me, stenographer seems to be working as expected when I run it via systemd after setting the right file permissions for the data directories. I can even enable seccomp after adding the two new rules I introduced in PR #225 .
I think your problems might be different from mine and are probably worth their own issue. For this reason, I will close this issue now.

Answer 8 · 2022-06-17T14:16:05.000Z

@FabFaeb
Would you be able to clarify what file permissions for which directories you changed?
I believe I am having this issue on Ubuntu 20.04 with stenographer installed from apt (0.0~git20190917.0.3545c54-1build1)

I have tried to verify the permissions match the ones modified in install.sh but to no avail.

Here are the errors I am getting, maybe it is a different issue:

Jun 16 20:31:55 l3c-ids stenographer[66445]: 2022/06/16 20:31:55 Stenotype stopped after 2m0.755949623s: stenotype wait failed: signal: aborted (core dumped)
Jun 16 20:31:55 l3c-ids stenographer[66445]: /usr/sbin/stenotype(+0x7ebb) [0x55cedae42ebb]
                                             /usr/sbin/stenotype(+0x7956) [0x55cedae42956]
                                             /lib/x86_64-linux-gnu/libstdc++.so.6(+0xd6de4) [0x7f9b19898de4]
                                             /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f9b199ce609]
                                             /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f9b196d2133]
Jun 16 20:31:55 l3c-ids stenographer[66445]: 2022-06-16T20:31:55.316355Z T:97c517 [util.cc:117] WATCHDOG FAILURE: Thread 0
                                             ABORTABORTABORT

Answer 9 · 2022-06-18T15:12:25.000Z

@zwatts00
Since this was quite some time ago and I am longer working with stenographer, I am not able to say what exactly I changed - sorry.
However, I am relatively sure that I changed permissions on the "data-directory", where the pcaps should be written.
I would suggest to (temporarily) chmod 777 this directory and then run with --seccomp=none to see if this is related to this issue.