rabobank-cdc/DeTTECT

APT19 Group part of the data set

DFIRmadness opened this issue · 8 comments

DFIRmadness commented 
root@f41cd8582e67:/opt/DeTTECT# python dettect.py g -g apt19 -p all -of test.json
[!] Group not part of the data set: apt19
root@f41cd8582e67:/opt/DeTTECT# python dettect.py g -g G0073 -p all -of test.json
[!] Group not part of the data set: g0073

It was the same for apt17, and some others I tested.

It works great for many of the others I am trying. I am sure I am missing something here and the issue is with a misunderstanding of mine.

Thanks in advance. I love the project.

rubinatorz commented

Hi @DFIRmadness

I tried to reproduce the error but I couldn't. Can you try to clear the cache directory and try it again?

If that doesn't work, please let us know the DeTT&CT version and your python packages versions (pip list).

Regards,
Ruben

DFIRmadness commented

Hey Ruben,

Thanks for the help.

Here are the results:

root@f41cd8582e67:/opt/DeTTECT# rm cache/*
root@f41cd8582e67:/opt/DeTTECT# python dettect.py g -g G0073 -p all -of test.json
[taxii2client.common] [WARNING ] [2021-10-18 20:26:12,696] TAXII Server Response did not include 'Content-Range' header - results could be incomplete
[taxii2client.common] [WARNING ] [2021-10-18 20:26:12,915] TAXII Server Response did not include 'Content-Range' header - results could be incomplete
[taxii2client.common] [WARNING ] [2021-10-18 20:26:13,051] TAXII Server Response did not include 'Content-Range' header - results could be incomplete
[taxii2client.common] [WARNING ] [2021-10-18 20:26:14,811] TAXII Server Response did not include 'Content-Range' header - results could be incomplete
[taxii2client.common] [WARNING ] [2021-10-18 20:26:16,777] TAXII Server Response did not include 'Content-Range' header - results could be incomplete
[taxii2client.common] [WARNING ] [2021-10-18 20:26:17,295] TAXII Server Response did not include 'Content-Range' header - results could be incomplete
[taxii2client.common] [WARNING ] [2021-10-18 20:26:19,528] TAXII Server Response did not include 'Content-Range' header - results could be incomplete
[taxii2client.common] [WARNING ] [2021-10-18 20:26:20,240] TAXII Server Response did not include 'Content-Range' header - results could be incomplete
[taxii2client.common] [WARNING ] [2021-10-18 20:26:20,579] TAXII Server Response did not include 'Content-Range' header - results could be incomplete
[!] Group not part of the data set: g0073
root@f41cd8582e67:/opt/DeTTECT# python dettect.py --version
dettect.py 1.4.3

I downloaded this container on Friday, October 15th.

rubinatorz commented

hi @DFIRmadness

Thanks for your input. I'm now able reproduce the error on our latest docker image. It was a struggle to find out what the root cause was. I found out that the latest stix2 package (version 3.x) has some issues with the version of the attackcti library we are using. Anyways, the solution is to use a 2.x version of stix2. I updated the requirements.txt file and we built a new docker container. So please pull the latest docker container and that one should work.

cheers,
Ruben

DFIRmadness commented

Ruben,

Thanks! I am sure that was not easy to trace down. I downloaded the latest docker and confirmed it is now working as expected! Thanks!

DFIRmadness commented

All,

I was doing some further testing. I was trying to get overlays etc.

I removed all docker images. Re-installed on (Evening east coast time of the 20th). The following same bug occurred.

root@a608b6b9230c:/opt/DeTTECT# ls -l
total 408
-rw-rw-rw- 1 root root   597 Oct 20 11:28 Dockerfile
-rw-rw-rw- 1 root root 35149 Oct 20 11:28 LICENSE
-rw-rw-rw- 1 root root  4124 Oct 20 11:28 README.md
drwxr-xr-x 2 root root  4096 Oct 20 23:17 __pycache__
drwxr-xr-x 2 root root  4096 Oct 20 23:34 cache
-rw-rw-rw- 1 root root 18580 Oct 20 11:28 constants.py
-rw-rw-rw- 1 root root 29869 Oct 20 11:28 data_source_mapping.py
-rw-rw-rw- 1 root root 25881 Oct 20 11:28 dettect.py
drwxrwxrwx 6 root root  4096 Oct 20 11:28 editor
-rw-rw-rw- 1 root root  2169 Oct 20 11:28 editor.py
-rw-rw-rw- 1 root root 17553 Oct 20 11:28 eql_yaml.py
-rw-rw-rw- 1 root root 52768 Oct 20 11:28 generic.py
-rw-rw-rw- 1 root root 29665 Oct 20 11:28 group_mapping.py
-rw-rw-rw- 1 root root 19014 Oct 20 11:28 health.py
drwxr-xr-x 2 root root  4096 Oct 20 20:23 input
-rw-rw-rw- 1 root root 21754 Oct 20 11:28 interactive_menu.py
drwxrwxrwx 2 root root  4096 Oct 20 11:28 mitre-data
drwxr-xr-x 2 root root  4096 Oct 20 23:26 output
-rw-rw-rw- 1 root root   148 Oct 20 11:28 requirements.txt
drwxrwxrwx 2 root root  4096 Oct 20 11:28 sample-data
-rw-rw-rw- 1 root root 29726 Oct 20 11:28 scoring_table.xlsx
-rw-rw-rw- 1 root root 26977 Oct 20 11:28 technique_mapping.py
drwxrwxrwx 4 root root  4096 Oct 20 11:28 threat-actor-data
-rw-rw-rw- 1 root root 29050 Oct 20 11:28 upgrade.py
root@a608b6b9230c:/opt/DeTTECT# python dettect.py g -g G0025,G0026,G0073,G0045,G0116,G0027,G0016,G0096,G0117,G0083,G0065 -o sample-data/techniques-administration-endpoints.yaml -t visibility -of Test
[!] Group not part of the data set: g0025
root@a608b6b9230c:/opt/DeTTECT# rm cache/*
root@a608b6b9230c:/opt/DeTTECT# python dettect.py g -g G0025,G0026,G0073,G0045,G0116,G0027,G0016,G0096,G0117,G0083,G0065 -o sample-data/techniques-administration-endpoints.yaml -t visibility -of Test
[!] Group not part of the data set: g0025
root@a608b6b9230c:/opt/DeTTECT# python dettect.py g -g G0025 -o sample-data/techniques-administration-endpoints.yaml -t visibility -of Test
[!] Group not part of the data set: g0025
root@a608b6b9230c:/opt/DeTTECT# python dettect.py g -g g0025 -o sample-data/techniques-administration-endpoints.yaml -t visibility -of Test
[!] Group not part of the data set: g0025
root@a608b6b9230c:/opt/DeTTECT# python dettect.py g -g g0025
[!] Group not part of the data set: g0025
root@a608b6b9230c:/opt/DeTTECT# python dettect.py g -g g0026
File written:   output/attack_windows_g0026.json
root@a608b6b9230c:/opt/DeTTECT#

I tried on another computer all together and the bug was not present. I am very confused -any help would be appreciated. Thanks.

rubinatorz commented

hi @DFIRmadness,

Same error, different cause this time :-)

G0025/APT7 is a group that only has "techniques used" in the PRE platform:
https://attack.mitre.org/groups/G0025/

For the group mode of DeTT&CT, the default value for platform is Windows (check dettect.py g -h):

-p {all,PRE,Windows,macOS,Linux,Office 365,Azure AD,Google Workspace,IaaS,SaaS,Network,Containers}, --platform {all,PRE,Windows,macOS,Linux,Office 365,Azure AD,Google Workspace,IaaS,SaaS,Network,Containers}
                        specify the platform (default = Windows). Multiple platforms can be provided with extra '-p/--platform' arguments

And that results in G0025/APT17 not being recognised. So to resolve your issue, add "-p all" to your command and it will use all platforms.

rubinatorz commented

While thinking of this default behaviour and discussing it with Marcus... We are going to change the default platform for groups mode to "all". That makes much more sense.

Anyways, your quickfix is using "-p all" and soon we will make this the default.

rubinatorz commented

While thinking of this default behaviour and discussing it with Marcus... We are going to change the default platform for groups mode to "all". That makes much more sense.

Anyways, your quickfix is using "-p all" and soon we will make this the default.

'all' is default since recent commit ee0decb. Closing the issue, since both issues discussed are solved.