Even if this project's name is one of the worst ever, Siggregator could be very useful.
In malware analysis, we often have to categorize many binary samples based on static signatures.
In my humble experience, I noticed that the best signatures are those of:
-
The evergreen libmagic library
However, the approaches and use cases are totally different.
First of all, libmagic
just recognizes file types based on magic numbers, so no packers, compilers, installers, protectors, etc.
Detect-It-Easy (DIE) uses finely tuned signatures bases on a dedicated scripting language (and therefore a dedicated engine),
while Retdec provides well-maintained Yara signatures.
DIE is very accurate but slow, and its database lacks some rules, while Yara is very fast and has a vast database but prone to false positives.
Moreover, they use different technologies, which are sometimes tedious to configure on different operating systems.
Siggregator was born to be used as a Docker container à la build&run plug&play minimum-effort&maximum-result, to compare the output of the tools mentioned above.
git clone https://github.com/packmad/Siggregator.git
cd Siggregator
docker build . -t packmad:siggregator
./siggregator.sh IN_DIR OUT_FILE
IN_DIR
- is the input directory that contains the files to be analyzedOUT_FILE
- will be a json file containing the output of those tools
In order to minimize false positives, and given the fact that Retdec's signatures are organized into fileformat/architecture:
- Fast check by inspecting the first bytes of the files (MZ, .ELF, ...), non-executable files are not analysed
- Precise file type and architecture identification from libmagic
- Scan with the proper Yara rules for such file type/architecture
- DIE scan
- Output pre-processing
Of course! Have a look at results_to_csv.py, it has been written in pure Python3 (i.e., no dependencies), you do not need Docker BUT be careful: the script is mixing DIE with Yara, removing versions, etc. it is likely that you need to fix the code for your use case.
Usage:
./results_to_csv.py input.json output.csv
In the same way, if you want to analyze a folder containing multiple directories with binaries, you can use multi_dir_csv.sh:
./multi_dir_csv.sh IN_DIR OUT_DIR
This command will launch siggreagator in all the directories containted in IN_DIR and will save the JSON files in OUT_DIR with the format folder_name.json and then all those files will be converted in the csv format by using results_to_csv.py.
Moreover, if a cell has multiple values, they are joined in alphabetical order using the semicolon as a separator, for example:
# cat example.csv
SHA256,FILE_FORMAT,ARCH_BITS,ENDIANESS,COMPILER,LINKER,LIBRARY,PACKER/PROTECTOR,INSTALLER,SFX/ARCHIVE,OVERLAY,OTHER
verylongsha256,pe,32,LE,aut2exe;msvc,microsoft linker,autoit,,,,,
It means that the detected compilers are aut2exe
and msvc
.
- 37bea5b0a24fa6fed0b1649189a998a0e51650dd640531fe78b6db6a196917a7
- d7e1d13cab1bd8be1f00afbec993176cc116c2b233209ea6bd33e6a9b1ec7a7f
# ./siggregator.sh /tmp/twoviruses /tmp/twoviruses.json
> Scanning input directory...
> Found 2 files. Analysis in progress...
100%|██████████| 2/2 [00:01<00:00, 32.76it/s]
> Analysis done!
> Found 2 executable files. Writing output file...
> "out.json" written. Bye!
# cat /tmp/out.json | jq
[
{
"sha256":"37bea5b0a24fa6fed0b1649189a998a0e51650dd640531fe78b6db6a196917a7",
"magic":"PE32 executable (GUI) Intel 80386, for MS Windows",
"format":"pe",
"arch":"x86",
"die":{
"arch":"I386",
"detects":[
{
"name":"VMProtect",
"options":null,
"type":"protector",
"version":null
},
{
"name":"Microsoft Visual C/C++",
"options":null,
"type":"compiler",
"version":"2017 v.15.6"
},
{
"name":"Microsoft Linker",
"options":"GUI32",
"type":"linker",
"version":"14.13, Visual Studio 2017 15.6*"
}
],
"endianess":"LE",
"filetype":"PE32",
"mode":"32",
"type":"GUI"
},
"yara":[
{
"type":"packer",
"rule":"vmprotect_2x_xx",
"name":"VMProtect",
"version":"2.04+"
}
]
},
{
"sha256":"d7e1d13cab1bd8be1f00afbec993176cc116c2b233209ea6bd33e6a9b1ec7a7f",
"magic":"PE32 executable (GUI) Intel 80386, for MS Windows",
"format":"pe",
"arch":"x86",
"die":{
"arch":"I386",
"detects":[
{
"name":"WinRAR",
"options":null,
"type":"sfx",
"version":null
},
{
"name":"Microsoft Visual C/C++",
"options":null,
"type":"compiler",
"version":"2015 v.14.0"
},
{
"name":"Microsoft Linker",
"options":"GUI32,admin",
"type":"linker",
"version":"14.0, Visual Studio 2015 14.0*"
},
{
"name":"RAR archive",
"options":null,
"type":"overlay",
"version":null
},
{
"name":"RAR",
"options":"57.2%,2 files",
"type":"archive",
"version":"4"
}
],
"endianess":"LE",
"filetype":"PE32",
"mode":"32",
"type":"GUI"
},
"yara":[
{
"type":"installer",
"rule":"winrar_sfx_540",
"name":"WinRAR SFX",
"version":"5.40"
}
]
}
]
# ./results_to_csv.py /tmp/twoviruses.json /tmp/twoviruses.csv
> Input json file contains 2 elements
> "/tmp/twoviruses.csv" written. Bye!
# cat /tmp/twoviruses.csv
SHA256,FILE_FORMAT,ARCH_BITS,ENDIANESS,COMPILER,LINKER,LIBRARY,PACKER/PROTECTOR,INSTALLER,SFX/ARCHIVE,OVERLAY,OTHER
37...a7,pe,32,LE,msvc,microsoft linker,,vmprotect,,,,
d7...7f,pe,32,LE,msvc,microsoft linker,,,winrar sfx,rar,rar archive,