Wiggle
An executable binary metadata search engine. Currently MachO only.
Prerequisite
- docker and docker-compose
- Python3 (no support for Python2)
- radare2 (on collector)
Usage
-
Run the collector
Install dependencies first:
pipenv install
Note: docker is not required to run the collector.
python3 scan.py /System/Library/Frameworks /System/Library/PrivateFrameworks /usr/lib /usr/bin /usr/sbin /sbin /usr/libexec
Or specifying a file that includes the directories you want to scan
python3 scan.py --rule presets/app.txt
For mounted filesystem, you can set the filesystem root by specifying
--sysroot.To skip some the directories, use
--block block_list.txt(wildcard not supported yet) -
Start the server
docker-compose up -d
-
Migrate the data to the full text search
# todo: docker mount read only docker-compose run web python /agent/indexer.py 10.14.2 /agent/archive.db -
Open localhost:8000 in the browser
FAQ
Why doesn't the collector directly save the document, but using a SQLite database pipeline instead?
So the collector and the server don't have to be the same machine.
TODO
- Backend
- Support other formats: ELF, PE etc
- Support dyld_shared_cache
- Consider moving to postgres
- UI
- Autocomplete
- AJAX
Known Issue
Stability
The collector is extermely unstable now, the scanning process can be interrupted any time
Nested documents limitation
The number of nested documents has exceeded the allowed limit of [10000]. This limit can be set by changing the [index.mapping.nested_objects.limit] index level setting.
Need to improve the database design the overcome this instead of just changing the value
Numeric value -1
Numeric value (12727174745972277247) out of range of long (-9223372036854775808 - 9223372036854775807)
Some of the values from radare2 are 0xffffffffffffffff that cause the exception
Performance
Single node ElasticSearch sucks. Really. Should consider migrating to postgreSQL.
Acknowledgement
Wiggle is based on these awesome open source projects: