Tests fail after installation
arfe1012 opened this issue · 2 comments
Hey I've installed AIL and everything seems to be running fine. However, when I run the tests with ./LAUNCH.sh -t I get the following result. Is this a bug inside the test module or is there an issue with the installation?
copying /home/xxx/ail-framework/samples/2021/01/01/keys_certificat_sample.gz -> /home/xxx/ail-framework/PASTES/tests/2021/01/01
copying /home/xxx/ail-framework/samples/2021/01/01/onion.gz -> /home/xxx/ail-framework/PASTES/tests/2021/01/01
copying /home/xxx/ail-framework/samples/2021/01/01/keys.gz -> /home/xxx/ail-framework/PASTES/tests/2021/01/01
copying /home/xxx/ail-framework/samples/2021/01/01/domain_classifier.gz -> /home/xxx/ail-framework/PASTES/tests/2021/01/01
copying /home/xxx/ail-framework/samples/2021/01/01/categ.gz -> /home/xxx/ail-framework/PASTES/tests/2021/01/01
copying /home/xxx/ail-framework/samples/2021/01/01/credit_cards.gz -> /home/xxx/ail-framework/PASTES/tests/2021/01/01
copying /home/xxx/ail-framework/samples/2021/01/01/api_keys.gz -> /home/xxx/ail-framework/PASTES/tests/2021/01/01
.Module ApiKey initialized
found google api key: ApiKey;tests;20210101;api_keys.gz;
found AWS key: ApiKey;tests;20210101;api_keys.gz;
found AWS secret key
.2 CreditCards
1 Mail
1 Onion
1 Urls
2 Credential
1 Cve
.Module CreditCards initialized
341039324930797
6011613905509166
5492981206527330
3547151714018657
4024007132849695
CreditCard;tests;20210101;credit_cards.gz;Checked 5 valid number(s);item::tests/2021/01/01/credit_cards.gz
.Module: DomClassifier Launched
['foo.be']
set()
FModule Global initialized
tests/2021/01/01/global.gz
File already exist /home/xxx/ail-framework/PASTES/tests/2021/01/01/global.gz
File already exist /home/xxx/ail-framework/PASTES/tests/2021/01/01/global.gz
ignore duplicated file /home/xxx/ail-framework/PASTES/tests/2021/01/01/global.gz
File already exist /home/xxx/ail-framework/PASTES/tests/2021/01/01/global.gz
File already exist /home/xxx/ail-framework/PASTES/tests/2021/01/01/global.gz
tests/2021/01/01/global_831875da824fc86ab5cc0e835755b520.gz
..Module: Onion Launched
http://eswpccgr5xyovsahffkehgleqthrasfpfdblwbs4lstd345dwq5qumqd.onion
invalid.onion
www.facebookcorewwwi.onion
Onion;tests;20210101;onion.gz;Detected 2 .onion(s);item::tests/2021/01/01/onion.gz
.Module Telegram initialized
.
FAIL: test_module (test_modules.TestModuleDomClassifier.test_module)
Traceback (most recent call last):
File "/home/xxx/ail-framework/tests/test_modules.py", line 105, in test_module
self.assertTrue(len(result))
AssertionError: 0 is not true
Ran 9 tests in 5.905s
FAILED (failures=1)
Name Stmts Miss Cover
bin/AIL_Init.py 15 15 0%
bin/Update-conf.py 58 58 0%
bin/Update.py 343 343 0%
bin/lib/ConfigLoader.py 63 12 81%
bin/lib/Duplicate.py 73 45 38%
bin/lib/Investigations.py 322 240 25%
bin/lib/Language.py 273 220 19%
bin/lib/Tag.py 1073 885 18%
bin/lib/Tracker.py 1321 1107 16%
bin/lib/Users.py 187 133 29%
bin/lib/ail_core.py 131 98 25%
bin/lib/ail_logger.py 20 1 95%
bin/lib/ail_queues.py 281 181 36%
bin/lib/btc_ail.py 45 35 22%
bin/lib/chats_viewer.py 452 374 17%
bin/lib/correlations_engine.py 113 71 37%
bin/lib/crawlers.py 1422 1116 22%
bin/lib/d4.py 57 39 32%
bin/lib/data_retention_engine.py 58 24 59%
bin/lib/exceptions.py 15 2 87%
bin/lib/item_basic.py 152 102 33%
bin/lib/objects/ChatSubChannels.py 64 35 45%
bin/lib/objects/ChatThreads.py 63 39 38%
bin/lib/objects/Chats.py 100 66 34%
bin/lib/objects/CookiesNames.py 69 43 38%
bin/lib/objects/CryptoCurrencies.py 139 109 22%
bin/lib/objects/Cves.py 108 76 30%
bin/lib/objects/Decodeds.py 441 360 18%
bin/lib/objects/Domains.py 542 450 17%
bin/lib/objects/Etags.py 69 43 38%
bin/lib/objects/Favicons.py 95 64 33%
bin/lib/objects/FilesNames.py 52 29 44%
bin/lib/objects/HHHashs.py 84 56 33%
bin/lib/objects/Images.py 105 69 34%
bin/lib/objects/Items.py 496 381 23%
bin/lib/objects/Messages.py 221 170 23%
bin/lib/objects/Ocrs.py 239 188 21%
bin/lib/objects/Pgps.py 99 74 25%
bin/lib/objects/Screenshots.py 108 75 31%
bin/lib/objects/Titles.py 65 40 38%
bin/lib/objects/Usernames.py 84 61 27%
bin/lib/objects/UsersAccount.py 140 96 31%
bin/lib/objects/abstract_chat_object.py 233 174 25%
bin/lib/objects/abstract_daterange_object.py 201 151 25%
bin/lib/objects/abstract_object.py 215 105 51%
bin/lib/objects/abstract_subtype_object.py 146 109 25%
bin/lib/objects/ail_objects.py 411 329 20%
bin/lib/regex_helper.py 162 122 25%
bin/lib/relationships_engine.py 68 51 25%
bin/lib/telegram.py 53 39 26%
bin/lib/timeline_engine.py 114 93 18%
bin/modules/ApiKey.py 47 2 96%
bin/modules/Categ.py 49 6 88%
bin/modules/Credential.py 75 75 0%
bin/modules/CreditCards.py 54 10 81%
bin/modules/Cryptocurrencies.py 48 48 0%
bin/modules/CveModule.py 32 32 0%
bin/modules/Decoder.py 85 85 0%
bin/modules/DomClassifier.py 56 16 71%
bin/modules/Duplicates.py 51 51 0%
bin/modules/Exif.py 32 32 0%
bin/modules/Global.py 119 34 71%
bin/modules/Hosts.py 32 32 0%
bin/modules/IPAddress.py 53 53 0%
bin/modules/Iban.py 63 63 0%
bin/modules/Indexer.py 91 91 0%
bin/modules/Keys.py 99 45 55%
bin/modules/Languages.py 21 21 0%
bin/modules/LibInjection.py 44 44 0%
bin/modules/MISP_Thehive_Auto_Push.py 46 46 0%
bin/modules/Mail.py 110 110 0%
bin/modules/Mixer.py 90 90 0%
bin/modules/ModuleStats.py 24 24 0%
bin/modules/OcrExtractor.py 81 81 0%
bin/modules/Onion.py 62 15 76%
bin/modules/Pasties.py 93 93 0%
bin/modules/PgpDump.py 167 167 0%
bin/modules/Phone.py 30 30 0%
bin/modules/SQLInjectionDetection.py 36 36 0%
bin/modules/SentimentAnalysis.py 107 107 0%
bin/modules/SubmitPaste.py 186 186 0%
bin/modules/Tags.py 19 19 0%
bin/modules/Telegram.py 61 29 52%
bin/modules/TemplateModule.py 15 15 0%
bin/modules/Tools.py 35 35 0%
bin/modules/Urls.py 42 42 0%
bin/modules/init.py 0 0 100%
bin/modules/abstract_module.py 111 50 55%
bin/packages/Date.py 202 165 18%
bin/packages/Hash.py 30 30 0%
bin/packages/Import_helper.py 55 55 0%
bin/packages/init.py 0 0 100%
bin/packages/git_status.py 132 115 13%
bin/packages/lib_refine.py 96 77 20%
bin/packages/lib_words.py 51 51 0%
bin/update-background.py 50 50 0%
TOTAL 14442 11256 22%
Hi @arfe1012 !
It seem like it was an issue with your DNS server configuration.
Let me know if you have any issue with the DomClassifier module
Hello @Terrtia!
AIL was setup in an environment, where a proxy is needed to connect to the internet. Everything seemed fine with the crawling and the analysis of the crawling-data. You're right though! the DomClassifier was a huuge bottleneck, so it might not be working correctly. I set the proxy globally and everything else worked perfectly with that. Would I need to set the proxy manually somewhere else for the DomClassifier module? I didn't look deeper into the problem but just noticed that the processing of a single item in the queue took around 5 seconds for the DomClassifier.
Maybe this information helps you. Thanks for your great work!