AIL Framework not starting v4.2 (same like #442)
lobojc opened this issue · 5 comments
lobojc commented
Hello AIL Framework team, I have the same problem running AIL Framework, the Website doesn't start.
I installed pybgpranking manually because it failed and I updated jsonschema to 4.9.1
ERROR: pymisp 2.4.160.1 has requirement jsonschema<5.0.0,>=4.9.1, but you'll have jsonschema 3.2.0 which is incompatible.
After check the screens I detected:
- Flask_AIL ERROR:
Flask_base_template.py create_default_user.py server.key
Flask_server.py create_new_web_module.py static
__pycache__ modules templates
blueprints server.crt update_thirdparty.sh
Traceback (most recent call last):
File "./Flask_server.py", line 24, in <module>
from pytaxonomies import Taxonomies
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/pytaxonomies/__init__.py", line 1, in <module>
from .api import Taxonomies, Taxonomy, Predicate, Entry
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/pytaxonomies/api.py", line 13, in <module>
import requests
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/requests/__init__.py", line 147, in <module>
from . import packages, utils
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 844, in exec_module
File "<frozen importlib._bootstrap_external>", line 980, in get_code
File "<frozen importlib._bootstrap_external>", line 1037, in get_data
PermissionError: [Errno 1] Operation not permitted: '/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/requests/utils.py'
- Script_AIL ERROR:
Flask_base_template.py create_default_user.py server.key
Flask_server.py create_new_web_module.py static
__pycache__ modules templates
blueprints server.crt update_thirdparty.sh
Traceback (most recent call last):
File "./Flask_server.py", line 24, in <module>
from pytaxonomies import Taxonomies
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/pytaxonomies/__init__.py", line 1, in <module>
from .api import Taxonomies, Taxonomy, Predicate, Entry
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/pytaxonomies/api.py", line 13, in <module>
import requests
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/requests/__init__.py", line 147, in <module>
from . import packages, utils
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 844, in exec_module
File "<frozen importlib._bootstrap_external>", line 980, in get_code
File "<frozen importlib._bootstrap_external>", line 1037, in get_data
PermissionError: [Errno 1] Operation not permitted: '/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/zmq/sugar/poll.py'
- Redis_AIL info:
52383:M 10 Aug 2022 22:27:17.193 # Server initialized
52383:M 10 Aug 2022 22:27:17.193 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
52383:M 10 Aug 2022 22:27:17.193 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
52383:M 10 Aug 2022 22:27:17.193 * Ready to accept connections
- Other screen without error messages !
PS: AIL Starting with ./LAUNCH.sh -l output without ERRORS:
* Launching Redis servers
* Launching ARDB servers
* Launching logging process
* Launching all the queues
* Checking configuration
Config File: Nothing to update
Config File: Nothing to update
* Configuration up-to-date
* Launching core scripts ...
* Launching scripts
* Launching Flask server
Thank you
lobojc commented
BTW the errors change every time when I run LAUNCH -k and LAUNCH -l
- SOME STOP + START ->
Flask_base_template.py create_new_web_module.py static
Flask_server.py modules templates
blueprints server.crt update_thirdparty.sh
create_default_user.py server.key
Traceback (most recent call last):
File "./Flask_server.py", line 16, in <module>
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/flask/__init__.py", line 4, in <module>
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 844, in exec_module
File "<frozen importlib._bootstrap_external>", line 980, in get_code
File "<frozen importlib._bootstrap_external>", line 1037, in get_data
PermissionError: [Errno 1] Operation not permitted: '/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/flask/json/__init__.py'
- Another STOP + START ->
Flask_base_template.py create_new_web_module.py static
Flask_server.py modules templates
blueprints server.crt update_thirdparty.sh
create_default_user.py server.key
Fatal Python error: init_import_size: Failed to import the site module
Python runtime state: initialized
Traceback (most recent call last):
File "/usr/lib/python3.8/site.py", line 597, in <module>
main()
File "/usr/lib/python3.8/site.py", line 580, in main
known_paths = venv(known_paths)
File "/usr/lib/python3.8/site.py", line 498, in venv
with open(virtual_conf, encoding='utf-8') as f:
PermissionError: [Errno 1] Operation not permitted: '/home/ubuntu/ailframework/AILENV/pyvenv.cfg'
lobojc commented
And to finish ./LAUNCH -t command output:
ubuntu@ailframework:~/ailframework/bin$ ./LAUNCH.sh -t
EEE
======================================================================
ERROR: Failure: PermissionError ([Errno 1] Operation not permitted: '/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/pymispgalaxies/data/misp-galaxy/clusters/tea-matrix.json')
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/failure.py", line 39, in runTest
raise self.exc_val.with_traceback(self.tb)
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/loader.py", line 417, in loadTestsFromName
module = self.importer.importFromPath(
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/importer.py", line 47, in importFromPath
return self.importFromDir(dir_path, fqname)
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/importer.py", line 94, in importFromDir
mod = load_module(part_fqname, fh, filename, desc)
File "/usr/lib/python3.8/imp.py", line 234, in load_module
return load_source(name, filename, file)
File "/usr/lib/python3.8/imp.py", line 171, in load_source
module = _load(spec)
File "<frozen importlib._bootstrap>", line 702, in _load
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 848, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/ubuntu/ailframework/tests/testApi.py", line 17, in <module>
from Flask_server import app
File "/home/ubuntu/ailframework/var/www/Flask_server.py", line 160, in <module>
importlib.import_module(name)
File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/home/ubuntu/ailframework/var/www/modules/Tags/Flask_Tags.py", line 39, in <module>
clusters = Clusters(skip_duplicates=True)
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/pymispgalaxies/api.py", line 294, in __init__
with open(cluster_file, 'r') as f:
PermissionError: [Errno 1] Operation not permitted: '/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/pymispgalaxies/data/misp-galaxy/clusters/tea-matrix.json'
-------------------- >> begin captured logging << --------------------
pymisp: DEBUG: pymisp loaded properly
--------------------- >> end captured logging << ---------------------
======================================================================
ERROR: Failure: ImportError (libzmq-68c212d3.so.5.2.4: cannot open shared object file: No such file or directory)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/failure.py", line 39, in runTest
raise self.exc_val.with_traceback(self.tb)
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/loader.py", line 417, in loadTestsFromName
module = self.importer.importFromPath(
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/importer.py", line 47, in importFromPath
return self.importFromDir(dir_path, fqname)
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/importer.py", line 94, in importFromDir
mod = load_module(part_fqname, fh, filename, desc)
File "/usr/lib/python3.8/imp.py", line 234, in load_module
return load_source(name, filename, file)
File "/usr/lib/python3.8/imp.py", line 171, in load_source
module = _load(spec)
File "<frozen importlib._bootstrap>", line 702, in _load
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 848, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/ubuntu/ailframework/tests/testHelper.py", line 9, in <module>
from Helper import Process
File "/home/ubuntu/ailframework/bin/Helper.py", line 17, in <module>
import zmq
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/zmq/__init__.py", line 103, in <module>
from zmq import backend
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/zmq/backend/__init__.py", line 31, in <module>
raise original_error from None
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/zmq/backend/__init__.py", line 26, in <module>
_ns = select_backend(first)
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/zmq/backend/select.py", line 31, in select_backend
mod = import_module(name)
File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/zmq/backend/cython/__init__.py", line 6, in <module>
from . import (
ImportError: libzmq-68c212d3.so.5.2.4: cannot open shared object file: No such file or directory
======================================================================
ERROR: Failure: ImportError (libzmq-68c212d3.so.5.2.4: cannot open shared object file: No such file or directory)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/failure.py", line 39, in runTest
raise self.exc_val.with_traceback(self.tb)
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/loader.py", line 417, in loadTestsFromName
module = self.importer.importFromPath(
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/importer.py", line 47, in importFromPath
return self.importFromDir(dir_path, fqname)
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/nose/importer.py", line 94, in importFromDir
mod = load_module(part_fqname, fh, filename, desc)
File "/usr/lib/python3.8/imp.py", line 234, in load_module
return load_source(name, filename, file)
File "/usr/lib/python3.8/imp.py", line 171, in load_source
module = _load(spec)
File "<frozen importlib._bootstrap>", line 702, in _load
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 848, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/ubuntu/ailframework/tests/test_modules.py", line 15, in <module>
from modules.ApiKey import ApiKey
File "/home/ubuntu/ailframework/bin/modules/ApiKey.py", line 21, in <module>
from modules.abstract_module import AbstractModule
File "/home/ubuntu/ailframework/bin/modules/abstract_module.py", line 17, in <module>
from Helper import Process
File "/home/ubuntu/ailframework/bin/Helper.py", line 17, in <module>
import zmq
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/zmq/__init__.py", line 103, in <module>
from zmq import backend
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/zmq/backend/__init__.py", line 31, in <module>
raise original_error from None
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/zmq/backend/__init__.py", line 26, in <module>
_ns = select_backend(first)
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/zmq/backend/select.py", line 31, in select_backend
mod = import_module(name)
File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/home/ubuntu/ailframework/AILENV/lib/python3.8/site-packages/zmq/backend/cython/__init__.py", line 6, in <module>
from . import (
ImportError: libzmq-68c212d3.so.5.2.4: cannot open shared object file: No such file or directory
Name Stmts Miss Cover
----------------------------------------------------------
bin/BankAccount.py 72 72 0%
bin/Crawler.py 281 281 0%
bin/Cryptocurrencies.py 78 78 0%
bin/Cve.py 26 26 0%
bin/DumpValidOnion.py 23 23 0%
bin/Duplicates.py 116 116 0%
bin/Helper.py 158 153 3%
bin/IPAddress.py 50 50 0%
bin/Languages.py 21 21 0%
bin/MISP_The_Hive_feeder.py 156 156 0%
bin/Mail.py 135 135 0%
bin/Mixer.py 114 114 0%
bin/ModuleStats.py 80 80 0%
bin/ModulesInformationV2.py 549 549 0%
bin/NotificationHelper.py 52 52 0%
bin/PgpDump.py 164 164 0%
bin/QueueIn.py 13 13 0%
bin/QueueOut.py 14 14 0%
bin/Release.py 46 46 0%
bin/SourceCode.py 39 39 0%
bin/Tools.py 62 62 0%
bin/Update-conf.py 59 59 0%
bin/Update.py 329 329 0%
bin/__init__.py 0 0 100%
bin/ailleakObject.py 90 90 0%
bin/core/ail_2_ail.py 852 697 18%
bin/core/screen.py 122 105 14%
bin/empty_queue.py 18 18 0%
bin/export/AILObjects.py 39 22 44%
bin/export/Export.py 49 26 47%
bin/export/MispExport.py 331 276 17%
bin/export/MispImport.py 172 140 19%
bin/import/importer.py 61 34 44%
bin/indexer_lookup.py 60 60 0%
bin/launch_queues.py 49 49 0%
bin/lib/ConfigLoader.py 41 11 73%
bin/lib/Correlate_object.py 371 327 12%
bin/lib/Decoded.py 260 210 19%
bin/lib/Domain.py 573 463 19%
bin/lib/Investigations.py 294 219 26%
bin/lib/Language.py 27 18 33%
bin/lib/Screenshot.py 161 120 25%
bin/lib/Tracker.py 944 799 15%
bin/lib/Username.py 13 1 92%
bin/lib/btc_ail.py 40 31 22%
bin/lib/crawlers.py 1009 833 17%
bin/lib/d4.py 58 39 33%
bin/lib/domain_basic.py 16 5 69%
bin/lib/exceptions.py 8 2 75%
bin/lib/item_basic.py 161 117 27%
bin/lib/objects/CryptoCurrencies.py 34 19 44%
bin/lib/objects/Decodeds.py 40 20 50%
bin/lib/objects/Domains.py 33 17 48%
bin/lib/objects/Items.py 426 319 25%
bin/lib/objects/Pgps.py 30 15 50%
bin/lib/objects/Screenshots.py 24 8 67%
bin/lib/objects/Usernames.py 31 16 48%
bin/lib/objects/abstract_object.py 57 31 46%
bin/lib/objects/ail_objects.py 54 27 50%
bin/lib/queues_modules.py 53 34 36%
bin/modules/ApiKey.py 50 44 12%
bin/modules/Categ.py 49 49 0%
bin/modules/Credential.py 111 111 0%
bin/modules/CreditCards.py 40 40 0%
bin/modules/Decoder.py 99 99 0%
bin/modules/DomClassifier.py 51 51 0%
bin/modules/Global.py 122 122 0%
bin/modules/Hosts.py 31 31 0%
bin/modules/Indexer.py 89 89 0%
bin/modules/Keys.py 116 116 0%
bin/modules/LibInjection.py 57 57 0%
bin/modules/Onion.py 61 61 0%
bin/modules/Phone.py 40 40 0%
bin/modules/SQLInjectionDetection.py 48 48 0%
bin/modules/SentimentAnalysis.py 98 98 0%
bin/modules/Tags.py 26 26 0%
bin/modules/Telegram.py 53 53 0%
bin/modules/Urls.py 40 40 0%
bin/modules/Zerobins.py 30 30 0%
bin/modules/__init__.py 0 0 100%
bin/modules/abstract_module.py 52 46 12%
bin/modules/submit_paste.py 189 189 0%
bin/packages/Correlation.py 266 213 20%
bin/packages/Cryptocurrency.py 68 49 28%
bin/packages/Date.py 127 101 20%
bin/packages/Hash.py 30 19 37%
bin/packages/HiddenServices.py 231 190 18%
bin/packages/Import_helper.py 56 43 23%
bin/packages/Item.py 401 298 26%
bin/packages/Paste.py 195 146 25%
bin/packages/Pgp.py 17 5 71%
bin/packages/Tag.py 483 396 18%
bin/packages/Term.py 392 325 17%
bin/packages/User.py 40 24 40%
bin/packages/__init__.py 0 0 100%
bin/packages/git_status.py 110 95 14%
bin/packages/lib_refine.py 94 94 0%
bin/packages/lib_words.py 51 51 0%
bin/template.py 18 18 0%
bin/tor_fetcher.py 46 46 0%
bin/update-background.py 44 44 0%
----------------------------------------------------------
TOTAL 13359 11447 14%
----------------------------------------------------------------------
Ran 3 tests in 0.002s
FAILED (errors=3)
Thank you
lobojc commented
And I have had to comment the next lines 126 to 130 of installing_deps.sh file to run the script successfully: git fetch failed -> git@github.com: Permission denied (publickey)
git fetch --depth=500 --tags --prune
if [ ! -z "$TRAVIS" ]; then
echo "Travis detected"
git fetch --unshallow
fi
lobojc commented
Other error details in the installating_steps.sh step:
Successfully built pymispgalaxies
Installing collected packages: jsonschema, pymispgalaxies
Attempting uninstall: jsonschema
Found existing installation: jsonschema 4.9.1
Uninstalling jsonschema-4.9.1:
Successfully uninstalled jsonschema-4.9.1
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
pymisp 2.4.160.1 requires jsonschema<5.0.0,>=4.9.1, but you have jsonschema 3.2.0 which is incompatible.
Successfully installed jsonschema-3.2.0 pymispgalaxies-0.2
And later ->
Installing collected packages: jsonschema
Attempting uninstall: jsonschema
Found existing installation: jsonschema 3.2.0
Uninstalling jsonschema-3.2.0:
Successfully uninstalled jsonschema-3.2.0
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
pymispgalaxies 0.2 requires jsonschema<4.0,>=3.2, but you have jsonschema 4.9.1 which is incompatible.
Successfully installed jsonschema-4.9.1
Requirement already satisfied: thehive4py in /home/ubuntu/ailframework/AILENV/lib/python3.9/site-packages (1.8.1)
Requirement already satisfied: python-magic in /home/ubuntu/ailframework/AILENV/lib/python3.9/site-packages (from thehive4py) (0.4.27)
It looks that some package require one jsonschema version and another package requires a different version
lobojc commented
FYI, the same results using python 3.8 and python 3.9 with Ubuntu 20.04 OS