metadata page checksum error on repo.db
secureray opened this issue · 12 comments
Under Ubuntu 18.04, after installing mongodb, Pyton 2.7, the Robot Framework, nodejs, Mermaid CLI, setting up a virtualenv for the Python install, running "source venv/bin/activate", and successfully running "pip install ThreatPlaybook", attempting to run:
threat-playbook TestProject1
gives:
BDB0210 /home/raya/ThreatPlaybook/venv/local/lib/python2.7/site-packages/threat_playbook/repo.db: metadata page checksum error
Traceback (most recent call last):
File "/home/raya/ThreatPlaybook/venv/bin/threat-playbook", line 7, in
from threat_playbook import main
File "/home/raya/ThreatPlaybook/venv/local/lib/python2.7/site-packages/threat_playbook/init.py", line 39, in
rdb = shelve.open(os.path.join(module_path, "repo"))
File "/usr/lib/python2.7/shelve.py", line 243, in open
return DbfilenameShelf(filename, flag, protocol, writeback)
File "/usr/lib/python2.7/shelve.py", line 227, in init
Shelf.init(self, anydbm.open(filename, flag), protocol, writeback)
File "/usr/lib/python2.7/anydbm.py", line 85, in open
return mod.open(file, flag, mode)
File "/usr/lib/python2.7/dbhash.py", line 18, in open
return bsddb.hashopen(file, flag, mode)
File "/usr/lib/python2.7/bsddb/init.py", line 364, in hashopen
d.open(file, db.DB_HASH, flags, mode)
bsddb.db.DBError: (21, 'Is a directory')
type threat-playbook
threat-playbook is hashed (/home/raya/ThreatPlaybook/venv/bin/threat-playbook)
Any suggestions on what the problem might be?
The command that you are using to initialize a project is the old version.
You need to use the following command to do it: threat-playbook new-project <ProjectName>
Can you please try that and let me know? Also please check if the version of ThreatPlaybook you are using is 1.2
I get the same metadata page checksum error message when I try:
threat-playbook new-project TestProject1
from my /home/raya/ThreatPlaybook directory
(No new files are created in that directory).
Based on the tar file downloaded, I am using version 1.2 of threat-playbook.
When installing, I saw the following messages:
pip install ThreatPlaybook
Collecting ThreatPlaybook
Downloading https://files.pythonhosted.org/packages/54/02/9e3324b2fc512bfafa7b25458f8af8c68edd5aecc77c90779bb28a8ce89e/ThreatPlaybook-1.2.tar.gz (88kB)
100% |████████████████████████████████| 92kB 1.1MB/s
...
I just ran an install on a Mac in separate venv and it didnt throw any errors.
Is it possible that you may have some file system permission errors? Ref: https://stackoverflow.com/questions/39553264/got-gdbm-error-13-permission-denied-despite-posix-permissions-seeming-ok
It looks like shelve is looking for the repo.db file and is only finding a directory for some reason. I haven't come across this issue anywhere else. Let me check on an ubuntu machine and see if I can recreate that error.
Also, which version of python 2.7 are you using?
Python was installed using:
sudo apt install python-minimal
python --version
Python 2.7.15rc1
Python package management (pip) was installed with:
sudo apt-get install python-dev python-pip
pip --version
pip 9.0.1 from /usr/lib/python2.7/dist-packages (python 2.7)
When I run dbhash.py or whichdb.py manually, they seem to work ok.
pwd
/home/raya/ThreatPlaybook/venv/local/lib/python2.7/site-packages/threat_playbook
python /usr/lib/python2.7/dbhash.py repo.db
(venv) raya@raya-VirtualBox:/ThreatPlaybook/venv/local/lib/python2.7/site-packages/threat_playbook$ echo $?/ThreatPlaybook/venv/local/lib/python2.7/site-packages/threat_playbook$ python /usr/lib/python2.7/whichdb.py repo.db
0
(venv) raya@raya-VirtualBox:
bsddb185 repo.db
(venv) raya@raya-VirtualBox:~/ThreatPlaybook/venv/local/lib/python2.7/site-packages/threat_playbook$ echo $?
0
Created a new VM of Ubuntu 18.04 yesterday (11-7-2018) and re-installed just to double check things.
Still have same error.
Install steps and tests are in the attached file.
Posted a question to Stack Overflow to see if anyone had some ideas about why bsddb was giving this error.
Threat_Playbook_install-2.txt
I haven't been able to recreate this issue. I have tried with a Mac, Ubuntu 16, Xubuntu 16, Kali Linux and Windows. Its weird why its happening. This seems to have been an issue with Fedora and CentOS in the past
https://bugzilla.redhat.com/show_bug.cgi?id=1206791
https://tor.stackexchange.com/questions/4284/tbb-lost-my-stored-passwords
There's also a possibility that the bsddb library in your OS may be corrupt/outdated right?
You can also try (in a python REPL environment on your OS)
>>> import shelve
>>> db = shelve.open('test')
>>>Interestingly, as you suggested, if I use a simple python program to write a file, such as:
import shelve
s = shelve.open("mydata.dat")
s["name"]= ["Ray"]
s.close()
and run
python shelve_test_write.py
file mydata.dat
it shows:
mydata.dat: Berkeley DB (Hash, version 9, native byte-order)
The file is readable using:
import shelve
r = shelve.open("mydata.dat")
print r["name"]
r.close()
python shelve_test_read.py
['Ray']
--
Is the repo.db file created initially at run time, or is it something that is read from an existing file?
--
Using strings to look at the repo.db file shows there is probably a lot that you would want pre-defined in terms of tests, etc.
Perhaps what my Ubunt 18.04 system needs is to be able to access the db file using later BSD libraries.
apt list --installed | grep python | grep bsd
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
python-bsddb3/bionic,now 6.1.0-1build4 amd64 [installed]
However, a python file with the first line as:
import bsddb3
gives the message:
Traceback (most recent call last):
File "shelve_test_read_repo-1.py", line 1, in
import bsddb3
ImportError: No module named bsddb3
I have also tried:
pip install bsddb3
Collecting bsddb3
Downloading https://files.pythonhosted.org/packages/e9/fc/ebfbd4de236b493f9ece156f816c21df0ae87ccc22604c5f9b664efef1b9/bsddb3-6.2.6.tar.gz (239kB)
100% |████████████████████████████████| 245kB 3.6MB/s
Complete output from command python setup.py egg_info:
Can't find a local Berkeley DB installation.
(suggestion: try the --berkeley-db=/path/to/bsddb option)
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-gtHl64/bsddb3/
and have tried:
pip install bsddb3 --berkeley-db=/usr/lib/python2.7/bsddb
Usage:
pip install [options] [package-index-options] ...
pip install [options] -r [package-index-options] ...
pip install [options] [-e] ...
pip install [options] [-e] ...
pip install [options] <archive url/path> ...
no such option: --berkeley-db
@secureray you shouldn't have to do any of that. It's a simple shelve db that loads a python dict to memory.
The repo.db file has all of the "canned threat scenarios" that you can reuse to frame your threat model.
Ill probably change this to a simple JSON file and parse the dictionary in the next release.
Closing issue now, as I have not been able to recreate and no real solution seems to be available for the issue you are facing. I have tested this on Win, Ubuntu 16, 14, Mac, but am not able to recreate the issue.
@secureray latest release has tinyDB where I am using a simple JSON file. Test it out if you want and let me know please
Thanks Abhay!
I was able to set up a new python virtual environment on my Ubuntu 18.04 machine and install the new version without seeing any errors, as shown below (for those who might want to know).
pwd
/home/raya2/ThreatPlaybook
Deactivate my previous python envionrment
deactivate
Create a new Python virtual environment
virtualenv -p /usr/bin/python2.7 venv4
source venv4/bin/activate
venv4 prompt is displayed on subsequent prompts but omitted here for clarity
pip install ThreatPlaybook
type threat-playbook
threat-playbook is /home/raya2/ThreatPlaybook/venv4/bin/threat-playbook
try the install of the new version
threat-playbook new-project TestProject4
[+] Project: TestProject4 created successfully. You should have generated boilerplate code
echo $?
0