magic open (bat/bruteforcescan.py) vs magic from_file
Closed this issue · 11 comments
I am trying to run some of the tools in the bat suite, but I ran into an issue with the python-magic/libmagic library.
$ bat-scan
Traceback (most recent call last):
File "/usr/local/bin/bat-scan", line 16, in
import bat.bruteforcescan
File "/usr/local/lib/python2.7/dist-packages/bat/bruteforcescan.py", line 68, in
ms = magic.open(magic.MAGIC_NONE)
AttributeError: 'module' object has no attribute 'open'
the python-magic api specify from_file function to read a file instead of the open function, which is used in this case. What version of libmagic/python-magic is this tool dependent on?
thanks.
hi!
the python-magic api specify from_file function to read a file instead of
the open function, which is used in this case. What version of
libmagic/python-magic is this tool dependent on?
There appear to be different python-magic packages, unfortunately:
The one that I use is this one:
$ rpm -qi python-magic
Name : python-magic
Version : 5.22
Release : 6.fc23
Architecture: noarch
Install Date: Sun 01 May 2016 03:43:29 PM CEST
Group : Development/Libraries
Size : 17292
License : BSD
Signature : RSA/SHA256, Wed 06 Jan 2016 06:10:48 PM CET, Key ID
32474cf834ec9cba
Source RPM : file-5.22-6.fc23.src.rpm
Build Date : Wed 06 Jan 2016 08:28:09 AM CET
Build Host : arm04-builder06.arm.fedoraproject.org
Relocations : (not relocatable)
Packager : Fedora Project
Vendor : Fedora Project
URL : http://www.darwinsys.com/file/
Summary : Python 2 bindings for the libmagic API
Description :
This package contains the Python 2 bindings to allow access to the
libmagic API. The libmagic library is also used by the familiar
file(1) command.
I will put this on my TODO.
Hi,
where did you get magic-python from? I tried several python-magic sources, but none of them works for me.
On 11/11/16 10:53, maxi1192 wrote:
where did you get magic-python from? I tried several python-magic
sources, but none of them works for me.
On Fedora this is actually standard package:
$ rpm -qi python-magic
Name : python-magic
Version : 5.22
Release : 6.fc23
Architecture: noarch
Install Date: Fri 20 May 2016 02:07:23 CEST
Group : Development/Libraries
Size : 17292
License : BSD
Signature : RSA/SHA256, Wed 06 Jan 2016 18:10:48 CET, Key ID
32474cf834ec9cba
Source RPM : file-5.22-6.fc23.src.rpm
Build Date : Wed 06 Jan 2016 08:28:09 CET
Build Host : arm04-builder06.arm.fedoraproject.org
Relocations : (not relocatable)
Packager : Fedora Project
Vendor : Fedora Project
URL : http://www.darwinsys.com/file/
Summary : Python 2 bindings for the libmagic API
Description :
This package contains the Python 2 bindings to allow access to the
libmagic API. The libmagic library is also used by the familiar
file(1) command.
On Ubuntu the same package is used AFAIK.
Armijn Hemel, MSc
Tjaldur Software Governance Solutions
On 11/11/16 11:15, maxi1192 wrote:
actually it does not work with the python-magic you used on ubuntu. I
converted the .rpm package into .deb package and installed it, but it
still shows the error message pointed above.
Are you sure there isn't a python-magic from another source that is
being picked up? I tested on a stock Ubuntu and it simply works.
$ dpkg -s python-magic
Package: python-magic
Status: install ok installed
Priority: optional
Section: python
Installed-Size: 72
Maintainer: Ubuntu Developers ubuntu-devel-discuss@lists.ubuntu.com
Architecture: all
Source: file
Version: 1:5.14-2ubuntu3.3
Provides: python2.7-magic
Depends: python (>= 2.7), python (<< 2.8), python:any (>=
2.7.1-0ubuntu2), libmagic1 (= 1:5.14-2ubuntu3.3)
Description: File type determination library using "magic" numbers
(Python bindings)
This library can be used to classify files according to magic number
tests. It
implements the core functionality of the file command.
.
This package contains the Python bindings.
Homepage: http://www.darwinsys.com/file/
Original-Maintainer: Luk Claes luk@debian.org
Armijn Hemel, MSc
Tjaldur Software Governance Solutions
yes you are right, I installed a python-magic via pip in the beginning. This python-magic was preferred before every newly installed python-magics (installed via apt-get).
On 11/11/16 11:35, maxi1192 wrote:
yes you are right, I installed a python-magic via pip in the
beginning. This python-magic was preferred before every newly
installed python-magics.
I guess you can close this issue.
I will however put this one on my list for stricter checking. Thanks!
Armijn Hemel, MSc
Tjaldur Software Governance Solutions
hi!
This does not solve the problem. I installed the same magic-python as
you (Version: 1:5.14-2ubuntu3.3). But when I run bat-scan on a binary
file the tool executes fine. But when I want to examine the output
file I get the following error:file output file: compiled magic version [522] does not match with shared library magic version [514] I guess there is still a python-magic versioning issue, but I can't solve it.
I always stick to the version as supplied by the distribution, so I am
not sure what is causing this :-(
armijn
Armijn Hemel, MSc
Tjaldur Software Governance Solutions
Hi this usually works for me on Ubuntu 16.04:
apt-get install -y software-properties-common python-software-properties
mkdir -p /tmp/bat/packages/
wget -O /tmp/bat/packages/python-support.deb https://launchpad.net/ubuntu/+archive/primary/+files/python-support_1.0.15_all.debAnd then just apt install -y /tmp/bat/packages/python-support.deb.
I'm working with a colleague to try make BAT work, our biggest challenge is creating the database and making the bat-scan actually use this database properly as there seems to be a few issues with hardcoded paths in the setup scripts, workarounds between SQLITE and the POSTGRESQL databases ...
So we have prepared a Dockerized environment that allow us to workaround without any concerns. It's still a work in progress but I'm happy to share the Dockerfile, this will get you Ubuntu 16.04 with a "working" BAT 27 instance.
FROM ubuntu:16.04
MAINTAINER Artur Monteiro <invalid@example.com>
RUN apt-get update
RUN apt-get upgrade -y
RUN apt-get install -y software-properties-common python-software-properties
RUN apt-add-repository multiverse
RUN apt-get update
RUN apt-get install -y wget
RUN apt-get install -y git
RUN mkdir -p /tmp/bat/packages/
RUN wget -O /tmp/bat/packages/python-support.deb https://launchpad.net/ubuntu/+archive/primary/+files/python-support_1.0.15_all.deb
RUN wget -O /tmp/bat/packages/bat_all.deb http://www.binaryanalysis.org/download/ubuntu/bat_27.0_all.deb
RUN wget -O /tmp/bat/packages/bat_extratools.deb http://www.binaryanalysis.org/download/ubuntu/bat-extratools_27.0_amd64.deb
RUN wget -O /tmp/bat/packages/bat_extratools-java.deb http://www.binaryanalysis.org/download/ubuntu/bat-extratools-java_27.0_amd64.deb
RUN wget -O /tmp/bat/packages/bat-release.zip https://github.com/armijnhemel/binaryanalysis/archive/27.0.zip
RUN cd /tmp/bat/packages && apt install -y ./python-support.deb ./bat_all.deb ./bat_extratools.deb ./bat_extratools-java.deb
RUN wget -O /tmp/bat/packages/ninka.zip https://github.com/rybnik/ninka/archive/2.0-pre1.zip
RUN apt-get install -y postgresql postgresql-contrib
USER postgres
RUN /etc/init.d/postgresql start \
&& psql --command "CREATE USER bat WITH SUPERUSER PASSWORD 'bat';" \
&& createdb -O bat bat
ENTRYPOINT /bin/bashThis is still incomplete and the last part relating to Ninka and the database has not been properly tested... Anyway I hope this helps and I'm happy to make contributions to the BAT platform :)
On 13/11/16 11:13, Artur Monteiro wrote:
I'm working with a colleague to try make BAT work, our biggest
challenge is creating the database and making the bat-scan actually
use this database properly as there seems to be a few issues with
hardcoded paths in the setup scripts, workarounds between SQLITE and
the POSTGRESQL databases ...
In December I will be working on getting rid of all of the last sqlite bits.
By the way, you might want to switch to BAT 32, as that removes a few
dependencies (for example: Java)
armijn
Armijn Hemel, MSc
Tjaldur Software Governance Solutions
There are no .deb files for BAT 32 right? I need to clone it from github and install it manually right? Is there a release of .deb installation for BAT 32 planned?
Procedure: grab the source from GitHub and then roll your own packages according to the build documentation.