SySeVR_fix: A Python repository from kingnop

Environments

System

Ubuntu 18.04 (Tested)

Step 1

Joern 0.3.1
- JDK 1.7
- Neo4J 2.1.8 Community Edition
- Gremlin for Neo4J 2.X
- Apache Ant build tool
Python 2.7

Step 2

Python 3.6
Tensorflow 1.6
Gensim 3.4

Installation

Joern 0.3.1

JDK 1.7

extract the tarball

tar -xvf jdk-7u80-linux-x64.tar.gz -C /usr/loacl/java

set environment variable

/etc/profile

# Add These Content at the End of the File
#######################################
JAVA_HOME=/usr/local/java/jdk1.7.0_80
JRE_HOME=/usr/local/java/jdk1.7.0_80
PATH=$PATH:$JRE_HOME/bin:$JAVA_HOME/bin

export JAVA_HOME
export JRE_HOME
export PATH
#######################################

update-alternatives --install "/usr/bin/java" "java" "/usr/local/java/jdk1.7.0_80/bin/java" 1

update-alternatives --install "/usr/bin/javac" "javac" "/usr/local/java/jdk1.7.0_80/bin/javac" 1

update-alternatives --install "/usr/bin/javaws" "javaws" "/usr/local/java/jdk1.7.0_80/bin/javaws" 1

update-alternatives --set java /usr/local/java/jdk1.7.0_80/bin/java

update-alternatives --set javac /usr/local/java/jdk1.7.0_80/bin/javac

update-alternatives --set javaws /usr/local/java/jdk1.7.0_80/bin/javaws

source /etc/profile

verify

java -version

You should receive a message which displays

java version "1.7.0_80"
Java(TM) SE Runtime Environment (build 1.7.0_80-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)

Neo4j 2.1.8 Community Edition

extract the tarball

unzip neo4j-community-2.1.8.zip
mkdir -d /usr/local/neo4j

mv /usr/local/neo4j ./Neo4j/neo4j-community-2.1.8/*


2. modify configure files

> **configure files are located in /usr/local/neo4j/conf**

**neo4j-server.properties**

```bash
# location of the database directory
org.neo4j.server.database.location=/home/joern-0.3.1/.joernIndex/
# Let the webserver only listen on the specified IP. Default is localhost (only
# accept local connections). Uncomment to allow any connection. Please see the
# security section in the neo4j manual before modifying this.
org.neo4j.server.webserver.address=0.0.0.0

neo4j-wrapper.conf

# Java Heap Size: by default the Java heap size is dynamically
# calculated based on available system resources.
# Uncomment these lines to set specific initial and maximum
# heap size in MB.
wrapper.java.initmemory=512
wrapper.java.maxmemory=10240 #as large as you can

set environment variable

/etc/profile

# Add These Content at the End of the File
#######################################
NEO4J_HOME=/usr/local/neo4j
PATH=$PATH:$NEO4J_HOME/bin

export NEO4J_HOME
export PATH
#######################################
source /etc/profile

start && verify
```
/usr/local/neo4j/bin/neo4j console
```

Gremlin for Neo4J 2.X

https://github.com/neo4j-contrib/gremlin-plugin

unzip neo4j-gremlin-plugin-tp2-2.1.5-server-plugin.zip -d $NEO4J_HOME/plugins/gremlin-plugin

Apache Ant build tool

extract the tarball

mkdir /usr/local/ant
unzip -d /usr/local/ant apache-ant-1.8.4-bin.zip

set environment variable

/etc/profile

# Add These Content at the End of the File
#######################################
ANT_HOME=/usr/local/ant
PATH=$PATH:$ANT_HOME/bin

export ANT_HOME
export PATH
#######################################

source /etc/profile

verify
```
ant -version
```

Joern 0.3.1

https://joern.readthedocs.io/en/latest/installation.html#system-requirements-and-dependencies
1. extract the tarball
tar -xvf 0.3.1.tar.gz

extract the tarball of build dependencies
```
  cd joern-0.3.1
  tar -xvf lib.tar.gz
```
build the project
```
  cd joern-0.3.1
  ant
```
The executable JAR file is located in joern-0.3.1/bin/joern.jar

set environment variable (optional)

/etc/profile

  # Add These Content at the End of the File
  #######################################
  JOERN_HOME=/home/joern-0.3.1/
  
  export JOERN_HOME
  #######################################

~/.bashrc

  # Add These Content at the End of the File
  #######################################
  alias joern='java -jar $JOERN/bin/joern.jar'
  #######################################

  source /etc/profile
  source ~/.bashrc

build additional tools (optional)
```
  cd joern-0.3.1
  ant tools
```

install python-joern

  apt install python-setuptools python-dev python-pip

  pip2 install py2neo==2.0 -i https://pypi.tuna.tsinghua.edu.cn/simple

  pip2 install py2neo-gremlin -i https://pypi.tuna.tsinghua.edu.cn/simple

  tar -xvf python-joern-0.3.1.tar.gz
  cd python-joern-0.3.1
  python2 setup.py install

install joern-tools

  pip2 install chardet -i https://pypi.tuna.tsinghua.edu.cn/simple
  pip2 install pygraphviz -i https://pypi.tuna.tsinghua.edu.cn/simple

  git clone https://github.com/fabsx00/joern-tools
  cd joern-tools
  python2 setup.py install

verify
```
  joern-lookup
```

Using

Step1 Generating Slices

Work Dir: /home/SySeVR-master/Implementation/source2slice

Code Dir: /home/code

Recommended Memory Size: >=16GB (according to your code size)

If you want to slice the NVD and SARD, you may divide them into parts.

install dependences
```
apt install python-igraph
```
parse the source code

input: source codes

output: .joernIndex
```
rm -rf ./.joernIndex
```
```
java -Xmx16g -jar $JOERN_HOME/bin/joern.jar /home/code
```
This will create a neo4j database directory .joernIndex in this directory.

generate CFG

input: .joernIndex

output: cfg_db

# start neo4j at other terminal
/usr/local/neo4j/bin/neo4j console

mkdir cfg_db
python2 get_cfg_relation.py

generate PDG

input: cfg_db .joernIndex

output: pdg_db

# modify access_db_operate.py
http.socket_timeout = 999999999 # a big number

mkdir pdg_db
python2 complete_PDG.py

generate call graph of functions

input: pdg_db .joernIndex

output: dict_call2cfgNodeID_funcID
```
mkdir dict_call2cfgNodeID_funcID
python2 access_db_operate.py
```

generate four kinds of SyCVs

input: dict_call2cfgNodeID_funcID

outout: arrayuse_slice_points.pkl, integeroverflow_slice_points_new.pkl, pointuse_slice_points.pkl, sensifun_slice_points.pkl

# modify points_get.py near the 128 rows
# change "location" to ",location"
for i in list_usenodes:
                if str(i).find(",location")==-1:
                    list_usenodes.remove(i)
            loc_list=[]
            final_list=[]
            for i in list_usenodes:
                #print(i)
                if ',location' in str(i):
                    print(str(i))
                    location=str(i).split(",type:")[0].split(",location:")[1][1:-1].split(":")
                    count=int(location[0])
                    loc_list.append(count)

python2 points_get.py

extract slices

input: dict_call2cfgNodeID_funcID, arrayuse_slice_points.pkl, integeroverflow_slice_points_new.pkl, pointuse_slice_points.pkl, sensifun_slice_points.pkl

output: api_slices.txt, arrayuse_slice.txt, integeroverflow_slices.txt, pointeruse_slice.txt
```
# modify save-file-path in extract_df.py
store_filepath = "integeroverflow_slices.txt"
store_filepath = "arraysuse_slices.txt"
store_filepath = "pointersuse_slices.txt"
store_filepath = "api_slices.txt"
```
```
# add slice_op.py at 348 rows
if not os.path.exists(path):
	i += 1
	continue
```
Due to the limit of memory, you may execute four functions in extract_df.py one by one.
```
python2 extract_df.py
```
get labels of slices

input: api_slices.txt, arrayuse_slice.txt, integeroverflow_slices.txt, pointeruse_slice.txt

output: apt_slices_label.pkl, api_slices-vulline.pkl, array_slice_label.pkl, expr_slice_label.pkl, pointer_slice_label.pkl
```
# modify make_label.py at 70 rows
# wrong format
_dict_cwe2line = {}
for _dict in dict:
	for key in _dict.keys():
		if _dict[key] not in _dict_cwe2line_target.keys():
```
```
python2 make_label.py
```

combine labels with slices

input: apt_slices_label.pkl, api_slices-vulline.pkl, array_slice_label.pkl, expr_slice_label.pkl, pointer_slice_label.pkl, api_slices.txt, arrayuse_slice.txt, integeroverflow_slices.txt, pointeruse_slice.txt

output: api_slices, arrayuse_slices.txt, integeroverflow_slices.txt, pointersuse_slices.txt

mkdir slices label_source slice_label
cp api_slices.txt arrayuse_slice.txt integeroverflow_slices.txt pointeruse_slice.txt ./slices
cp apt_slices_label.pkl api_slices-vulline.pkl array_slice_label.pkl expr_slice_label.pkl pointer_slice_label.pkl ./label_source
cd label_source
mv expr_slice_label.pkl integeroverflow_slices.pkl
mv apt_slices_label.pkl api_slices.pkl
mv array_slice_label.pkl arrayuse_slice.pkl
mv pointer_slice_label.pkl pointeruse_slice.pkl

python2 data_preprocess.py

Step2 Data Perprocess

pass

Step3 Deep Learning

pass

kingnop/SySeVR_fix

Environments

System

Step 1

Step 2

Installation

Joern 0.3.1

JDK 1.7

Neo4j 2.1.8 Community Edition

Gremlin for Neo4J 2.X

Apache Ant build tool

Joern 0.3.1

Using

Step1 Generating Slices

Step2 Data Perprocess

Step3 Deep Learning