This repository is for the artifact evaluation of the paper "FalsifAI: Falsification of AI-Enabled Hybrid Control Systems Guided by Time-Aware Coverage Criteria".
-
Operating system: Linux or MacOS;
-
Matlab (Simulink/Stateflow) version: >= 2020a. (Matlab license needed)
-
Python version: >= 3.3
-
MATLAB toolboxes dependency:
- Model Predictive Control Toolbox for ACC benchmark
- Stateflow
- Deep Learning Toolbox
.
├── Makefile
├── README.md
├── benchmarks
│ ├── train
│ │ ├── ACC
│ │ │ ├── ACC_config.txt
│ │ │ ├── ACC_falsification.m
│ │ │ ├── ACC_falsify.m
│ │ │ └── ACC_trainController.m
│ │ ├── AFC
│ │ │ ├── AFC_config.txt
│ │ │ ├── AFC_falsification.m
│ │ │ ├── AFC_falsify.m
│ │ │ └── AFC_trainController.m
│ │ └── DPC
│ │ ├── buck_config.txt
│ │ ├── buck_falsification.m
│ │ ├── buck_falsify.m
│ │ └── buck_trainController.m
│ ├── ACC
│ │ ├── dataset
│ │ │ └── ACC_trainset.mat
│ │ ├── model
│ │ │ ├── mpcACCsystem.slx
│ │ │ └── nncACCsystem.slx
│ │ ├── nnconfig
│ │ └── nncontroller
│ ├── AFC
│ │ ├── dataset
│ │ │ └── AFC_trainset.mat
│ │ ├── model
│ │ │ ├── fuel_control.slx
│ │ │ └── nn_fuel_control.slx
│ │ ├── nnconfig
│ │ └── nncontroller
│ └── DPC
│ ├── dataset
│ │ └── buck_trainset.mat
│ ├── model
│ │ ├── my_buck_pid.slx
│ │ └── buck_nn.slx
│ ├── nnconfig
│ └── nncontroller
├── log/
├── results/
├── run
├── robustness_calculator.m(relied on Breach)
├── src
│ ├── TestGen.m
│ ├── main.m
│ ├── nc
│ │ ├── NC.m
│ │ ├── TKC.m
│ │ ├── TNC.m
│ │ ├── TTK.m
│ │ ├── PD.m
│ │ ├── ND.m
│ │ ├── MI.m
│ │ └── MD.m
│ └── util
│ └── CQueue.m
└── test
│ ├── falsifai_test.py
│ ├── breach_test.py
│ ├── scripts
│ └── config
│ ├── AI
│ │ ├── acc.conf
│ │ ├── afc.conf
│ │ └── dpc.conf
│ └── breach
│ ├── acc.conf
│ ├── afc.conf
│ └── dpc.conf
└── analyses
├── results.txt
└── statTest.R
-
Clone the repository
git clone https://github.com/lyudeyun/FalsifAI.git
-
Install Breach
- start matlab, set up a C/C++ compiler using the command
mex -setup
. (Refer to here for more details.) - navigate to
breach/
in Matlab commandline, and runInstallBreach
- start matlab, set up a C/C++ compiler using the command
To reproduce the experimental results, users should follow the steps below:
- The user-specified configuration files are stored in the directory
test/config/
. Replace the paths ofFalsifAI
andbreach
in user-specified file under the lineaddpath 2
with their own paths. Users can also specify other configurations, such as model, input ranges, optimization methods, and etc. - Navigate to the directory
test/
. Run the commandpython [type]_test.py config/[user-specified configuration file]
. Users can generate the testing scripts byai_test.py
orbreach_test.py
. - Now the executable scripts have been generated under the directory
test/benchmarks/
. Users need to edit the executable scripts permission using the commandchmod -R 777 *
. - Navigate to the root directory
falsifAI/
and run the commandmake
. The automatically generated .csv experimental results will be stored in directoryresults/
. - The corresponding log will be stored under directory
output/
.