via Iterative-Accumulating Algorithm with Optional CUDA Acceleration
-
CPython 3.8 or above is recommanded, CPython 3.6 should be compatible but not tested
-
Dependent packages
Pillow
opencv-python # for ground-truth mask processing and PRO's CCA
numpy
torch # Optional, for CUDA acceleration- Optional dependent packages for comparing with reference implementation
scikit-learn
scipy
bisect- Quick Start
# example 1: evaluate result of object 'audiojack' from Real-IAD under Multi-view UIAD setting
python3 -m adeval --sample_key_pat "([a-zA-Z][a-zA-Z0-9_]*_[0-9]{4}_[A-Z][A-Z_]*[A-Z])_C[0-9]_" real_iad-audiojack-Mv_UIAD.pkl
# example 2: evaluate 4 results of object 'audiojack' from Real-IAD under FUAD settings (i.e. NR = 0.0, 0.1, 0.2, 0.4)
python3 -m adeval --sample_key_pat "([a-zA-Z][a-zA-Z0-9_]*_[0-9]{4}_[A-Z][A-Z_]*[A-Z])_C[0-9]_" \
real_iad-audiojack-FUIAD0.pkl \
-a real_iad-audiojack-FUIAD1.pkl \
-a real_iad-audiojack-FUIAD2.pkl \
-a real_iad-audiojack-FUIAD4.pkl- Detailed Description of Arguments
usage: python3 -m adeval [-h] [--remove_prefix REMOVE_PREFIX] [--add_prefix ADD_PREFIX] [--sample_key_pat SAMPLE_KEY_PAT] [--expected_views_per_sample EXPECTED_VIEWS_PER_SAMPLE] [-a MORE_ANOMAP] anomap
positional arguments:
anomap path to a pickle of a dict[str, tuple[ndarray, float, str | None]], whose key is path to image
of dataset, and values are predicted anomap, predicted image-level score and path to the ground
truth anomaly mask respectively
optional arguments:
-h, --help show this help message and exit
--remove_prefix REMOVE_PREFIX
remove prefix from path to image & mask first
--add_prefix ADD_PREFIX
add another prefix to path to image & mask
--evaluate_size EVALUATE_SIZE
calculate metrics at specific size rather than original size of images, can be [size] (a single integer) or a [w,h] (a tuple of integers)
--sample_key_pat SAMPLE_KEY_PAT
regex to capture the sample id from image path, for evaluating sample-level metrics
--expected_views_per_sample EXPECTED_VIEWS_PER_SAMPLE
if given, script will check if each sample has the expected views, and raise error if not
-a MORE_ANOMAP, --more_anomap MORE_ANOMAP
Here is a code snippet to construct anomap .pkl file that can be passed to ADEval commandline tool
import numpy as np
import pickle
anomaps = {}
for image_path, mask_path in test_dataset:
assert isinstance(image_path, str)
assert mask_path is None or isinstance(mask_path, str)
img = load_image(image_path)
anomap, image_score = inference(img)
assert isinstance(image_score, float)
assert (isinstance(anomap, np.ndarray) and
anomap.ndim == 2 and
'float' in anomap.dtype.name)
anomaps[image_path] = (anomap, image_score, mask_path)
with open('/path/to/save/anomap.pkl', 'wb') as fp:
pickle.dump(anomaps, fp)Following the code snippet below to utilize the iterative style API, so we do not have to save ano-maps of all images in the test set.
from adeval import EvalAccumulatorCuda
# for cpu-only environment, use the class below and remove all cuda-related
# operation in the codes
from adeval import EvalAccumulator
def get_sample_id(path: str) -> str:
...
# get id string of sample from path of image
def validation(...):
# determin the lower & upper bound of image-level score according to your
# algorithm design, ensure that the scores of 99% of the images fall within
# the given lower & upper bound
score_min, score_max = ...
# also determine the lower & upper bound of values in anomap, or set to
# None to reuse bounds of image-level score
anomap_min, anomap_max = ...
accum = EvalAccumulatorCuda(score_min, score_max, anomap_min, anomap_max)
# the DataLoader of test set
test_loader = ...
image_scores: List[Tuple[str, float, int]] = []
for items in test_loader:
anomap, score = model(items['image'].cuda())
mask: Tensor = items['mask']
label: Tensor = items['label']
path: List[str] = items['path']
mask = mask.cuda(non_blocking=True)
accum.add_anomap_batch(anomap, mask)
accum.add_image(score, label)
image_scores += [(key, img_score.item(), img_label.item())
for key, img_score, img_label in
zip(path, score.reshape(-1), label.reshape(-1))]
sample_pairs = [([score for _, score, _ in views],
[label for _, _, label in views],
[path for path, _, _ in views], key)
for key, views in [(key, list(views)) for key, views in groupby(
sorted(image_scores, key=lambda tup: tup[0]), key=lambda tup: get_sample_id(tup[0])
)]]
accum.add_sample([max(score) for score, _, _, _ in sample_pairs],
[max(label) for _, label, _, _ in sample_pairs])
metrics = accum.summary()
print(metrics)
# contains:
# - sample-level auroc, aupr
# - image-level auroc, aupr
# - pixel-level auroc, aupr, auproAnd please refer to the functional example test.py for more details
For usage of direct APIs (i.e. auroc_and_aupr() etc.), please following implementation of CLI
The team of Real-IAD dataset has contributed numerous invaluable insights to this project.
The reference implementations are from repos AnomalyDetection-SoftPatch and open-iad:
metrics.py: reference of AUROC & AUPR based onscikit-learnpackage from AnomalyDetection-SoftPatchau_pro: reference of AUPRO from open-iad