Trusted-AI/adversarial-robustness-toolbox

BrendelBethgeAttack breaks with array clip range

KandBM opened this issue · 3 comments

KandBM commented

Describe the bug
This works
victim_policy = classifier(
model=policy_net,
loss=CrossEntropyLoss(), #most common func for classification
nb_classes=env.action_space[0].n,
input_shape=agent.observation_space.shape,
device_type='gpu',
clip_values = (0,1)
)
bba = BrendelBethgeAttack(estimator=victim_policy)
bba.generate(np.expand_dims(obs, axis=0))

but defining the clip range with arrays leads to the error below:
victim_policy = classifier(
model=policy_net,
loss=CrossEntropyLoss(), #most common func for classification
nb_classes=env.action_space[0].n,
input_shape=agent.observation_space.shape,
device_type='gpu',
clip_values =(agent.observation_space.low, agent.observation_space.high) #min and max values of each feature
)
Which seems to be an issue with numba's iadd.
{
"name": "TypingError",
"message": "Failed in nopython mode pipeline (step: nopython frontend)

  • Resolution failure for literal arguments:
    Failed in nopython mode pipeline (step: nopython frontend)
  • Resolution failure for literal arguments:
    Failed in nopython mode pipeline (step: nopython frontend)
    No implementation of function Function() found for signature:

iadd(float64, array(float64, 1d, C))

There are 18 candidate implementations:
- Of which 16 did not match due to:
Overload of function 'iadd': File: : Line N/A.
With argument(s): '(float64, array(float64, 1d, C))':
No match.
- Of which 2 did not match due to:
Operator Overload in function 'iadd': File: unknown: Line unknown.
With argument(s): '(float64, array(float64, 1d, C))':
No match for registered cases:
* (int64, int64) -> int64
* (int64, uint64) -> int64
* (uint64, int64) -> int64
* (uint64, uint64) -> uint64
* (float32, float32) -> float32
* (float64, float64) -> float64
* (complex64, complex64) -> complex64
* (complex128, complex128) -> complex128

During: typing of intrinsic-call at c:\Users\usr\anaconda3\envs\CityLearnART\lib\site-packages\art\attacks\evasion\brendel_bethge.py (697)

File "..\..\..\anaconda3\envs\CityLearnART\lib\site-packages\art\attacks\evasion\brendel_bethge.py", line 697:
def _max_logit_diff(self, x, b, _ell, _u, c):

if b[n] > 0:
cmax += b[n] * (_u - x[n])
^

  • Resolution failure for non-literal arguments:
    None

During: resolving callee type: BoundFunction((<class 'numba.core.types.misc.ClassInstanceType'>, '_max_logit_diff') for instance.jitclass.LinfOptimizer#28e19ab1300<bfgsb:instance.jitclass.BFGSB#28e19a5f8b0<>>)
During: typing of call at c:\Users\usr\anaconda3\envs\CityLearnART\lib\site-packages\art\attacks\evasion\brendel_bethge.py (656)

File "..\..\..\anaconda3\envs\CityLearnART\lib\site-packages\art\attacks\evasion\brendel_bethge.py", line 656:
def solve(self, x0, x, b, min_, max_, c, r):

x0, x, b = x0.astype(np.float64), x.astype(np.float64), b.astype(np.float64)
cmax, cmaxnorm = self.max_logit_diff(x, b, min, max_, c)
^

  • Resolution failure for non-literal arguments:
    None

During: resolving callee type: BoundFunction((<class 'numba.core.types.misc.ClassInstanceType'>, 'solve') for instance.jitclass.LinfOptimizer#28e19ab1300<bfgsb:instance.jitclass.BFGSB#28e19a5f8b0<>>)
During: typing of call at (3)

File "", line 3:

", "stack": "--------------------------------------------------------------------------- TypingError Traceback (most recent call last) c:\\Users\\usr\\Documents\\CityLearn Examples\\ARTtest 1d discrete.ipynb Cell 45 line 1 ----> 1 bba.generate(np.expand_dims(obs, axis=0))

File c:\Users\usr\anaconda3\envs\CityLearnART\lib\site-packages\art\attacks\evasion\brendel_bethge.py:2368, in BrendelBethgeAttack.generate(self, x, y, **kwargs)
2365 _c = corr_logits_diffs[k]
2366 r = region[sample]
-> 2368 delta = self._optimizer.solve(_x0, _x, _b, bounds[0], bounds[1], _c, r) # type: ignore
2369 deltas.append(delta)
2371 k += 1 # idx of masked sample

File c:\Users\usr\anaconda3\envs\CityLearnART\lib\site-packages
umba\experimental\jitclass\boxing.py:61, in _generate_method..wrapper(*args, **kwargs)
59 @wraps(func)
60 def wrapper(*args, **kwargs):
---> 61 return method(*args, **kwargs)

File c:\Users\usr\anaconda3\envs\CityLearnART\lib\site-packages
umba\core\dispatcher.py:468, in _DispatcherBase._compile_for_args(self, *args, **kws)
464 msg = (f"{str(e).rstrip()}

This error may have been caused "
465 f"by the following argument(s):
{args_str}
")
466 e.patch_message(msg)
--> 468 error_rewrite(e, 'typing')
469 except errors.UnsupportedError as e:
470 # Something unsupported is present in the user code, add help info
471 error_rewrite(e, 'unsupported_error')

File c:\Users\usr\anaconda3\envs\CityLearnART\lib\site-packages
umba\core\dispatcher.py:409, in _DispatcherBase._compile_for_args..error_rewrite(e, issue_type)
407 raise e
408 else:
--> 409 raise e.with_traceback(None)

TypingError: Failed in nopython mode pipeline (step: nopython frontend)

  • Resolution failure for literal arguments:
    Failed in nopython mode pipeline (step: nopython frontend)
  • Resolution failure for literal arguments:
    Failed in nopython mode pipeline (step: nopython frontend)
    No implementation of function Function() found for signature:

iadd(float64, array(float64, 1d, C))

There are 18 candidate implementations:
- Of which 16 did not match due to:
Overload of function 'iadd': File: : Line N/A.
With argument(s): '(float64, array(float64, 1d, C))':
No match.
- Of which 2 did not match due to:
Operator Overload in function 'iadd': File: unknown: Line unknown.
With argument(s): '(float64, array(float64, 1d, C))':
No match for registered cases:
* (int64, int64) -> int64
* (int64, uint64) -> int64
* (uint64, int64) -> int64
* (uint64, uint64) -> uint64
* (float32, float32) -> float32
* (float64, float64) -> float64
* (complex64, complex64) -> complex64
* (complex128, complex128) -> complex128

During: typing of intrinsic-call at c:\Users\usr\anaconda3\envs\CityLearnART\lib\site-packages\art\attacks\evasion\brendel_bethge.py (697)

File "..\..\..\anaconda3\envs\CityLearnART\lib\site-packages\art\attacks\evasion\brendel_bethge.py", line 697:
def _max_logit_diff(self, x, b, _ell, _u, c):

if b[n] > 0:
cmax += b[n] * (_u - x[n])
^

  • Resolution failure for non-literal arguments:
    None

During: resolving callee type: BoundFunction((<class 'numba.core.types.misc.ClassInstanceType'>, '_max_logit_diff') for instance.jitclass.LinfOptimizer#28e19ab1300<bfgsb:instance.jitclass.BFGSB#28e19a5f8b0<>>)
During: typing of call at c:\Users\usr\anaconda3\envs\CityLearnART\lib\site-packages\art\attacks\evasion\brendel_bethge.py (656)

File "..\..\..\anaconda3\envs\CityLearnART\lib\site-packages\art\attacks\evasion\brendel_bethge.py", line 656:
def solve(self, x0, x, b, min_, max_, c, r):

x0, x, b = x0.astype(np.float64), x.astype(np.float64), b.astype(np.float64)
cmax, cmaxnorm = self.max_logit_diff(x, b, min, max_, c)
^

  • Resolution failure for non-literal arguments:
    None

During: resolving callee type: BoundFunction((<class 'numba.core.types.misc.ClassInstanceType'>, 'solve') for instance.jitclass.LinfOptimizer#28e19ab1300<bfgsb:instance.jitclass.BFGSB#28e19a5f8b0<>>)
During: typing of call at (3)

File "", line 3:

" }

To Reproduce
please see description, I will provide more detail if required

System information (please complete the following information):

  • OS: win10
  • Python version: 3.10.12
  • ART version or commit number: 1.16
  • PyTorch version
  • numba version: 58.1
beat-buesser commented

Hi @KandBM Thank you for reporting this issue. I think your analysis of numba's iadd being part of the issue is correct. Do you think it is possible to replace it with code that can add floats and arrays?

KandBM commented

@beat-buesser It does at least work with floats, so the minimal solution would be the verify that the estimator's clip range contains a single min and max value (none or arrys break it). I'm not familiar enough with numba to rewrite the operation to work with arrays, but fortunately my project works with just a single min and max value. If I do find something I'll be sure to share it.

edit: as far as I can tell, b[n] contains the clip range, but numba can only run it if b[n] contains a float.

the fix I can suggest now is adding something like this where the BrendelBethgeAttack is initialized after lin 1943 (I'm new to git in general, but I'm happy to make some changes if you can point me in the right direction):

if not isinstance(estimator.clip_values[0], (int, float)) or not isinstance(estimator.clip_values[1], (int, float)):
        raise ValueError("Clip range must be a tuple of two values, ref issue: https://github.com/Trusted-AI/adversarial-robustness-toolbox/issues/2320")

Edit2: The code above would go in the _check_params at the end of the file

  • 👍1