Allow constraint of pool resources on multiprocess.pool() execution within parallel AutoAttack
Closed this issue · 1 comments
Is your feature request related to a problem? Please describe.
Problem
python=3.11
When leveraging attacks/evasion/auto_attack.py
in parallel, the multiprocess.pool() call is unconstrained and will use os.cpu_count()
as a default value (all cpus). This is problematic in scenarios when all resources should not be consumed by some mechanism leveraging AutoAttack.
multiprocess.pool documentation
processes is the number of worker processes to use. If processes is None then the number returned by os.cpu_count() is used.
Note: The functionality for multprocess.pool is slightly different in python>=3.13. It will leverage os.process_cpu_count()
instead, which is an abstraction of threads available.
- Jupyter notebook leveraging parallel AutoAttack
- All CPUs are maxed by this process until completion
Describe the solution you'd like
Ask
Allow users to optionally specify the number of processes that parallel AutoAttack will use within its pool.
Possible Solution:
- Expose new param for
parallel_pool_size
- Init pool size as 0/Unmodified:
parallel_pool_size: int = 0,
- If pool size was unmodified, capture max available - 1 (to prevent 100% resource consumption)
147 ++ pool_size = multiprocess.cpu_count() - 1 if multiprocess.cpu_count() > 1 else multiprocess.cpu_count()
148 ++ if self.parallel_pool_size > 0:
149 ++ pool_size = self.parallel_pool_size
- Use
pool_size
as the multiprocess.pool() constraint
291 ++ with multiprocess.get_context("spawn").Pool(processes=pool_size) as pool:
Describe alternatives you've considered
This is a feature to allow users the ability to control resource consumption with parallel execution of AutoAttack (with the addition of not consuming all CPUs in all cases if unmodified by a user). The above example could be reworked for cleaner code, but the premise of exposing a new param for users to control pool size would still be necessary for this feature.
Additional context
@beat-buesser My work here is in context with the internal heart-library research I conducted for IBM located in this Internal GitLab thread
Hi @lockwoodar Thank you very much for reporting this issue!