race condition on route deletion scripts
sprocketsecurity opened this issue · 5 comments
the terraform local provisioner will execute the del_route.bash script on destruction of an exit-node. The way the del_route.bash script is written a race condition can exist and an exit-node IP will be removed but later added by another executing instance of del_route.bash.
engineering notes:
fix this by serializing script execution. try pidof del_script.bash
i've added some preventive measures in commit 712a98a but i'm not sure it solved this problem 100%. leaving open for now.
from @metaDNA:
When using terraform destroy and terraform apply, the AWS changes execute successfully but the proxycannon-ng/nodes/aws/.routecmd does not get updated to purge the old hosts out of the route table loadb set. The effect is that random packets get dropped as a result of being routed to hosts that don't exist anymore. Took me awhile to figure it out but manually reconciling it fixed the problem as a stopgap.
it appears to be a race condition when executing multiple instances of the route addition/deletion scripts. I've attempted to put checks to avoid this, but it still happens from time to time.
Looking for help on this. In the meantime this should probably be added to a troubleshooting wiki until resolved.
Here's the code I use to grab the IPs of the exit nodes when I manually mod the .routecmd file:
aws ec2 describe-instances --filter "Name=tag:Name,Values=Proxycannon-ng-exit-node" --output json | grep PrivateIp | grep -v '\[' | cut -d '"' -f4 | sort -u
Maybe you could incorporate something like that at the end of each terraform apply/destroy and do one update instead of an update for each host as it's running through to avoid the race condition?
it appears to be a race condition when executing multiple instances of the route addition/deletion scripts. I've attempted to put checks to avoid this, but it still happens from time to time.
Looking for help on this. In the meantime this should probably be added to a troubleshooting wiki until resolved.
Here's what I've shimmed in to run after terraform commands (attached). But note that you will need aws cli installed (and configured).
repair_routes.txt