[FEATURE] Replace atexit with __exit__

[InvincibleRMC]

Is your feature request related to a problem? Please describe.
It's goofy the way we do it tbh.

Describe the solution you'd like
Write __exit__ functions everywhere we use atexit

Describe alternatives you've considered
Having the goofy atexit