`App.new_temporary()` should set no-op caches

[bartfeenstra]

App.new_temporary() should set no-op caches to reduce unnecessary disk IO during tests. Note that this may not work for the binary file cache depending on our use cases (assumptions we make).