Skip cache in the test workflows

[divyansh42]

Ideally to test the action in a nightly job, we should not use cache to fetch the binaries. This way we won't be able to detect any bug if something got changed in the mirror download path. So, to deal with this issue we will have to set env to skip the cache.

[tetchel]

maybe we could test both, or test randomly (like roll 0 or 1 and skip cache 50% of the time)