Rework the panic handling in `$cast()`

[etiennebacher]

I've been a bit too quick in #1219, the unwrap() I left in cast() provokes a panic in one of the tests (which is known) and this isn't handled great when running the tests non-interactively (there's a big panic message, even though the tests still pass)