Unstable fingerprints tracking issue

[Mark-Simulacrum]

For users hitting this bug: please read this blog post which discusses what you can do. An excerpt is below:

The Internal Compiler Error asks you to report a bug, and if you can do so, we still want that information. We want to know about the cases that are failing.

But regardless of whether or not you file a bug, the problem here can be resolved by either:

1. deleting your incremental compilation cache (e.g. by running cargo clean for the whole project or cargo clean -p crate-causing-the-error), or
2. force incremental compilation to be disabled, by setting CARGO_INCREMENTAL=0 in your environment or setting build.incremental to false in your $HOME/.cargo/config.toml file (which might also be called $HOME/.cargo/config) by adding this:

build.incremental = 'false'

We recommend that users of 1.52.0 disable incremental compilation, to avoid running into this problem.

We do not recommend that users of 1.52.0 downgrade to an earlier version of Rust in response to this problem. There is at least one instance of a silent miscompilation caused by incremental compilation that was not caught until we added the fingerprint checking.

---

I believe the (new) ICEs as a result of improved error detection in the incremental code will land in 1.53 onto stable, which is in 6 weeks. We are unlikely to fix all of the fallout (i.e. all bugs, at least 25 of which are currently open, though some are duplicates) by that time and be willing to backport the fixes. I suspect we are also unlikely to want to disable the new assertions; they are catching known unsoundness. I am opening this issue to discuss possible alternatives, which are hopefully more user friendly than the current ICE.

I think one direct improvement is to adjust the panic/assert on encountering this to be a proper compile error, which can tell the user that they should (for example) delete the incremental directory and invoke the compiler again. This is already a significant improvement over the current message, I suspect. We can still ask that they file a bug report.

We should also consider if it's worth considering applying performance-hurting mitigations; it seems definitely true we can drop incremental support on beta/stable until these bugs are fixed, but perhaps a smaller hammer is also viable.

Table of ICEs:

Issue	Query name	Has MCVE	PR fixing issue	fixed in 1.5x?	PR adding regression test	Done
#83259	predicates_of	✅	#84233	1.53	#84233	✅
#83311	proc_macro_decls_static					N/A
#83126	extern_mod_stmt_cnum	mcve-83126	#83153	1.52	None yet	❌
#85197	optimized_mir	mcve-85197	#85211	1.54	#85211	✅
#84225	exported_symbols	?	#84226	1.53	None yet	❌
#85019	item_children	?	#83901	1.54	None yet	❌
#83538	evaluate_obligation (EvaluatedToOk and EvaluatedToOkModuloRegions)	✅	#85186	1.54	#85186	✅
#84963	evaluate_obligation (OverflowError)	✅	None yet	?	None yet	❌
	native_libraries	?	#85702	1.54	#85702	✅
#85360	evaluate_obligation	✅	#85868	1.56	#91065	✅

cc @Aaron1011 (driver of fixing the bugs, I believe)
cc @pnkfelix @wesleywiser (T-compiler leads)
cc @rust-lang/release for awareness

[Mark-Simulacrum]

Nominating for T-compiler as I believe we should surface this during a meeting - likely for some small discussion, but mostly for awareness, and to see if we can get some help for possible ideas. I think this is a P-critical bug - not in the sense that we must have a fix for the underlying cause, but in the sense that this issue needs to be addressed in time for the next stable release.

I think it is likely worth applying the same fix to nightly as well, but perhaps in a channel-gated way if it has (for example) simply performance implications. This is a bit unclear.

[Aaron1011]

I think emitting a compiler error would be a great idea. However, I think it should still encourage users to open an issue (emphasizing that this is an internal compiler bug), so that we can see if any new issues pop up.

[Aaron1011]

OUTDATED: See the new table in #84970 (comment)

Here's the current state of the crashes involving different queries:

• item_children - this is caused by not hashing attributes. PR #83901 fixed this
• predicate_of - this was fixed by #84233. However, we could still use a regression test before closing the issue #84341
• evaluate_obligation (EvaluatedToOk/EvaluatedToOkModuloRegions) - this has an open PR #83913, but that PR has a large performance impact. I've been working on minimizing the syn crash, which might yield more insight into how to avoid the performance problem.
• evaluate_obligation (OverflowError) - this is issue #84963. This is completely different from the other evaluate_obligation crash, and I haven't had a chance to investigate it yet. Hopefully, the fix will be much simpler.
• optimized_mir - this is issue #84960 and #85064

As far as I know, these are all of the currently existing crashes. If anyone sees a crash involving a query other than one of these, please add it to this list.

[jyn514]

item_children - this is caused by not hashing attributes. PR #79519 is open with a fix

I think you linked to the wrong pr - 79519 is storing attributes in a side table and has already been merged.

[lqd]

item_children (which was seen in #83887 and one the reported cases of #84335) should be fixed by #83901

[jonas-schievink]

I suspect we are also unlikely to want to disable the new assertions; they are catching known unsoundness.

I think this would actually most likely be fine – these ICE-causing bugs have existed for years, and it took very long for someone to run into a miscompilation.

[Mark-Simulacrum]

Well, a known miscompilation - certainly the amount of ICEs we've seen suggests plenty of wrong results, right? Maybe in practice those don't cause problems, I don't know, but I'd rather give an error, especially if we can also e.g. poison the incremental cache automatically and make it just a matter of rerunning the compiler. In theory we could even teach Cargo to do that, though maybe that's going too far.

[Mark-Simulacrum]

We discussed this during T-compiler meeting today, and decided on these actions:

• @Aaron1011 will work on a backportable PR that switches the ICE to a nicer-looking (but still clearly 'our fault' vs. user's fault message) error, using the standard error infrastructure. This is intended to more clearly communicate that the bug is known, and that we're working on it, while also providing a clear suggestion on how to fix it (cargo clean? rm -rf /path/to/incremental/dir?).
  • There was some thoughts about possibly wanting to change all ICEs to be nicer to users, but this was left out of scope; we definitely don't want to backport that.
• We will not disable the ICE on beta/stable, since it is a pretty clear bug, and even though known problems arising from it are rare, better to be safe.

[Aaron1011]

I've opened #84998 to show a nicer error.

[carols10cents]

1. deleting your incremental compilation cache (e.g. by running `cargo clean`), or

2. force incremental compilation to be disabled, by setting `CARGO_INCREMENTAL=0` in your environment or `build.incremental` to `false` in the `config.toml`.

We recommend that users of 1.52.0 disable incremental compilation, to avoid running into this problem.

We do not recommend that users of 1.52.0 downgrade to an earlier version of Rust in response to this problem. As noted above, there is at least one instance of a silent miscompilation caused by incremental compilation that was not caught until we added the fingerprint checking.

These workarounds are... not ideal for developer workflow time. I'd rather downgrade to an older stable than turn off incremental compilation. I see #82920 says:

This reproduces both on stable 1.50.0,

and there's some discussion of bisecting but I don't easily see a result of bisecting. Does anyone know the earliest stable version that doesn't have the miscompilation?

[Mark-Simulacrum]

We are not aware, to my knowledge, of a "guaranteed to work" release - the known miscompilation was introduced at some point, but there may and likely are other unknown miscompilations as well, which is why the recommendation is either to clean or disable. You can clean specifically the most local crate (e.g., cargo clean -p crate-that-iced) and we are working on suggesting that in the error message from rustc.

[archseer]

I think I've found another one that's not included on the list (#84970 (comment)):

thread 'rustc' panicked at 'found unstable fingerprints for native_libraries(helix_syntax[382e]): [NativeLib { kind: StaticBundle, name: Some("tree-sitter-php-c"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-php-cpp"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-ruby-c"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-python-c"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-ruby-cpp"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-python-cpp"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-swift-c"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-css-c"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-bash-c"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-bash-cpp"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-go-c"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-json-c"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-cpp-c"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-cpp-cpp"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-scala-c"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-javascript-c"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-java-c"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-html-c"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-c-sharp-c"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-html-cpp"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-julia-c"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-rust-c"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-toml-c"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-c-c"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-agda-c"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-agda-cpp"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: Unspecified, name: Some("stdc++"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-tsx-c"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-typescript-c"), cfg: None, foreign_module: None, wasm_import_module: None }]', /rustc/88f19c6dab716c6281af7602e30f413e809c5974/compiler/rustc_query_system/src/query/plumbing.rs:593:5

I'll see if I can trigger it again, I think it happens sometimes if the linked C/C++ library is rebuilt by build.rs.

[Aaron1011]

@archseer: Can you post the full error message you got?
Looking at the native_libraries query implementation, I suspect that the issue you encountered may have been fixed by #83901.

[camelid]

I saw someone post an ICE here: #80336 (comment)

[archseer]

@Aaron1011 sure thing, here's the backtrace:

Error message

thread 'rustc' panicked at 'found unstable fingerprints for native_libraries(helix_syntax[382e]): [NativeLib { kind: StaticBundle, name: Some("tree-sitter-html-c"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-json-c"), cfg: None, foreign_modu
le: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-toml-c"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-css-c"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: Static
Bundle, name: Some("tree-sitter-javascript-c"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-scala-c"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-swift-c"), cfg: None,
foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-c-c"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-python-c"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib {
kind: StaticBundle, name: Some("tree-sitter-java-c"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-go-c"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-php-c"), cfg: None,
 foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-bash-c"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-python-cpp"), cfg: None, foreign_module: None, wasm_import_module: None }, Native
Lib { kind: StaticBundle, name: Some("tree-sitter-html-cpp"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-bash-cpp"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-php-cpp
"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-ruby-c"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-rust-c"), cfg: None, foreign_module: None, wasm_import_module: None
 }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-cpp-c"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-julia-c"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-r
uby-cpp"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-cpp-cpp"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-c-sharp-c"), cfg: None, foreign_module: None, wasm_import_m
odule: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-agda-c"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-agda-cpp"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: Unspecified, name: Some("s
tdc++"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-tsx-c"), cfg: None, foreign_module: None, wasm_import_module: None }, NativeLib { kind: StaticBundle, name: Some("tree-sitter-typescript-c"), cfg: None, foreign_module: None, wasm_import_mo
dule: None }]', /rustc/88f19c6dab716c6281af7602e30f413e809c5974/compiler/rustc_query_system/src/query/plumbing.rs:593:5
stack backtrace:
   0:     0x7f1e3a1e79a0 - std::backtrace_rs::backtrace::libunwind::trace::h74532f8b485906a1
                               at /rustc/88f19c6dab716c6281af7602e30f413e809c5974/library/std/src/../../backtrace/src/backtrace/libunwind.rs:90:5
   1:     0x7f1e3a1e79a0 - std::backtrace_rs::backtrace::trace_unsynchronized::h2c8425dd0be5999b
                               at /rustc/88f19c6dab716c6281af7602e30f413e809c5974/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x7f1e3a1e79a0 - std::sys_common::backtrace::_print_fmt::hd8ce6577f46119ce
                               at /rustc/88f19c6dab716c6281af7602e30f413e809c5974/library/std/src/sys_common/backtrace.rs:67:5
   3:     0x7f1e3a1e79a0 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h6289eeebc1f97d57
                               at /rustc/88f19c6dab716c6281af7602e30f413e809c5974/library/std/src/sys_common/backtrace.rs:46:22
   4:     0x7f1e3a255f2f - core::fmt::write::h3d3f8a8bb9d4c367
                               at /rustc/88f19c6dab716c6281af7602e30f413e809c5974/library/core/src/fmt/mod.rs:1092:17
   5:     0x7f1e3a1dbcc2 - std::io::Write::write_fmt::ha117cbfa72664a7b
                               at /rustc/88f19c6dab716c6281af7602e30f413e809c5974/library/std/src/io/mod.rs:1572:15
   6:     0x7f1e3a1eb7e5 - std::sys_common::backtrace::_print::h112e8a4ac2720a21
                               at /rustc/88f19c6dab716c6281af7602e30f413e809c5974/library/std/src/sys_common/backtrace.rs:49:5
   7:     0x7f1e3a1eb7e5 - std::sys_common::backtrace::print::h2a4ab5824600ebce
                               at /rustc/88f19c6dab716c6281af7602e30f413e809c5974/library/std/src/sys_common/backtrace.rs:36:9
   8:     0x7f1e3a1eb7e5 - std::panicking::default_hook::{{closure}}::h399ab06c3b83a1f2
                               at /rustc/88f19c6dab716c6281af7602e30f413e809c5974/library/std/src/panicking.rs:208:50
   9:     0x7f1e3a1eb293 - std::panicking::default_hook::h7481e93ca9c2d739
                               at /rustc/88f19c6dab716c6281af7602e30f413e809c5974/library/std/src/panicking.rs:225:9
  10:     0x7f1e3a9f97eb - rustc_driver::report_ice::h95067985755a845f
  11:     0x7f1e2a3620d3 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h5d01fb651b450bfc
                               at /nix/store/kamyzfvnm1l12i1km5qlqak3md8b86cw-rust-default-1.52.0/lib/rustlib/src/rust/library/alloc/src/boxed.rs:1560:9
  12:     0x7f1e2a3c9816 - proc_macro::bridge::client::<impl proc_macro::bridge::Bridge>::enter::{{closure}}::{{closure}}::h3adc72702b16a2dd
                               at /nix/store/kamyzfvnm1l12i1km5qlqak3md8b86cw-rust-default-1.52.0/lib/rustlib/src/rust/library/proc_macro/src/bridge/client.rs:320:21
  13:     0x7f1e3a1ebf50 - std::panicking::rust_panic_with_hook::h01f9d4e9485ccb8b
                               at /rustc/88f19c6dab716c6281af7602e30f413e809c5974/library/std/src/panicking.rs:595:17
  14:     0x7f1e3a1ebac7 - std::panicking::begin_panic_handler::{{closure}}::h25bff4eb752cc444
                               at /rustc/88f19c6dab716c6281af7602e30f413e809c5974/library/std/src/panicking.rs:497:13
  15:     0x7f1e3a1e7e5c - std::sys_common::backtrace::__rust_end_short_backtrace::h5dfd3f7920e58cc5
                               at /rustc/88f19c6dab716c6281af7602e30f413e809c5974/library/std/src/sys_common/backtrace.rs:141:18
  16:     0x7f1e3a1eba29 - rust_begin_unwind
                               at /rustc/88f19c6dab716c6281af7602e30f413e809c5974/library/std/src/panicking.rs:493:5
  17:     0x7f1e3a1b017b - std::panicking::begin_panic_fmt::h5ea05d30b78d4fb2
                               at /rustc/88f19c6dab716c6281af7602e30f413e809c5974/library/std/src/panicking.rs:435:5
  18:     0x7f1e3ca12208 - rustc_query_system::query::plumbing::incremental_verify_ich::h47a128fd2d21d867
  19:     0x7f1e3ca20940 - rustc_query_system::query::plumbing::load_from_disk_and_cache_in_memory::hf2afe55e6c58afa6
  20:     0x7f1e3c9d1245 - rustc_query_system::query::plumbing::get_query_impl::hf9d3349225b93a37
  21:     0x7f1e3ca9cdef - <rustc_query_impl::Queries as rustc_middle::ty::query::QueryEngine>::native_libraries::haa37d73efd384498
  22:     0x7f1e3ccaa5ce - rustc_codegen_ssa::base::<impl rustc_codegen_ssa::CrateInfo>::new::h3187b5f7ea6bae4c
  23:     0x7f1e3c67bd6e - rustc_codegen_ssa::back::write::start_async_codegen::h1ec7167f2e5faa59
  24:     0x7f1e3c6bd632 - <rustc_codegen_llvm::LlvmCodegenBackend as rustc_codegen_ssa::traits::backend::CodegenBackend>::codegen_crate::hffa2e429d7f8b220
  25:     0x7f1e3c643c81 - rustc_interface::passes::QueryContext::enter::h7609beb470aa8216
  26:     0x7f1e3c64c646 - rustc_interface::queries::Queries::ongoing_codegen::hdae2d9fee8a32b6b
  27:     0x7f1e3c60a213 - rustc_interface::queries::<impl rustc_interface::interface::Compiler>::enter::h1d93706229b66524
  28:     0x7f1e3c604901 - rustc_span::with_source_map::h7c8bdfd885d5eae7
  29:     0x7f1e3c60b1be - rustc_interface::interface::create_compiler_and_run::h3a24cbe4fe4fc490
  30:     0x7f1e3c605798 - scoped_tls::ScopedKey<T>::set::hb080f16fd4dd580e
  31:     0x7f1e3c60b56b - std::sys_common::backtrace::__rust_begin_short_backtrace::h19d7a04d4fdf4bf2
  32:     0x7f1e3c622bb5 - core::ops::function::FnOnce::call_once{{vtable.shim}}::hb375368372f2550c
  33:     0x7f1e3a1fb73a - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h72cc29c4d47b6ef1
                               at /rustc/88f19c6dab716c6281af7602e30f413e809c5974/library/alloc/src/boxed.rs:1546:9
  34:     0x7f1e3a1fb73a - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hb6246b2a6989332b
                               at /rustc/88f19c6dab716c6281af7602e30f413e809c5974/library/alloc/src/boxed.rs:1546:9
  35:     0x7f1e3a1fb73a - std::sys::unix::thread::Thread::new::thread_start::h09db1c25841995a6
                               at /rustc/88f19c6dab716c6281af7602e30f413e809c5974/library/std/src/sys/unix/thread.rs:71:17
  36:     0x7f1e3a134e9e - start_thread
  37:     0x7f1e3a05449f - __GI___clone
  38:                0x0 - <unknown>

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: rustc 1.52.0 (88f19c6da 2021-05-03) running on x86_64-unknown-linux-gnu

note: compiler flags: -C embed-bitcode=no -C debuginfo=2 -C incremental -C link-arg=-fuse-ld=lld -C target-cpu=native --crate-type lib

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
#0 [native_libraries] looking up the native libraries of a linked crate
end of query stack
error: could not compile `helix-core`

To learn more, run the command again with --verbose.

Let me know if it's a separate bug and I'll open a new issue if necessary.

[wesleywiser]

OUTDATED: See the new table in #84970 (comment)

I just spent some time going over the issue list and closing duplicates so I thought I'd leave a comment summarizing the currently known issues in the queue.

• predicates_of #83259 ("Should be fixed by #84233")
• proc_macro_decls_static #83311
• extern_mod_stmt_cnum #83126 ("Should have been fixed by #83153")
• optimized_mir #83817
• exported_symbols comment on #84116
• item_children #85019 (fixed by #83901)
• evaluate_obligation => Ok(EvaluatedToOk) #83538
• evaluate_obligation => Ok(EvaluatedToOkModuloRegions) #83381
• evaluate_obligation => Err(OverflowError) #84963
• evaluate_obligation (insufficient details) #83115, #83226, #83291

Interestingly, the predicates_of issue was the most reported with 17 duplicates. The fix for that (#83220) is in 1.53 beta.

[Aaron1011]

@wesleywiser - PR #83220 is a fix for a different evaluate_obligation issue (not the one affecting syn). PR #84233 should fix the predicates_of crash.

[the8472]

We recommend that users of 1.52.0 disable incremental compilation, to avoid running into this problem.

Maybe also point users to sccache if they're doing repeated non-incremental builds?

[landaire]

I'm hitting this pretty reliably while trying to compile the Rust toolchain on my own branch with minor changes. Full log here: https://gist.github.com/landaire/242bb87b6c3c026eb5c988781499a507

[Aaron1011]

@landaire: Are you able to reliably reproduce that particular crash (e.g. after running cargo clean)? If so, could you post your steps here? I've been unable to reproduce that particular issue (involving optimized_mir) so far.

[landaire]

@Aaron1011 yes. Here is the same error after doing a clean build then re-build: https://gist.github.com/landaire/0216e9abd4112066b901978e759119be.

I fail to compile unless I clean build. I think this only started happening when I started compiling the stage2 compiler which I think makes sense.

This gist contains my changes to Rust on commit 6fd7a6d and my config.toml: https://gist.github.com/landaire/6432c8ffe84e6c65e85edba0bed52b75

To repro:

• Clone the Rust project
• Use my config.toml
• Maybe apply my patches? Probably irrelevant
• Run python .\x.py build
• First build should go smoothly
• Make a change (I was changing the linker options I added in compiler\rustc_codegen_ssa\src\back\link.rs on line 890)
• Second time running python .\x.py build fails

If someone could write me the bisect script I can run it overnight to try to find the problematic commit.

[pnkfelix]

discussed at T-compiler meeting. Assigning to self just to track overall work being done. We'll have a Friday steering meeting about this issue tomorrow; we may downgrade it to P-high at that time, based on the current state of affairs.

[Aaron1011]

While #84233 does not contain an exact regression test for predicate_of, it does test that we invalidate the entire incremental cache when the rust-src component status changes. Since this was the root of the predicates_of issue, I think it's sufficient - adding a more specific test would require us to either rebuild libstd during the test, or refactor the logic around file remapping.

[carols10cents]

Just wanted to note that I hit evaluate_obligation EvaluatedToOk today on 1.53.0-beta.3.

Full error output

thread 'rustc' panicked at 'assertion failed: `(left == right)`
  left: `Some(Fingerprint(8537439170242672706, 4648092694241280842))`,
 right: `Some(Fingerprint(212923312148573672, 18320173992410164289))`: found unstable fingerprints for evaluate_obligation(b0fb4a538915940f-122ed1eb1e3c4fae): Ok(EvaluatedToOk)', /rustc/82b86216422e1ee696e20511180fada7a7a87949/compiler/rustc_query_system/src/query/plumbing.rs:588:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: rustc 1.53.0-beta.3 (82b862164 2021-05-22) running on x86_64-apple-darwin

note: compiler flags: -C embed-bitcode=no -C split-debuginfo=unpacked -C debuginfo=2 -C incremental --crate-type bin

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
#0 [evaluate_obligation] evaluating trait selection obligation `lock_api::rwlock::RwLock<parking_lot::raw_rwlock::RawRwLock, read_buffer::chunk::TableData>: std::marker::Sync`
#1 [typeck] type-checking `myproject::http::write`
end of query stack
error: could not compile `myproject`

[Mark-Simulacrum]

Moving the milestone to 1.54 as we have resolved the regression on 1.53 by again applying the force-incremental patch (#86004).

[umanwizard]

Is it known yet what the probability of this being fixed in 1.54 is?

[est31]

@umanwizard right now I can't see a backport for 1.54, but the backport to disable incremental for 1.53 has been quite late. There are still at least two open issues: #85360 and #84963. I guess users are encouraged to try out the 1.54 release while it's in beta and if those two issues (or any still unreported ones) cause too severe problems, it might be disabled again.

[gilescope]

Will be happy to not be doing cargo clean regularly on nightly before we worry about backporting! Thx all for looking into this. (Turning off incremental compilation is extreamly painful for large rust projects)

[pickfire]

I think I had to cargo clean every day now (3 days in a row as of today, but I don't code on the project daily). After git revert or git rebase, cargo check will always fail whereas cargo run always work even after I got it compiling. I am using stable 1.53.0 and not using nightly. But I don't get any ICE or panics from rustc.

[Mark-Simulacrum]

@pickfire could you file a new issue with more details? It doesn't sound like that quite matches the behavior we're expecting from this issue (which should always lead to an ICE)

[gilescope]

This is the one I was getting. I tried over the weekend to catch it with something smaller but it was hiding.
Is this one of the two known open issues or is this something else?

error: internal compiler error: encountered incremental compilation error with evaluate_obligation(376a67ce9b6af12-57853260916b513e)
  |
  = help: This is a known issue with the compiler. Run `cargo clean -p node_runtime` or `cargo clean` to allow your project to compile
  = note: Please follow the instructions below to create a bug report with the provided information
  = note: See <https://github.com/rust-lang/rust/issues/84970> for more information

thread 'rustc' panicked at 'Found unstable fingerprints for evaluate_obligation(376a67ce9b6af12-57853260916b513e): Ok(EvaluatedToOk)', /rustc/6a758ea7e48416b968955535094479dc2e7cc9e1/compiler/rustc_query_system/src/query/plumbing.rs:624:9
stack backtrace:
error: internal compiler error: encountered incremental compilation error with evaluate_obligation(5b8eb1437fa0177d-1ae52e4e95ec6257)
  |
  = help: This is a known issue with the compiler. Run `cargo clean -p node_runtime` or `cargo clean` to allow your project to compile
  = note: Please follow the instructions below to create a bug report with the provided information
  = note: See <https://github.com/rust-lang/rust/issues/84970> for more information

thread 'rustc' panicked at 'Found unstable fingerprints for evaluate_obligation(5b8eb1437fa0177d-1ae52e4e95ec6257): Ok(EvaluatedToOk)', /rustc/6a758ea7e48416b968955535094479dc2e7cc9e1/compiler/rustc_query_system/src/query/plumbing.rs:624:9
stack backtrace:
   0: _rust_begin_unwind
   1: std::panicking::begin_panic_fmt
   2: rustc_query_system::query::plumbing::incremental_verify_ich
   3: rustc_query_system::query::plumbing::load_from_disk_and_cache_in_memory
   4: rustc_data_structures::stack::ensure_sufficient_stack
   5: rustc_query_system::query::plumbing::get_query_impl
   6: <rustc_query_impl::Queries as rustc_middle::ty::query::QueryEngine>::evaluate_obligation
   7: <rustc_infer::infer::InferCtxt as rustc_trait_selection::traits::query::evaluate_obligation::InferCtxtExt>::evaluate_obligation
   8: <rustc_infer::infer::InferCtxt as rustc_trait_selection::traits::query::evaluate_obligation::InferCtxtExt>::evaluate_obligation_no_overflow
   9: rustc_trait_selection::traits::fulfill::FulfillProcessor::process_trait_obligation
  10: rustc_trait_selection::traits::fulfill::FulfillProcessor::progress_changed_obligations
  11: rustc_data_structures::obligation_forest::ObligationForest<O>::process_obligations
  12: <rustc_trait_selection::traits::fulfill::FulfillmentContext as rustc_infer::traits::engine::TraitEngine>::select_where_possible
  13: <rustc_trait_selection::traits::fulfill::FulfillmentContext as rustc_infer::traits::engine::TraitEngine>::select_all_or_error
  14: rustc_infer::infer::InferCtxtBuilder::enter
  15: rustc_typeck::check::compare_method::check_type_bounds
  16: rustc_typeck::check::compare_method::compare_ty_impl
  17: rustc_typeck::check::check::check_item_type
  18: rustc_middle::hir::map::Map::visit_item_likes_in_module
  19: rustc_typeck::check::check::check_mod_item_types
  20: rustc_query_system::dep_graph::graph::DepGraph<K>::with_task_impl
  21: rustc_data_structures::stack::ensure_sufficient_stack
  22: rustc_query_system::query::plumbing::force_query_with_job
  23: rustc_query_system::query::plumbing::get_query_impl
  24: <rustc_query_impl::Queries as rustc_middle::ty::query::QueryEngine>::check_mod_item_types
  25: rustc_session::utils::<impl rustc_session::session::Session>::time
  26: rustc_typeck::check_crate
  27: rustc_interface::passes::analysis
  28: rustc_query_system::dep_graph::graph::DepGraph<K>::with_task_impl
  29: rustc_data_structures::stack::ensure_sufficient_stack
  30: rustc_query_system::query::plumbing::force_query_with_job
  31: rustc_query_system::query::plumbing::get_query_impl
  32: <rustc_query_impl::Queries as rustc_middle::ty::query::QueryEngine>::analysis
  33: rustc_interface::queries::<impl rustc_interface::interface::Compiler>::enter
  34: rustc_span::with_source_map
  35: rustc_interface::interface::create_compiler_and_run
  36: scoped_tls::ScopedKey<T>::set
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: rustc 1.55.0-nightly (6a758ea7e 2021-06-22) running on x86_64-apple-darwin

note: compiler flags: -C embed-bitcode=no -C split-debuginfo=unpacked -C debuginfo=2 -C incremental --crate-type lib

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
   0: _rust_begin_unwind
   1: std::panicking::begin_panic_fmt
   2: rustc_query_system::query::plumbing::incremental_verify_ich
   3: rustc_query_system::query::plumbing::load_from_disk_and_cache_in_memory
   4: rustc_data_structures::stack::ensure_sufficient_stack
   5: rustc_query_system::query::plumbing::get_query_impl
   6: <rustc_query_impl::Queries as rustc_middle::ty::query::QueryEngine>::evaluate_obligation
   7: <rustc_infer::infer::InferCtxt as rustc_trait_selection::traits::query::evaluate_obligation::InferCtxtExt>::evaluate_obligation
   8: <rustc_infer::infer::InferCtxt as rustc_trait_selection::traits::query::evaluate_obligation::InferCtxtExt>::evaluate_obligation_no_overflow
   9: rustc_trait_selection::traits::fulfill::FulfillProcessor::process_trait_obligation
  10: rustc_trait_selection::traits::fulfill::FulfillProcessor::progress_changed_obligations
  11: rustc_data_structures::obligation_forest::ObligationForest<O>::process_obligations
  12: <rustc_trait_selection::traits::fulfill::FulfillmentContext as rustc_infer::traits::engine::TraitEngine>::select_where_possible
  13: <rustc_trait_selection::traits::fulfill::FulfillmentContext as rustc_infer::traits::engine::TraitEngine>::select_all_or_error
  14: rustc_infer::infer::InferCtxtBuilder::enter
  15: rustc_typeck::check::compare_method::check_type_bounds
  16: rustc_typeck::check::compare_method::compare_ty_impl
  17: rustc_typeck::check::check::check_item_type
  18: rustc_middle::hir::map::Map::visit_item_likes_in_module
  19: rustc_typeck::check::check::check_mod_item_types
  20: rustc_query_system::dep_graph::graph::DepGraph<K>::with_task_impl
  21: rustc_data_structures::stack::ensure_sufficient_stack
  22: rustc_query_system::query::plumbing::force_query_with_job
  23: rustc_query_system::query::plumbing::get_query_impl
  24: <rustc_query_impl::Queries as rustc_middle::ty::query::QueryEngine>::check_mod_item_types
  25: rustc_session::utils::<impl rustc_session::session::Session>::time
  26: rustc_typeck::check_crate
  27: rustc_interface::passes::analysis
  28: rustc_query_system::dep_graph::graph::DepGraph<K>::with_task_impl
  29: rustc_data_structures::stack::ensure_sufficient_stack
  30: rustc_query_system::query::plumbing::force_query_with_job
  31: rustc_query_system::query::plumbing::get_query_impl
  32: <rustc_query_impl::Queries as rustc_middle::ty::query::QueryEngine>::analysis
  33: rustc_interface::queries::<impl rustc_interface::interface::Compiler>::enter
  34: rustc_span::with_source_map
  35: rustc_interface::interface::create_compiler_and_run
  36: scoped_tls::ScopedKey<T>::set
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

[frank-king]

I got this error while compiling. It is probably because I changed the target directory in the global cargo/config.toml.

My operations are:

• First I set [build] target-dir = "A", then run cargo build, and I encountered with a write error due to no free space on my disk storage.
• Then I set [build] target-dir = "B", and move the directory A to B. Then run cargo build.
• Then the compilation (incremental) failed. From the error message, it seems that the target directory $HOME/./target/... displayed in the error message is the previous path A, instead of B.

Here is the error message. I have replaced my home directory as $HOME. Hope this be helpful. :)

error: internal compiler error: encountered incremental compilation error with predicates_of(cranelift_codegen[b615]::ir::builder::InstBuilder)
  |
  = help: This is a known issue with the compiler. Run `cargo clean -p lightjit` or `cargo clean` to allow your project to compile
  = note: Please follow the instructions below to create a bug report with the provided information
  = note: See <https://github.com/rust-lang/rust/issues/84970> for more information

thread 'rustc' panicked at 'Found unstable fingerprints for predicates_of(cranelift_codegen[b615]::ir::builder::InstBuilder): GenericPredicates { parent: None, predicates: [(Binder(TraitPredicate(<Self as cranelift::cranelift_codegen::ir::InstBuilderBase<'f>>), []), $HOME/./target/debug/build/cranelift-codegen-acf32dd415726ab4/out/inst_builder.rs:14:28: 14:47 (#0)), (Binder(TraitPredicate(<Self as cranelift::prelude::InstBuilder<'f>>), []), /home/whjpji/./target/debug/build/cranelift-codegen-acf32dd415726ab4/out/inst_builder.rs:14:1: 6205:2 (#0))] }', /rustc/e6b4c252ea33e9f80ab8b8d7b3f6393e54166127/compiler/rustc_query_system/src/query/plumbing.rs:624:9
stack backtrace:
   0: rust_begin_unwind
             at /rustc/e6b4c252ea33e9f80ab8b8d7b3f6393e54166127/library/std/src/panicking.rs:515:5
   1: std::panicking::begin_panic_fmt
             at /rustc/e6b4c252ea33e9f80ab8b8d7b3f6393e54166127/library/std/src/panicking.rs:457:5
   2: rustc_query_system::query::plumbing::incremental_verify_ich
   3: rustc_query_system::query::plumbing::load_from_disk_and_cache_in_memory
   4: rustc_query_system::query::plumbing::get_query_impl
   5: <rustc_query_impl::Queries as rustc_middle::ty::query::QueryEngine>::predicates_of
   6: rustc_middle::ty::generics::GenericPredicates::instantiate_identity_into
   7: rustc_middle::ty::generics::GenericPredicates::instantiate_identity
   8: rustc_ty_utils::ty::param_env
   9: rustc_middle::dep_graph::<impl rustc_query_system::dep_graph::DepKind for rustc_middle::dep_graph::dep_node::DepKind>::with_deps
  10: rustc_query_system::query::plumbing::load_from_disk_and_cache_in_memory
  11: rustc_data_structures::stack::ensure_sufficient_stack
  12: rustc_query_system::query::plumbing::get_query_impl
  13: <rustc_query_impl::Queries as rustc_middle::ty::query::QueryEngine>::param_env
  14: rustc_middle::ty::layout::<impl rustc_middle::ty::instance::Instance>::fn_sig_for_fn_abi
  15: <rustc_target::abi::call::FnAbi<&rustc_middle::ty::TyS> as rustc_middle::ty::layout::FnAbiExt<C>>::of_instance
  16: rustc_codegen_ssa::mir::block::<impl rustc_codegen_ssa::mir::FunctionCx<Bx>>::codegen_terminator
  17: rustc_codegen_ssa::mir::codegen_mir
  18: rustc_codegen_ssa::base::codegen_instance
  19: <rustc_middle::mir::mono::MonoItem as rustc_codegen_ssa::mono_item::MonoItemExt>::define
  20: rustc_codegen_llvm::base::compile_codegen_unit::module_codegen
  21: rustc_query_system::dep_graph::graph::DepGraph<K>::with_task
  22: rustc_codegen_llvm::base::compile_codegen_unit
  23: <rustc_codegen_llvm::LlvmCodegenBackend as rustc_codegen_ssa::traits::backend::CodegenBackend>::codegen_crate
  24: rustc_interface::passes::QueryContext::enter
  25: rustc_interface::queries::Queries::ongoing_codegen
  26: rustc_interface::queries::<impl rustc_interface::interface::Compiler>::enter
  27: rustc_span::with_source_map
  28: rustc_interface::interface::create_compiler_and_run
  29: scoped_tls::ScopedKey<T>::set
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
error: internal compiler error: unexpected panic
note: the compiler unexpectedly panicked. this is a bug.
note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md
note: rustc 1.55.0-nightly (e6b4c252e 2021-06-25) running on x86_64-unknown-linux-gnu
note: compiler flags: -C embed-bitcode=no -C debuginfo=2 -C incremental --crate-type lib
note: some of the compiler flags provided by cargo are hidden
query stack during panic:
#0 [predicates_of] computing predicates of `cranelift::prelude::InstBuilder`
#1 [param_env] computing normalized predicates of `cranelift::prelude::InstBuilder::return_`
end of query stack

[gilescope]

Stats from me: I ran into this 3-4 times in 7 days of rust dev which is bearable.

[Mark-Simulacrum]

@gilescope Can you say more about which version of Rust that's with (nightly? which nightly?) -- my impression from meetings and such was that we expect the error rate to be even lower than that.

[est31]

@Mark-Simulacrum from @gilescope 's shared ICE error message, it seems to be a recent nightly from June 22.

On a surface level, the EvaluatedToOk reads similar to #86771 which I think shouldn't have been closed.

[est31]

Most importantly, both @gilescope 's compiler and the compiler of the person who filed #86771 has included PR #85186 which has fixed EvaluatedToOk according to the table, but maybe it's a different EvaluatedToOk error... no idea.

[Aaron1011]

There's still at least one open evaluate_obligation issue: #85360

[dvdplm]

FWIW me and @gilescope work on the same code base. I run nightly, updated every morning. The number of occurrences of this ice is roughly equal over the past few weeks.

[carols10cents]

It's unclear to me based on this issue being open and me being confused about what state the trains were in when, but is incremental compilation back on by default in the just-released 1.54.0 stable? I would have expected that to be called out in the release notes either way.

EDIT: Ah, I see that incremental compilation IS back on in 1.54.0 as mentioned in the blog post now-- I checked RELEASES.md first :)

[berkus]

There are currently still two known issues which can result in an ICE.

Says the announcement and links here, but it's not clear which two it mentions and how to find them here? The table at the top shows 4 unfixed issues.

[jyn514]

@berkus "done" and "fixed" are different (done means there is a regression test). The two unfixed issues are the last two evaluate_obligation ICEs.

[apiraino]

After the release of 1.54 the situation about this issue indicates that the issue priority can be downgraded (Zulip discussion)

@rustbot label -P-critical P-high

[gilescope]

Fixing regressions should be higher priority than adding new features (E.g. higher kinded types). This still hurts people daily. (Yesterday it wasn’t a problem, today I have hit it 8 times).

…

On Thu, 5 Aug 2021 at 16:01, apiraino ***@***.***> wrote: After the release of 1.54 the situation about this issue indicates that the issue priority can be downgraded (Zulip discussion <https://rust-lang.zulipchat.com/#narrow/stream/238009-t-compiler.2Fmeetings/topic/.5Bweekly.5D.202021-08-05.20.2354818/near/248488507> ) @rustbot <https://github.com/rustbot> label -P-critical P-high — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#84970 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGEJCA5BIIQ3EMAL5KRNJDT3KRTPANCNFSM44GDKPXA> .

[XAMPPRocky]

Fixing regressions should be higher priority than adding new features (E.g. higher kinded types). This still hurts people daily.

The people who can work on new features are not the same set of people who can fix regressions, this was not caused by the introduction of a new feature so the two are unrelated. Can you include more info? As from the Zulip discussion this is the first report since being re-enabled.

[gilescope]

We don’t report already raised issues. There’s two open issues due to the regression and people are waiting patiently for these to be fixed. Meanwhile lots of cargo cleans are happening. One of those issues has a PR. I guess rather than moaning about it I can build that PR and report back if that seems to fix the issue for our codebase. I will do that this week - sorry should have thought of that earlier.

…

On Fri, 6 Aug 2021 at 18:18, XAMPPRocky ***@***.***> wrote: Fixing regressions should be higher priority than adding new features (E.g. higher kinded types). This still hurts people daily. The people who can work on new features are not the same set of people who can fix regressions, this was not caused by the introduction of a new feature so the two are unrelated. Can you include more info? As from the Zulip discussion this is the first report since being re-enabled. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#84970 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGEJCBNCA4JLEDY3QUSFO3T3QKPHANCNFSM44GDKPXA> .

[Aaron1011]

All of the known issues should now be fixed in the latest nightly (with the exception of #84963, which should be both rare and easy-to-fix). However, not all of them have corresponding test cases.

[gj]

All of the known issues should now be fixed in the latest nightly (with the exception of #84963, which should be both rare and easy-to-fix). However, not all of them have corresponding test cases.

Given this update, is it still helpful/encouraged to create a bug report when encountering this issue on 1.54 stable? Not sure if it would be helpful for coming up with corresponding test cases or redundant.

[Aaron1011]

It's not always obvious whether or not a particulsr incremental ICE is fixed in nightly. I think it's fine to have people keep opening issues, which will l(hopefully) all get closed as duplicates. As new versions are released and people upgrade, the rate of new issues should hopefully fall to zero.

[SwishSwushPow]

Hey everyone, so it seems that I still hit this error on nightly-2021-09-12. I get the following error message:
thread 'rustc' panicked at 'Found unstable fingerprints for evaluate_obligation(c852859f41286a6-7af6d3069c3e430d): Ok(EvaluatedToOkModuloRegions)', /rustc/8c2b6ea37d7719a0370bd404030eef9702c1752c/compiler/rustc_query_system/src/query/plumbing.rs:632:13

From the table above and the other issues it seems that this particular ICE should be fixed?

note: rustc 1.57.0-nightly (8c2b6ea37 2021-09-11) running on x86_64-apple-darwin
note: compiler flags: -C embed-bitcode=no -C split-debuginfo=unpacked -C debuginfo=2 -C incremental

If helpful I can try to provide more information. Creating a minimum viable example is tricky since the code base in question is quite complex and closed source. Are there any recommended ways to proceed?

[Aaron1011]

@MrMuetze Could you provide the full backtrace from the error message?

[SwishSwushPow]

I hope you mean this :) If not just let me know!

error: internal compiler error: encountered incremental compilation error with evaluate_obligation(c852859f41286a6-7af6d3069c3e430d)
  |
  = help: This is a known issue with the compiler. Run `cargo clean -p lib` or `cargo clean` to allow your project to compile
  = note: Please follow the instructions below to create a bug report with the provided information
  = note: See <https://github.com/rust-lang/rust/issues/84970> for more information

thread 'rustc' panicked at 'Found unstable fingerprints for evaluate_obligation(c852859f41286a6-7af6d3069c3e430d): Ok(EvaluatedToOkModuloRegions)', /rustc/8c2b6ea37d7719a0370bd404030eef9702c1752c/compiler/rustc_query_system/src/query/plumbing.rs:632:13
stack backtrace:
   0: _rust_begin_unwind
   1: std::panicking::begin_panic_fmt
   2: rustc_query_system::query::plumbing::incremental_verify_ich
   3: rustc_query_system::query::plumbing::try_load_from_disk_and_cache_in_memory
   4: rustc_query_system::query::plumbing::get_query
   5: <rustc_query_impl::Queries as rustc_middle::ty::query::QueryEngine>::evaluate_obligation
   6: <rustc_infer::infer::InferCtxt as rustc_trait_selection::traits::query::evaluate_obligation::InferCtxtExt>::evaluate_obligation
   7: <rustc_infer::infer::InferCtxt as rustc_trait_selection::traits::query::evaluate_obligation::InferCtxtExt>::evaluate_obligation_no_overflow
   8: rustc_trait_selection::traits::fulfill::FulfillProcessor::process_trait_obligation
   9: rustc_trait_selection::traits::fulfill::FulfillProcessor::progress_changed_obligations
  10: rustc_data_structures::obligation_forest::ObligationForest<O>::process_obligations
  11: rustc_trait_selection::traits::fulfill::FulfillmentContext::select
  12: <rustc_trait_selection::traits::fulfill::FulfillmentContext as rustc_infer::traits::engine::TraitEngine>::select_where_possible
  13: <rustc_infer::infer::InferCtxtBuilder as rustc_trait_selection::infer::InferCtxtBuilderExt>::enter_canonical_trait_query
  14: rustc_traits::normalize_projection_ty::normalize_projection_ty
  15: rustc_middle::dep_graph::<impl rustc_query_system::dep_graph::DepKind for rustc_middle::dep_graph::dep_node::DepKind>::with_deps
  16: rustc_query_system::query::plumbing::try_load_from_disk_and_cache_in_memory
  17: rustc_query_system::query::plumbing::get_query
  18: <rustc_query_impl::Queries as rustc_middle::ty::query::QueryEngine>::normalize_projection_ty
  19: <rustc_trait_selection::traits::query::normalize::QueryNormalizer as rustc_middle::ty::fold::TypeFolder>::fold_ty
  20: rustc_middle::ty::structural_impls::<impl rustc_middle::ty::fold::TypeFoldable for rustc_middle::ty::sty::Binder<T>>::fold_with
  21: rustc_middle::ty::util::fold_list
  22: rustc_middle::ty::structural_impls::<impl rustc_middle::ty::fold::TypeFoldable for &rustc_middle::ty::TyS>::super_fold_with
  23: <rustc_trait_selection::traits::query::normalize::QueryNormalizer as rustc_middle::ty::fold::TypeFolder>::fold_ty
  24: rustc_middle::ty::subst::<impl rustc_middle::ty::fold::TypeFoldable for &rustc_middle::ty::list::List<rustc_middle::ty::subst::GenericArg>>::super_fold_with
  25: rustc_middle::ty::structural_impls::<impl rustc_middle::ty::fold::TypeFoldable for &rustc_middle::ty::TyS>::super_fold_with
  26: <rustc_trait_selection::traits::query::normalize::QueryNormalizer as rustc_middle::ty::fold::TypeFolder>::fold_ty
  27: <rustc_infer::infer::at::At as rustc_trait_selection::traits::query::normalize::AtExt>::normalize
  28: rustc_infer::infer::InferCtxtBuilder::enter
  29: core::ops::function::FnOnce::call_once
  30: rustc_middle::dep_graph::<impl rustc_query_system::dep_graph::DepKind for rustc_middle::dep_graph::dep_node::DepKind>::with_deps
  31: rustc_query_system::query::plumbing::try_load_from_disk_and_cache_in_memory
  32: rustc_query_system::query::plumbing::get_query
  33: rustc_middle::ty::normalize_erasing_regions::NormalizeAfterErasingRegionsFolder::normalize_generic_arg_after_erasing_regions
  34: <rustc_middle::ty::normalize_erasing_regions::NormalizeAfterErasingRegionsFolder as rustc_middle::ty::fold::TypeFolder>::fold_ty
  35: <rustc_monomorphize::collector::MirNeighborCollector as rustc_middle::mir::visit::Visitor>::visit_rvalue
  36: rustc_monomorphize::collector::collect_neighbours
  37: rustc_monomorphize::collector::collect_items_rec
  38: rustc_monomorphize::collector::collect_items_rec
  39: rustc_monomorphize::collector::collect_items_rec
  40: rustc_monomorphize::collector::collect_items_rec
  41: rustc_monomorphize::collector::collect_items_rec
  42: rustc_monomorphize::collector::collect_items_rec
  43: rustc_monomorphize::collector::collect_items_rec
  44: rustc_monomorphize::collector::collect_items_rec
  45: rustc_monomorphize::collector::collect_items_rec
  46: rustc_monomorphize::collector::collect_items_rec
  47: rustc_monomorphize::collector::collect_items_rec
  48: rustc_monomorphize::collector::collect_items_rec
  49: rustc_monomorphize::collector::collect_items_rec
  50: rustc_monomorphize::collector::collect_items_rec
  51: rustc_session::utils::<impl rustc_session::session::Session>::time
  52: rustc_monomorphize::collector::collect_crate_mono_items
  53: rustc_monomorphize::partitioning::collect_and_partition_mono_items
  54: rustc_query_system::dep_graph::graph::DepGraph<K>::with_task
  55: rustc_data_structures::stack::ensure_sufficient_stack
  56: rustc_query_system::query::plumbing::try_execute_query
  57: rustc_query_system::query::plumbing::get_query
  58: <rustc_query_impl::Queries as rustc_middle::ty::query::QueryEngine>::collect_and_partition_mono_items
  59: rustc_codegen_ssa::base::codegen_crate
  60: <rustc_codegen_llvm::LlvmCodegenBackend as rustc_codegen_ssa::traits::backend::CodegenBackend>::codegen_crate
  61: rustc_interface::queries::Queries::ongoing_codegen
  62: rustc_interface::queries::<impl rustc_interface::interface::Compiler>::enter
  63: rustc_span::with_source_map
  64: scoped_tls::ScopedKey<T>::set
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
error: internal compiler error: unexpected panic
note: the compiler unexpectedly panicked. this is a bug.
note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md
note: rustc 1.57.0-nightly (8c2b6ea37 2021-09-11) running on x86_64-apple-darwin
note: compiler flags: -C embed-bitcode=no -C split-debuginfo=unpacked -C debuginfo=2 -C incremental
note: some of the compiler flags provided by cargo are hidden
query stack during panic:
#0 [evaluate_obligation] evaluating trait selection obligation `fn(<gst_geometry::buffered_geometry::BufferedFactory as gst_geometry::factory::ReadableGeometryDefinition>::PointSet) -> gst_geometry::ReadableGeometry<gst_geometry::buffered_geometry::BufferedFactory> {gst_geometry::ReadableGeometry::<gst_geometry::buffered_geometry::BufferedFactory>::PointSet}: std::ops::FnOnce<(gst_geometry::buffered_geometry::BufferedPointSet,)>`
#1 [normalize_projection_ty] normalizing `Canonical { max_universe: U0, variables: [], value: ParamEnvAnd { param_env: ParamEnv { caller_bounds: [], reveal: All }, value: ProjectionTy { substs: [futures::future::map::Map<futures::future::result_::FutureResult<gst_geometry::buffered_geometry::BufferedPointSet, gst_geometry::errors::Error>, fn(<gst_geometry::buffered_geometry::BufferedFactory as gst_geometry::factory::ReadableGeometryDefinition>::PointSet) -> gst_geometry::ReadableGeometry<gst_geometry::buffered_geometry::BufferedFactory> {gst_geometry::ReadableGeometry::<gst_geometry::buffered_geometry::BufferedFactory>::PointSet}>], item_def_id: DefId(100:731 ~ futures[f238]::future::Future::Item) } } }`
#2 [normalize_generic_arg_after_erasing_regions] normalizing `std::boxed::Box<dyn futures::future::Future<Item = <futures::future::map::Map<futures::future::result_::FutureResult<gst_geometry::buffered_geometry::BufferedPointSet, gst_geometry::errors::Error>, fn(<gst_geometry::buffered_geometry::BufferedFactory as gst_geometry::factory::ReadableGeometryDefinition>::PointSet) -> gst_geometry::ReadableGeometry<gst_geometry::buffered_geometry::BufferedFactory> {gst_geometry::ReadableGeometry::<gst_geometry::buffered_geometry::BufferedFactory>::PointSet}> as futures::future::Future>::Item, Error = <futures::future::map::Map<futures::future::result_::FutureResult<gst_geometry::buffered_geometry::BufferedPointSet, gst_geometry::errors::Error>, fn(<gst_geometry::buffered_geometry::BufferedFactory as gst_geometry::factory::ReadableGeometryDefinition>::PointSet) -> gst_geometry::ReadableGeometry<gst_geometry::buffered_geometry::BufferedFactory> {gst_geometry::ReadableGeometry::<gst_geometry::buffered_geometry::BufferedFactory>::PointSet}> as futures::future::Future>::Error> + std::marker::Send>`
#3 [collect_and_partition_mono_items] collect_and_partition_mono_items
end of query stack
error: aborting due to previous error

error: could not compile `gst-storage` due to 2 previous errors
Process finished with exit code 101

[Aaron1011]

@MrMuetze There should be more to the message - can you post the full compiler output?

[SwishSwushPow]

@Aaron1011 oh yes you are right, my apologies, I have edited my previous post.

[imlvts]

Tagging #88910 here, the backtrace seems to be similar to the one provided by @MrMuetze

[rubdos]

I have a case running in my public CI, rustc 1.55.0: https://gitlab.com/whisperfish/whisperfish/-/jobs/1602196024
Code is AGPLv3, so feel free to toy (and if necessary, send me a mail for an exception).

Seems it cannot get over the test build of a proc macro. Shall I try to open a bug report for this?

[Aaron1011]

@rubdos: Opening a new issue would be great, thanks.

[SwishSwushPow]

I just wanted to follow up on my reported issue. It seems that the merging of #89049 has solved my issue. I can't reproduce the error on nightly-2021-09-20. Thank you very much for this quick solution :)

[Aaron1011]

@MrMuetze That PR won't fix any issues by itself - it will just cause new ICEs if a different kind of compiler bug happens :)

Upgrading the compiler will automatically cause the incremental cache to be cleared, so that's why the issue went away for you. If it happens again (or you get a different ICE) using the newer compiler, please let me know,

[SwishSwushPow]

I have no in-depth knowledge of the compiler or incremental compilation, but I've tried a local code change that would lead to an ICE with the old nightly version (building -> changing code -> building again -> ICE) and now everything seems to work (building -> changing code -> building again -> no ICE). So I'm not sure if the situation is as you describe it. I think my repeated building should create a new incremental cache.

[iacore]

I came across this bug when disk is full during compilation. #89300
When I move the whole project to another disk and continue compilation, the compiler panics.

It's fixed after the crates that was compiling when the disk became full have their cache cleared with cargo clean -p <crate>.

This should be easy to reproduce with tmpfs with limited space.

[albertodiazdurana]

Just got an error the brought me here:

cargo build

   Compiling runtime-main v2.0.12 (/root/chain/runtimes/main)
   Compiling runtime-staking v2.0.12 (/root/chain/runtimes/staking)
   Compiling nodle-chain v2.0.12 (/root/chain/node)
error: internal compiler error: encountered incremental compilation error with evaluate_obligation(76eb3d63ecdbd736-255d01c12a95011a)
  |
  = help: This is a known issue with the compiler. Run `cargo clean -p nodle_chain` or `cargo clean` to allow your project to compile
  = note: Please follow the instructions below to create a bug report with the provided information
  = note: See <https://github.com/rust-lang/rust/issues/84970> for more information

thread 'rustc' panicked at 'Found unstable fingerprints for evaluate_obligation(76eb3d63ecdbd736-255d01c12a95011a): Ok(EvaluatedToOkModuloRegions)', /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/compiler/rustc_query_system/src/query/plumbing.rs:644:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: rustc 1.56.1 (59eed8a2a 2021-11-01) running on x86_64-unknown-linux-gnu

note: compiler flags: -C embed-bitcode=no -C debuginfo=2 -C incremental --crate-type bin

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
#0 [evaluate_obligation] evaluating trait selection obligation `sc_finality_grandpa::until_imported::BlockGlobalMessage<sp_runtime::generic::Block<sp_runtime::generic::Header<u32, sp_runtime::traits::BlakeTwo256>, sp_runtime::OpaqueExtrinsic>>: std::marker::Sized`
#1 [normalize_projection_ty] normalizing `Canonical { max_universe: U0, variables: [], value: ParamEnvAnd { param_env: ParamEnv { caller_bounds: [], reveal: All }, value: ProjectionTy { substs: [sc_finality_grandpa::until_imported::UntilImported<sp_runtime::generic::Block<sp_runtime::generic::Header<u32, sp_runtime::traits::BlakeTwo256>, sp_runtime::OpaqueExtrinsic>, sc_service::Arc<sc_service::client::client::Client<sc_client_db::Backend<sp_runtime::generic::Block<sp_runtime::generic::Header<u32, sp_runtime::traits::BlakeTwo256>, sp_runtime::OpaqueExtrinsic>>, sc_service::LocalCallExecutor<sc_client_db::Backend<sp_runtime::generic::Block<sp_runtime::generic::Header<u32, sp_runtime::traits::BlakeTwo256>, sp_runtime::OpaqueExtrinsic>>, sc_executor::NativeExecutor<service::Executor>>, sp_runtime::generic::Block<sp_runtime::generic::Header<u32, sp_runtime::traits::BlakeTwo256>, sp_runtime::OpaqueExtrinsic>, runtime_main::RuntimeApi>>, sc_finality_grandpa::communication::NetworkBridge<sp_runtime::generic::Block<sp_runtime::generic::Header<u32, sp_runtime::traits::BlakeTwo256>, sp_runtime::OpaqueExtrinsic>, sc_service::Arc<sc_network::NetworkService<sp_runtime::generic::Block<sp_runtime::generic::Header<u32, sp_runtime::traits::BlakeTwo256>, sp_runtime::OpaqueExtrinsic>, sp_core::H256>>>, futures::stream::FilterMap<futures::stream::FilterMap<futures::futures_channel::mpsc::Receiver<sc_network_gossip::state_machine::TopicNotification>, futures::future::Ready<std::option::Option<(sc_network_gossip::state_machine::TopicNotification, sc_finality_grandpa::communication::gossip::GossipMessage<sp_runtime::generic::Block<sp_runtime::generic::Header<u32, sp_runtime::traits::BlakeTwo256>, sp_runtime::OpaqueExtrinsic>>)>>, [closure@sc_finality_grandpa::communication::incoming_global<sp_runtime::generic::Block<sp_runtime::generic::Header<u32, sp_runtime::traits::BlakeTwo256>, sp_runtime::OpaqueExtrinsic>>::{closure#2}]>, futures::future::Ready<std::option::Option<finality_grandpa::voter::CommunicationIn<sp_core::H256, u32, sp_finality_grandpa::app::Signature, sp_finality_grandpa::app::Public>>>, [closure@sc_finality_grandpa::communication::incoming_global<sp_runtime::generic::Block<sp_runtime::generic::Header<u32, sp_runtime::traits::BlakeTwo256>, sp_runtime::OpaqueExtrinsic>>::{closure#3}]>, sc_finality_grandpa::until_imported::BlockGlobalMessage<sp_runtime::generic::Block<sp_runtime::generic::Header<u32, sp_runtime::traits::BlakeTwo256>, sp_runtime::OpaqueExtrinsic>>>], item_def_id: DefId(170:128 ~ futures_core[b2f6]::stream::TryStream::Ok) } } }`
end of query stack
error: could not compile `nodle-chain` due to previous error

[kwvg]

error: internal compiler error: encountered incremental compilation error with evaluate_obligation(c09bbb65c5dc929e-8de1b6271f955983)
  |
  = help: This is a known issue with the compiler. Run `cargo clean -p node_template` or `cargo clean` to allow your project to compile
  = note: Please follow the instructions below to create a bug report with the provided information
  = note: See <https://github.com/rust-lang/rust/issues/84970> for more information

Alright then, reporting for duty!

thread 'rustc' panicked at 'Found unstable fingerprints for evaluate_obligation(c09bbb65c5dc929e-8de1b6271f955983): Ok(EvaluatedToOkModuloRegions)', /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/compiler/rustc_query_system/src/query/plumbing.rs:644:13
stack backtrace:
   0:     0x7f7e3d83951c - std::backtrace_rs::backtrace::libunwind::trace::h3fea1eb2e0ba2ac9
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/../../backtrace/src/backtrace/libunwind.rs:90:5
   1:     0x7f7e3d83951c - std::backtrace_rs::backtrace::trace_unsynchronized::h849d83492cbc0d59
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x7f7e3d83951c - std::sys_common::backtrace::_print_fmt::he3179d37290f23d3
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/sys_common/backtrace.rs:67:5
   3:     0x7f7e3d83951c - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h140f6925cad14324
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/sys_common/backtrace.rs:46:22
   4:     0x7f7e3d89722c - core::fmt::write::h31b9cd1bedd7ea38
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/core/src/fmt/mod.rs:1150:17
   5:     0x7f7e3d82a5a5 - std::io::Write::write_fmt::h1fdf66f83f70913e
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/io/mod.rs:1667:15
   6:     0x7f7e3d83ca80 - std::sys_common::backtrace::_print::he7ac492cd19c3189
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/sys_common/backtrace.rs:49:5
   7:     0x7f7e3d83ca80 - std::sys_common::backtrace::print::hba20f8920229d8e8
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/sys_common/backtrace.rs:36:9
   8:     0x7f7e3d83ca80 - std::panicking::default_hook::{{closure}}::h714d63979ae18678
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:210:50
   9:     0x7f7e3d83c637 - std::panicking::default_hook::hf1afb64e69563ca8
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:227:9
  10:     0x7f7e3e011b01 - rustc_driver::DEFAULT_HOOK::{{closure}}::{{closure}}::h045e68b20488b66e
  11:     0x7f7def30aab3 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h3aeb9c642834e828
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/alloc/src/boxed.rs:1650:9
  12:     0x7f7def3099cd - proc_macro::bridge::client::<impl proc_macro::bridge::Bridge>::enter::{{closure}}::{{closure}}::h82b1718cfe0673ae
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/proc_macro/src/bridge/client.rs:320:21
  13:     0x7f7e3d83d299 - std::panicking::rust_panic_with_hook::h02231a501e274a13
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:628:17
  14:     0x7f7e3d83cd50 - std::panicking::begin_panic_handler::{{closure}}::h5b5e738edf361af7
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:521:13
  15:     0x7f7e3d8399e4 - std::sys_common::backtrace::__rust_end_short_backtrace::h601a115def7987b1
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/sys_common/backtrace.rs:141:18
  16:     0x7f7e3d83ccb9 - rust_begin_unwind
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:517:5
  17:     0x7f7e3d80183b - std::panicking::begin_panic_fmt::hc15cd62c02c70986
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/panicking.rs:460:5
  18:     0x7f7e400bc87e - rustc_query_system::query::plumbing::incremental_verify_ich::he2d3f7a2e4187fa8
  19:     0x7f7e400c3c58 - rustc_query_system::query::plumbing::load_from_disk_and_cache_in_memory::h57d1a73da79dff93
  20:     0x7f7e3f772d06 - rustc_data_structures::stack::ensure_sufficient_stack::h24d7bbcfae6c0956
  21:     0x7f7e3f6c6b32 - rustc_query_system::query::plumbing::get_query_impl::h92d465cfb48a6d8a
  22:     0x7f7e3f749c73 - <rustc_query_impl::Queries as rustc_middle::ty::query::QueryEngine>::evaluate_obligation::hf1092e8d544f8fb5
  23:     0x7f7e3f95257b - <rustc_infer::infer::InferCtxt as rustc_trait_selection::traits::query::evaluate_obligation::InferCtxtExt>::evaluate_obligation::h1ece4ab7c62c6abf
  24:     0x7f7e3f952990 - <rustc_infer::infer::InferCtxt as rustc_trait_selection::traits::query::evaluate_obligation::InferCtxtExt>::evaluate_obligation_no_overflow::h52ee046878f392d3
  25:     0x7f7e3f969f5b - rustc_trait_selection::traits::fulfill::FulfillProcessor::process_trait_obligation::h64e29ac66859b695
  26:     0x7f7e3f968cc9 - rustc_trait_selection::traits::fulfill::FulfillProcessor::progress_changed_obligations::h28ac8af9dbf16398
  27:     0x7f7e3f97ef58 - rustc_data_structures::obligation_forest::ObligationForest<O>::process_obligations::h9e9031f4cf10c7ed
  28:     0x7f7e3f96822c - rustc_trait_selection::traits::fulfill::FulfillmentContext::select::hac008eae0bfd35c5
  29:     0x7f7e3f968896 - <rustc_trait_selection::traits::fulfill::FulfillmentContext as rustc_infer::traits::engine::TraitEngine>::select_where_possible::h1d14f6d42520e0f7
  30:     0x7f7e400008bf - <rustc_infer::infer::InferCtxtBuilder as rustc_trait_selection::infer::InferCtxtBuilderExt>::enter_canonical_trait_query::h5a6a7ef8d63f823a
  31:     0x7f7e40008487 - rustc_traits::normalize_projection_ty::normalize_projection_ty::h8fba200e8f45307c
  32:     0x7f7e40180a25 - rustc_middle::dep_graph::<impl rustc_query_system::dep_graph::DepKind for rustc_middle::dep_graph::dep_node::DepKind>::with_deps::h25a9525e5e4e1bdb
  33:     0x7f7e400c4fcd - rustc_query_system::query::plumbing::load_from_disk_and_cache_in_memory::h73e0d50fc4cd58fb
  34:     0x7f7e401e204c - rustc_data_structures::stack::ensure_sufficient_stack::h1b75f672e3eae3e6
  35:     0x7f7e4004d14b - rustc_query_system::query::plumbing::get_query_impl::ha36ab5802c406d74
  36:     0x7f7e401a22a7 - <rustc_query_impl::Queries as rustc_middle::ty::query::QueryEngine>::normalize_projection_ty::h6c6bba2e67550ec7
  37:     0x7f7e3f9ac5be - <rustc_trait_selection::traits::query::normalize::QueryNormalizer as rustc_middle::ty::fold::TypeFolder>::fold_ty::h28a2580a6d984896
  38:     0x7f7e3f963eae - rustc_middle::ty::subst::<impl rustc_middle::ty::fold::TypeFoldable for &rustc_middle::ty::list::List<rustc_middle::ty::subst::GenericArg>>::super_fold_with::h2848a4433a086f07
  39:     0x7f7e3f9a308a - rustc_middle::ty::structural_impls::<impl rustc_middle::ty::fold::TypeFoldable for &rustc_middle::ty::TyS>::super_fold_with::h7e4e850224585fe4
  40:     0x7f7e3f9ac06e - <rustc_trait_selection::traits::query::normalize::QueryNormalizer as rustc_middle::ty::fold::TypeFolder>::fold_ty::h28a2580a6d984896
  41:     0x7f7e3f963eae - rustc_middle::ty::subst::<impl rustc_middle::ty::fold::TypeFoldable for &rustc_middle::ty::list::List<rustc_middle::ty::subst::GenericArg>>::super_fold_with::h2848a4433a086f07
  42:     0x7f7e3f9a3025 - rustc_middle::ty::structural_impls::<impl rustc_middle::ty::fold::TypeFoldable for &rustc_middle::ty::TyS>::super_fold_with::h7e4e850224585fe4
  43:     0x7f7e3f9ac06e - <rustc_trait_selection::traits::query::normalize::QueryNormalizer as rustc_middle::ty::fold::TypeFolder>::fold_ty::h28a2580a6d984896
  44:     0x7f7e3f6a6669 - <rustc_infer::infer::at::At as rustc_trait_selection::traits::query::normalize::AtExt>::normalize::h1e909a87e446ce35
  45:     0x7f7e3f69badd - rustc_infer::infer::InferCtxtBuilder::enter::h8a82609332438ccf
  46:     0x7f7e3f6a84a8 - core::ops::function::FnOnce::call_once::hf98335709d8238cc
  47:     0x7f7e40187706 - rustc_middle::dep_graph::<impl rustc_query_system::dep_graph::DepKind for rustc_middle::dep_graph::dep_node::DepKind>::with_deps::hc710547e85ad8785
  48:     0x7f7e400cc7ac - rustc_query_system::query::plumbing::load_from_disk_and_cache_in_memory::hf74272c377e8ce4d
  49:     0x7f7e401f7b1a - rustc_data_structures::stack::ensure_sufficient_stack::hcd34f14ecbbc82e6
  50:     0x7f7e3f6ade87 - rustc_query_system::query::plumbing::get_query_impl::h1a6c0649c5ddb30e
  51:     0x7f7e3f749aaf - <rustc_query_impl::Queries as rustc_middle::ty::query::QueryEngine>::normalize_generic_arg_after_erasing_regions::h9a080611e25ef801
  52:     0x7f7e3fa6620b - rustc_middle::ty::normalize_erasing_regions::NormalizeAfterErasingRegionsFolder::normalize_generic_arg_after_erasing_regions::h4a5e623a5f9ea400
  53:     0x7f7e3fa66476 - <rustc_middle::ty::normalize_erasing_regions::NormalizeAfterErasingRegionsFolder as rustc_middle::ty::fold::TypeFolder>::fold_ty::h225ddd65985624e6
  54:     0x7f7e3f574aa2 - <rustc_mir::monomorphize::collector::MirNeighborCollector as rustc_middle::mir::visit::Visitor>::visit_terminator::h1d71390460fda2f1
  55:     0x7f7e3f577dc6 - rustc_mir::monomorphize::collector::collect_neighbours::hed29b95c637a5f56
  56:     0x7f7e3f571488 - rustc_mir::monomorphize::collector::collect_items_rec::hf80955cb546dcdb1
  57:     0x7f7e3f5715dd - rustc_mir::monomorphize::collector::collect_items_rec::hf80955cb546dcdb1
  58:     0x7f7e3f5715dd - rustc_mir::monomorphize::collector::collect_items_rec::hf80955cb546dcdb1
  59:     0x7f7e3f5715dd - rustc_mir::monomorphize::collector::collect_items_rec::hf80955cb546dcdb1
  60:     0x7f7e3f5715dd - rustc_mir::monomorphize::collector::collect_items_rec::hf80955cb546dcdb1
  61:     0x7f7e3f5715dd - rustc_mir::monomorphize::collector::collect_items_rec::hf80955cb546dcdb1
  62:     0x7f7e3f5715dd - rustc_mir::monomorphize::collector::collect_items_rec::hf80955cb546dcdb1
  63:     0x7f7e3f5715dd - rustc_mir::monomorphize::collector::collect_items_rec::hf80955cb546dcdb1
  64:     0x7f7e3f5715dd - rustc_mir::monomorphize::collector::collect_items_rec::hf80955cb546dcdb1
  65:     0x7f7e3f5715dd - rustc_mir::monomorphize::collector::collect_items_rec::hf80955cb546dcdb1
  66:     0x7f7e3f5715dd - rustc_mir::monomorphize::collector::collect_items_rec::hf80955cb546dcdb1
  67:     0x7f7e3f5715dd - rustc_mir::monomorphize::collector::collect_items_rec::hf80955cb546dcdb1
  68:     0x7f7e3f5715dd - rustc_mir::monomorphize::collector::collect_items_rec::hf80955cb546dcdb1
  69:     0x7f7e3f5715dd - rustc_mir::monomorphize::collector::collect_items_rec::hf80955cb546dcdb1
  70:     0x7f7e3f5715dd - rustc_mir::monomorphize::collector::collect_items_rec::hf80955cb546dcdb1
  71:     0x7f7e3f5715dd - rustc_mir::monomorphize::collector::collect_items_rec::hf80955cb546dcdb1
  72:     0x7f7e3f5715dd - rustc_mir::monomorphize::collector::collect_items_rec::hf80955cb546dcdb1
  73:     0x7f7e3f5715dd - rustc_mir::monomorphize::collector::collect_items_rec::hf80955cb546dcdb1
  74:     0x7f7e3f5715dd - rustc_mir::monomorphize::collector::collect_items_rec::hf80955cb546dcdb1
  75:     0x7f7e3f5715dd - rustc_mir::monomorphize::collector::collect_items_rec::hf80955cb546dcdb1
  76:     0x7f7e3f5715dd - rustc_mir::monomorphize::collector::collect_items_rec::hf80955cb546dcdb1
  77:     0x7f7e3fff0093 - rustc_session::utils::<impl rustc_session::session::Session>::time::h35490e2ae3f5aa78
  78:     0x7f7e3ff52d7a - rustc_mir::monomorphize::collector::collect_crate_mono_items::h635d08e9dc2d7947
  79:     0x7f7e3f578734 - rustc_mir::monomorphize::partitioning::collect_and_partition_mono_items::h4de3f1583dec98df
  80:     0x7f7e3e9b792a - rustc_middle::dep_graph::<impl rustc_query_system::dep_graph::DepKind for rustc_middle::dep_graph::dep_node::DepKind>::with_deps::h0e1379fcba4fd362
  81:     0x7f7e3e99f67d - rustc_query_system::dep_graph::graph::DepGraph<K>::with_task_impl::he354a490f738167c
  82:     0x7f7e3ea1cf34 - rustc_data_structures::stack::ensure_sufficient_stack::h545e76dc4817c8a9
  83:     0x7f7e3e7f27e1 - rustc_query_system::query::plumbing::force_query_with_job::haabd4665d837068b
  84:     0x7f7e400459cc - rustc_query_system::query::plumbing::get_query_impl::h83d1cf19ead34f49
  85:     0x7f7e401a1ed6 - <rustc_query_impl::Queries as rustc_middle::ty::query::QueryEngine>::collect_and_partition_mono_items::h68aa999c3a2af7d3
  86:     0x7f7e3fbae1a2 - rustc_codegen_ssa::base::codegen_crate::hb01224a2f93ef3dd
  87:     0x7f7e3fbe278a - <rustc_codegen_llvm::LlvmCodegenBackend as rustc_codegen_ssa::traits::backend::CodegenBackend>::codegen_crate::hf70c3f5210775c77
  88:     0x7f7e3fb4ee07 - rustc_interface::queries::Queries::ongoing_codegen::h5a31308287e8b4f3
  89:     0x7f7e3fb23cd2 - rustc_interface::queries::<impl rustc_interface::interface::Compiler>::enter::hff7e36ca6c2085f0
  90:     0x7f7e3fb1102c - rustc_span::with_source_map::he4ccdc3d435ea73b
  91:     0x7f7e3fb235cc - scoped_tls::ScopedKey<T>::set::hf1cc160e90acdba2
  92:     0x7f7e3fb1273b - std::sys_common::backtrace::__rust_begin_short_backtrace::h0b3064789b6c03d4
  93:     0x7f7e3fb0f905 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h3907954af9fcbd87
  94:     0x7f7e3d8498e3 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hf2a5e508372e2387
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/alloc/src/boxed.rs:1636:9
  95:     0x7f7e3d8498e3 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h46642f5b75478456
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/alloc/src/boxed.rs:1636:9
  96:     0x7f7e3d8498e3 - std::sys::unix::thread::Thread::new::thread_start::h1fbdb50adbde1cab
                               at /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/std/src/sys/unix/thread.rs:106:17
  97:     0x7f7e3d770609 - start_thread
  98:     0x7f7e3d684293 - clone
  99:                0x0 - <unknown>

---

System Information

$ cargo --version
cargo 1.56.0 (4ed5d137b 2021-10-04)

$ rustup show
Default host: x86_64-unknown-linux-gnu
rustup home:  /home/kitty/.rustup

installed toolchains
--------------------

stable-x86_64-unknown-linux-gnu (default)
nightly-x86_64-unknown-linux-gnu

active toolchain
----------------

stable-x86_64-unknown-linux-gnu (default)
rustc 1.56.1 (59eed8a2a 2021-11-01)

$ uname -ar
Linux kitty 5.11.0-41-generic #45~20.04.1-Ubuntu SMP Wed Nov 10 10:20:10 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Zorin
Description:	Zorin OS 16
Release:	16
Codename:	focal

[ogoffart]

Small side issue in the error message text: the project name in the suggestion is not correct because the dashes are replaced by underscore.
For example, the error message contains this line

  = help: This is a known issue with the compiler. Run `cargo clean -p foo_bar` or `cargo clean` to allow your project to compile

but then, typing

> cargo clean -p foo_bar
error: package ID specification `foo_bar` did not match any packages

        Did you mean `foo-bar`?

[AurevoirXavier]

Stalled?

[bjorn3]

AFAIK we are still fixing and have recently fixed some of the bugs. I just think people forgot to link back to this issue.

[neysofu]

Seems like there's another evaluate_obligation(EvaluatedToOk/EvaluatedToOkModuloRegions) ICE. It's tracked in #91114.

[Mingun]

Encounter another ICE -- #99774.

[pnkfelix]

visiting for P-high review

I think we need to keep this open, even two weeks ago someone found a bug related to unstable fingerprints

[Noratrieb]

I hit another one, this time in associated_item: #105953

[orf]

Have the following, was triggered by running cargo fmt concurrently with cargo watch (which runs cargo check):

error: internal compiler error: encountered incremental compilation error with mir_built(9d8933088bd2de00-2b70151a887b6f9e)
  |
  = help: This is a known issue with the compiler. Run `cargo clean -p pypi_data_scraper` or `cargo clean` to allow your project to compile
  = note: Please follow the instructions below to create a bug report with the provided information
  = note: See <https://github.com/rust-lang/rust/issues/84970> for more information

(huge output, see below)

stack backtrace:
   0:        0x10fdb77a2 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hdc69a6f447628e71
   1:        0x10fe0f64a - core::fmt::write::hb9e764fa47ae8444
   2:        0x10fda987c - std::io::Write::write_fmt::h8fc98987ed860a54
   3:        0x10fdb756a - std::sys_common::backtrace::print::h488fe4c0b1fb9d50
   4:        0x10fdba976 - std::panicking::default_hook::{{closure}}::h5618ea3156b8b833
   5:        0x10fdba6c7 - std::panicking::default_hook::h0421c26a8a92801c
   6:        0x1189f5c6d - rustc_driver[5519141fff5a68e4]::DEFAULT_HOOK::{closure#0}::{closure#0}
   7:        0x10fdbb171 - std::panicking::rust_panic_with_hook::h57383cd32463c250
   8:        0x10fdbaf03 - std::panicking::begin_panic_handler::{{closure}}::h1d1f7305cfe67fdd
   9:        0x10fdb7c38 - std::sys_common::backtrace::__rust_end_short_backtrace::hd8e12e82ff026bae
  10:        0x10fdbabcd - _rust_begin_unwind
  11:        0x10fe3c0c3 - core::panicking::panic_fmt::h7894cd1015cfee41
  12:        0x11d785c46 - rustc_query_system[1d1ba73aeba80243]::query::plumbing::incremental_verify_ich_cold
  13:        0x11c267923 - rustc_query_system[1d1ba73aeba80243]::query::plumbing::incremental_verify_ich::<rustc_query_impl[37d156f36bae809d]::plumbing::QueryCtxt, rustc_middle[4717f30296835aa8]::ty::WithOptConstParam<rustc_span[1da87cb7d9d2b509]::def_id::LocalDefId>, &rustc_data_structures[42680eda0c4d9688]::steal::Steal<rustc_middle[4717f30296835aa8]::mir::Body>>
  14:        0x11c28606d - rustc_query_system[1d1ba73aeba80243]::query::plumbing::try_load_from_disk_and_cache_in_memory::<rustc_query_impl[37d156f36bae809d]::plumbing::QueryCtxt, rustc_middle[4717f30296835aa8]::ty::WithOptConstParam<rustc_span[1da87cb7d9d2b509]::def_id::LocalDefId>, &rustc_data_structures[42680eda0c4d9688]::steal::Steal<rustc_middle[4717f30296835aa8]::mir::Body>>
  15:        0x11c1a495e - rustc_query_system[1d1ba73aeba80243]::query::plumbing::try_execute_query::<rustc_query_impl[37d156f36bae809d]::plumbing::QueryCtxt, rustc_query_system[1d1ba73aeba80243]::query::caches::DefaultCache<rustc_middle[4717f30296835aa8]::ty::WithOptConstParam<rustc_span[1da87cb7d9d2b509]::def_id::LocalDefId>, &rustc_data_structures[42680eda0c4d9688]::steal::Steal<rustc_middle[4717f30296835aa8]::mir::Body>>>
  16:        0x11c2d1e65 - rustc_query_system[1d1ba73aeba80243]::query::plumbing::get_query::<rustc_query_impl[37d156f36bae809d]::queries::mir_built, rustc_query_impl[37d156f36bae809d]::plumbing::QueryCtxt>
  17:        0x11c3777a7 - <rustc_query_impl[37d156f36bae809d]::Queries as rustc_middle[4717f30296835aa8]::ty::query::QueryEngine>::mir_built
  18:        0x11b6a66a6 - rustc_mir_transform[5967269ab6e8435f]::mir_const
  19:        0x11c33db1d - <rustc_middle[4717f30296835aa8]::dep_graph::dep_node::DepKind as rustc_query_system[1d1ba73aeba80243]::dep_graph::DepKind>::with_deps::<rustc_query_system[1d1ba73aeba80243]::query::plumbing::try_load_from_disk_and_cache_in_memory<rustc_query_impl[37d156f36bae809d]::plumbing::QueryCtxt, rustc_middle[4717f30296835aa8]::ty::WithOptConstParam<rustc_span[1da87cb7d9d2b509]::def_id::LocalDefId>, &rustc_data_structures[42680eda0c4d9688]::steal::Steal<rustc_middle[4717f30296835aa8]::mir::Body>>::{closure#1}, &rustc_data_structures[42680eda0c4d9688]::steal::Steal<rustc_middle[4717f30296835aa8]::mir::Body>>
  20:        0x11c286047 - rustc_query_system[1d1ba73aeba80243]::query::plumbing::try_load_from_disk_and_cache_in_memory::<rustc_query_impl[37d156f36bae809d]::plumbing::QueryCtxt, rustc_middle[4717f30296835aa8]::ty::WithOptConstParam<rustc_span[1da87cb7d9d2b509]::def_id::LocalDefId>, &rustc_data_structures[42680eda0c4d9688]::steal::Steal<rustc_middle[4717f30296835aa8]::mir::Body>>
  21:        0x11c1a495e - rustc_query_system[1d1ba73aeba80243]::query::plumbing::try_execute_query::<rustc_query_impl[37d156f36bae809d]::plumbing::QueryCtxt, rustc_query_system[1d1ba73aeba80243]::query::caches::DefaultCache<rustc_middle[4717f30296835aa8]::ty::WithOptConstParam<rustc_span[1da87cb7d9d2b509]::def_id::LocalDefId>, &rustc_data_structures[42680eda0c4d9688]::steal::Steal<rustc_middle[4717f30296835aa8]::mir::Body>>>
  22:        0x11c2d1f92 - rustc_query_system[1d1ba73aeba80243]::query::plumbing::get_query::<rustc_query_impl[37d156f36bae809d]::queries::mir_const, rustc_query_impl[37d156f36bae809d]::plumbing::QueryCtxt>
  23:        0x11c3777d7 - <rustc_query_impl[37d156f36bae809d]::Queries as rustc_middle[4717f30296835aa8]::ty::query::QueryEngine>::mir_const
  24:        0x11b6a73c5 - rustc_mir_transform[5967269ab6e8435f]::mir_promoted
  25:        0x11c33dbed - <rustc_middle[4717f30296835aa8]::dep_graph::dep_node::DepKind as rustc_query_system[1d1ba73aeba80243]::dep_graph::DepKind>::with_deps::<rustc_query_system[1d1ba73aeba80243]::query::plumbing::try_load_from_disk_and_cache_in_memory<rustc_query_impl[37d156f36bae809d]::plumbing::QueryCtxt, rustc_middle[4717f30296835aa8]::ty::WithOptConstParam<rustc_span[1da87cb7d9d2b509]::def_id::LocalDefId>, (&rustc_data_structures[42680eda0c4d9688]::steal::Steal<rustc_middle[4717f30296835aa8]::mir::Body>, &rustc_data_structures[42680eda0c4d9688]::steal::Steal<rustc_index[21b7379004f7b6e5]::vec::IndexVec<rustc_middle[4717f30296835aa8]::mir::Promoted, rustc_middle[4717f30296835aa8]::mir::Body>>)>::{closure#1}, (&rustc_data_structures[42680eda0c4d9688]::steal::Steal<rustc_middle[4717f30296835aa8]::mir::Body>, &rustc_data_structures[42680eda0c4d9688]::steal::Steal<rustc_index[21b7379004f7b6e5]::vec::IndexVec<rustc_middle[4717f30296835aa8]::mir::Promoted, rustc_middle[4717f30296835aa8]::mir::Body>>)>
  26:        0x11c286425 - rustc_query_system[1d1ba73aeba80243]::query::plumbing::try_load_from_disk_and_cache_in_memory::<rustc_query_impl[37d156f36bae809d]::plumbing::QueryCtxt, rustc_middle[4717f30296835aa8]::ty::WithOptConstParam<rustc_span[1da87cb7d9d2b509]::def_id::LocalDefId>, (&rustc_data_structures[42680eda0c4d9688]::steal::Steal<rustc_middle[4717f30296835aa8]::mir::Body>, &rustc_data_structures[42680eda0c4d9688]::steal::Steal<rustc_index[21b7379004f7b6e5]::vec::IndexVec<rustc_middle[4717f30296835aa8]::mir::Promoted, rustc_middle[4717f30296835aa8]::mir::Body>>)>
  27:        0x11c1a5b47 - rustc_query_system[1d1ba73aeba80243]::query::plumbing::try_execute_query::<rustc_query_impl[37d156f36bae809d]::plumbing::QueryCtxt, rustc_query_system[1d1ba73aeba80243]::query::caches::DefaultCache<rustc_middle[4717f30296835aa8]::ty::WithOptConstParam<rustc_span[1da87cb7d9d2b509]::def_id::LocalDefId>, (&rustc_data_structures[42680eda0c4d9688]::steal::Steal<rustc_middle[4717f30296835aa8]::mir::Body>, &rustc_data_structures[42680eda0c4d9688]::steal::Steal<rustc_index[21b7379004f7b6e5]::vec::IndexVec<rustc_middle[4717f30296835aa8]::mir::Promoted, rustc_middle[4717f30296835aa8]::mir::Body>>)>>
  28:        0x11c2bdab8 - rustc_query_system[1d1ba73aeba80243]::query::plumbing::get_query::<rustc_query_impl[37d156f36bae809d]::queries::mir_promoted, rustc_query_impl[37d156f36bae809d]::plumbing::QueryCtxt>
  29:        0x11c377997 - <rustc_query_impl[37d156f36bae809d]::Queries as rustc_middle[4717f30296835aa8]::ty::query::QueryEngine>::mir_promoted
  30:        0x11bd4c546 - rustc_borrowck[4c7f1bc30a6a378d]::mir_borrowck
  31:        0x11bd2d1f5 - <rustc_borrowck[4c7f1bc30a6a378d]::provide::{closure#0} as core[7fe5ef5db7b67845]::ops::function::FnOnce<(rustc_middle[4717f30296835aa8]::ty::context::TyCtxt, rustc_span[1da87cb7d9d2b509]::def_id::LocalDefId)>>::call_once
  32:        0x11c4bbe58 - <rustc_query_system[1d1ba73aeba80243]::dep_graph::graph::DepGraph<rustc_middle[4717f30296835aa8]::dep_graph::dep_node::DepKind>>::with_task::<rustc_middle[4717f30296835aa8]::ty::context::TyCtxt, rustc_span[1da87cb7d9d2b509]::def_id::LocalDefId, &rustc_middle[4717f30296835aa8]::mir::query::BorrowCheckResult>
  33:        0x11c1cc6dd - rustc_query_system[1d1ba73aeba80243]::query::plumbing::try_execute_query::<rustc_query_impl[37d156f36bae809d]::plumbing::QueryCtxt, rustc_query_system[1d1ba73aeba80243]::query::caches::DefaultCache<rustc_span[1da87cb7d9d2b509]::def_id::LocalDefId, &rustc_middle[4717f30296835aa8]::mir::query::BorrowCheckResult>>
  34:        0x11c2bd823 - rustc_query_system[1d1ba73aeba80243]::query::plumbing::get_query::<rustc_query_impl[37d156f36bae809d]::queries::mir_borrowck, rustc_query_impl[37d156f36bae809d]::plumbing::QueryCtxt>
  35:        0x118a7defb - <rustc_session[f9ac50bf1960ebc9]::session::Session>::time::<(), rustc_interface[a01dc1bc136058fc]::passes::analysis::{closure#2}>
  36:        0x118acdf53 - rustc_interface[a01dc1bc136058fc]::passes::analysis
  37:        0x11c4f28da - <rustc_query_system[1d1ba73aeba80243]::dep_graph::graph::DepGraph<rustc_middle[4717f30296835aa8]::dep_graph::dep_node::DepKind>>::with_task::<rustc_middle[4717f30296835aa8]::ty::context::TyCtxt, (), core[7fe5ef5db7b67845]::result::Result<(), rustc_errors[8f127cc083e3e66e]::ErrorGuaranteed>>
  38:        0x11c25715e - rustc_query_system[1d1ba73aeba80243]::query::plumbing::try_execute_query::<rustc_query_impl[37d156f36bae809d]::plumbing::QueryCtxt, rustc_query_system[1d1ba73aeba80243]::query::caches::DefaultCache<(), core[7fe5ef5db7b67845]::result::Result<(), rustc_errors[8f127cc083e3e66e]::ErrorGuaranteed>>>
  39:        0x11c2d0ec9 - rustc_query_system[1d1ba73aeba80243]::query::plumbing::get_query::<rustc_query_impl[37d156f36bae809d]::queries::analysis, rustc_query_impl[37d156f36bae809d]::plumbing::QueryCtxt>
  40:        0x118984db6 - <rustc_interface[a01dc1bc136058fc]::passes::QueryContext>::enter::<rustc_driver[5519141fff5a68e4]::run_compiler::{closure#1}::{closure#2}::{closure#3}, core[7fe5ef5db7b67845]::result::Result<(), rustc_errors[8f127cc083e3e66e]::ErrorGuaranteed>>
  41:        0x1189c911d - rustc_span[1da87cb7d9d2b509]::with_source_map::<core[7fe5ef5db7b67845]::result::Result<(), rustc_errors[8f127cc083e3e66e]::ErrorGuaranteed>, rustc_interface[a01dc1bc136058fc]::interface::run_compiler<core[7fe5ef5db7b67845]::result::Result<(), rustc_errors[8f127cc083e3e66e]::ErrorGuaranteed>, rustc_driver[5519141fff5a68e4]::run_compiler::{closure#1}>::{closure#0}::{closure#1}>
  42:        0x1189b8a2c - <scoped_tls[91bb82b093e5659a]::ScopedKey<rustc_span[1da87cb7d9d2b509]::SessionGlobals>>::set::<rustc_interface[a01dc1bc136058fc]::interface::run_compiler<core[7fe5ef5db7b67845]::result::Result<(), rustc_errors[8f127cc083e3e66e]::ErrorGuaranteed>, rustc_driver[5519141fff5a68e4]::run_compiler::{closure#1}>::{closure#0}, core[7fe5ef5db7b67845]::result::Result<(), rustc_errors[8f127cc083e3e66e]::ErrorGuaranteed>>
  43:        0x11898834a - std[a328e7b00c61b6a2]::sys_common::backtrace::__rust_begin_short_backtrace::<rustc_interface[a01dc1bc136058fc]::util::run_in_thread_pool_with_globals<rustc_interface[a01dc1bc136058fc]::interface::run_compiler<core[7fe5ef5db7b67845]::result::Result<(), rustc_errors[8f127cc083e3e66e]::ErrorGuaranteed>, rustc_driver[5519141fff5a68e4]::run_compiler::{closure#1}>::{closure#0}, core[7fe5ef5db7b67845]::result::Result<(), rustc_errors[8f127cc083e3e66e]::ErrorGuaranteed>>::{closure#0}::{closure#0}, core[7fe5ef5db7b67845]::result::Result<(), rustc_errors[8f127cc083e3e66e]::ErrorGuaranteed>>
  44:        0x11896e44b - <<std[a328e7b00c61b6a2]::thread::Builder>::spawn_unchecked_<rustc_interface[a01dc1bc136058fc]::util::run_in_thread_pool_with_globals<rustc_interface[a01dc1bc136058fc]::interface::run_compiler<core[7fe5ef5db7b67845]::result::Result<(), rustc_errors[8f127cc083e3e66e]::ErrorGuaranteed>, rustc_driver[5519141fff5a68e4]::run_compiler::{closure#1}>::{closure#0}, core[7fe5ef5db7b67845]::result::Result<(), rustc_errors[8f127cc083e3e66e]::ErrorGuaranteed>>::{closure#0}::{closure#0}, core[7fe5ef5db7b67845]::result::Result<(), rustc_errors[8f127cc083e3e66e]::ErrorGuaranteed>>::{closure#1} as core[7fe5ef5db7b67845]::ops::function::FnOnce<()>>::call_once::{shim:vtable#0}
  45:        0x10fdc4287 - std::sys::unix::thread::Thread::new::thread_start::h7b576c3bd89f934a
  46:     0x7ff81aee3259 - __pthread_start

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: rustc 1.66.0 (69f9c33d7 2022-12-12) running on x86_64-apple-darwin

note: compiler flags: --crate-type bin -C embed-bitcode=no -C split-debuginfo=unpacked -C debuginfo=2 -C incremental=[REDACTED]

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
#0 [mir_built] building MIR for `aws::check_aws_keys`
#1 [mir_const] preparing `aws::check_aws_keys` for borrow checking
#2 [mir_promoted] processing MIR for `aws::check_aws_keys`
#3 [mir_borrowck] borrow-checking `aws::check_aws_keys`
#4 [analysis] running analysis passes on this crate

Large debug output from some object

``` thread 'rustc' panicked at 'Found unstable fingerprints for mir_built(9d8933088bd2de00-2b70151a887b6f9e): Steal { value: RwLock(RefCell { value: Some(Body { basic_blocks: BasicBlocks { basic_blocks: [BasicBlockData { statements: [StorageLive(_2), StorageLive(_3)], terminator: Some(Terminator { source_info: SourceInfo { span: src/aws.rs:12:19: 12:49 (#0), scope: scope[0] }, kind: _3 = Runtime::new() -> [return: bb1, unwind: bb41] }), is_cleanup: false }, BasicBlockData { statements: [], terminator: Some(Terminator { source_info: SourceInfo { span: src/aws.rs:12:19: 12:58 (#0), scope: scope[0] }, kind: _2 = Result::::unwrap(move _3) -> [return: bb2, unwind: bb40] }), is_cleanup: false }, BasicBlockData { statements: [StorageDead(_3), FakeRead(ForLet(None), _2), StorageLive(_4), StorageLive(_5), _5 = &_2, StorageLive(_6), StorageLive(_7), _7 = [generator@src/aws.rs:13:39: 38:6 (#0)] { matches: move _1 }], terminator: Some(Terminator { source_info: SourceInfo { span: src/aws.rs:13:33: 38:6 (#0), scope: scope[1] }, kind: _6 = std::future::from_generator::<[static generator@src/aws.rs:13:39: 38:6]>(move _7) -> [return: bb3, unwind: bb38] }), is_cleanup: false }, BasicBlockData { statements: [StorageDead(_7)], terminator: Some(Terminator { source_info: SourceInfo { span: src/aws.rs:13:19: 38:7 (#0), scope: scope[1] }, kind: _4 = Runtime::spawn::>>(move _5, move _6) -> [return: bb4, unwind: bb37] }), is_cleanup: false }, BasicBlockData { statements: [StorageDead(_6), StorageDead(_5), FakeRead(ForLet(None), _4), StorageLive(_8), StorageLive(_9), StorageLive(_10), StorageLive(_11), _11 = &_2, StorageLive(_12), _12 = move _4], terminator: Some(Terminator { source_info: SourceInfo { span: src/aws.rs:39:8: 39:33 (#0), scope: scope[2] }, kind: _10 = Runtime::block_on::>>(move _11, move _12) -> [return: bb5, unwind: bb35] }), is_cleanup: false }, BasicBlockData { statements: [StorageDead(_12), StorageDead(_11)], terminator: Some(Terminator { source_info: SourceInfo { span: src/aws.rs:39:8: 39:34 (#277), scope: scope[2] }, kind: _9 = , JoinError> as Try>::branch(move _10) -> [return: bb6, unwind: bb34] }), is_cleanup: false }, BasicBlockData { statements: [StorageDead(_10), FakeRead(ForMatchedPlace(None), _9), _13 = discriminant(_9)], terminator: Some(Terminator { source_info: SourceInfo { span: src/aws.rs:39:8: 39:34 (#277), scope: scope[2] }, kind: switchInt(move _13) -> [0_isize: bb7, 1_isize: bb8, otherwise: bb9] }), is_cleanup: false }, BasicBlockData { statements: [StorageLive(_17), _17 = move ((_9 as Continue).0: std::vec::Vec), _8 = move _17], terminator: Some(Terminator { source_info: SourceInfo { span: src/aws.rs:39:33: 39:34 (#0), scope: scope[2] }, kind: drop(_17) -> [return: bb15, unwind: bb31] }), is_cleanup: false }, BasicBlockData { statements: [], terminator: Some(Terminator { source_info: SourceInfo { span: src/aws.rs:39:33: 39:34 (#278), scope: scope[2] }, kind: falseEdge -> [real: bb10, imaginary: bb7] }), is_cleanup: false }, BasicBlockData { statements: [], terminator: Some(Terminator { source_info: SourceInfo { span: src/aws.rs:39:8: 39:34 (#277), scope: scope[2] }, kind: unreachable }), is_cleanup: false }, BasicBlockData { statements: [StorageLive(_14), _14 = move ((_9 as Break).0: std::result::Result), StorageLive(_16), _16 = move _14], terminator: Some(Terminator { source_info: SourceInfo { span: src/aws.rs:39:8: 39:34 (#277), scope: scope[4] }, kind: _0 = , anyhow::Error> as FromResidual>>::from_residual(move _16) -> [return: bb11, unwind: bb29] }), is_cleanup: false }, BasicBlockData { statements: [StorageDead(_16)], terminator: Some(Terminator { source_info: SourceInfo { span: src/aws.rs:39:33: 39:34 (#278), scope: scope[4] }, kind: goto -> bb22 }), is_cleanup: false }, BasicBlockData { statements: [], terminator: Some(Terminator { source_info: SourceInfo { span: src/aws.rs:39:33: 39:34 (#278), scope: scope[4] }, kind: unreachable }), is_cleanup: false }, BasicBlockData { statements: [], terminator: Some(Terminator { source_info: SourceInfo { span: src/aws.rs:39:33: 39:34 (#278), scope: scope[2] }, kind: drop(_14) -> [return: bb14, unwind: bb31] }), is_cleanup: false }, BasicBlockData { statements: [StorageDead(_14)], terminator: Some(Terminator { source_info: SourceInfo { span: src/aws.rs:39:33: 39:34 (#278), scope: scope[2] }, kind: goto -> bb16 }), is_cleanup: false }, BasicBlockData { statements: [StorageDead(_17)], terminator: Some(Terminator { source_info: SourceInfo { span: src/aws.rs:39:33: 39:34 (#0), scope: scope[2] }, kind: goto -> bb16 }), is_cleanup: false }, BasicBlockData { statements: [_0 = Result::, anyhow::Error>::Ok(move _8)], terminator: Some(Terminator { source_info: SourceInfo { span: src/aws.rs:39:34: 39:35 (#0), scope: scope[2] }, kind: drop(_8) -> [return: bb17, unwind: bb31] }), is_cleanup: false }, BasicBlockData { statements: [StorageDead(_8)], terminator: Some(Terminator { source_info: SourceInfo { span: src/aws.rs:40:1: 40:2 (#0), scope: scope[1] }, kind: drop(_4) -> [return: bb18, unwind: bb32] }), is_cleanup: false }, BasicBlockData { statements: [StorageDead(_4)], terminator: Some(Terminator { source_info: SourceInfo { span: src/aws.rs:40:1: 40:2 (#0), scope: scope[0] }, kind: drop(_2) -> [return: bb19, unwind: bb33] }), is_cleanup: false }, BasicBlockData { statements: [StorageDead(_2)], terminator: Some(Terminator { source_info: SourceInfo { span: src/aws.rs:40:1: 40:2 (#0), scope: scope[0] }, kind: drop(_9) -> [return: bb20, unwind: bb41] }), is_cleanup: false }, BasicBlockData { statements: [StorageDead(_9)], terminator: Some(Terminator { source_info: SourceInfo { span: src/aws.rs:40:1: 40:2 (#0), scope: scope[0] }, kind: drop(_1) -> [return: bb21, unwind: bb42] }), is_cleanup: false }, BasicBlockData { statements: [], terminator: Some(Terminator { source_info: SourceInfo { span: src/aws.rs:40:2: 40:2 (#0), scope: scope[0] }, kind: goto -> bb28 }), is_cleanup: false }, BasicBlockData { statements: [], terminator: Some(Terminator { source_info: SourceInfo { span: src/aws.rs:39:33: 39:34 (#278), scope: scope[2] }, kind: drop(_14) -> [return: bb23, unwind: bb31] }), is_cleanup: false }, BasicBlockData { statements: [StorageDead(_14), StorageDead(_8)], terminator: Some(Terminator { source_info: SourceInfo { span: src/aws.rs:40:1: 40:2 (#0), scope: scope[1] }, kind: drop(_4) -> [return: bb24, unwind: bb32] }), is_cleanup: false }, BasicBlockData { statements: [StorageDead(_4)], terminator: Some(Terminator { source_info: SourceInfo { span: src/aws.rs:40:1: 40:2 (#0), scope: scope[0] }, kind: drop(_2) -> [return: bb25, unwind: bb33] }), is_cleanup: false }, BasicBlockData { statements: [StorageDead(_2)], terminator: Some(Terminator { source_info: SourceInfo { span: src/aws.rs:40:1: 40:2 (#0), scope: scope[0] }, kind: drop(_9) -> [return: bb26, unwind: bb41] }), is_cleanup: false }, BasicBlockData { statements: [StorageDead(_9)], terminator: Some(Terminator { source_info: SourceInfo { span: src/aws.rs:40:1: 40:2 (#0), scope: scope[0] }, kind: drop(_1) -> [return: bb27, unwind: bb42] }), is_cleanup: false }, BasicBlockData { statements: [], terminator: Some(Terminator { source_info: SourceInfo { span: src/aws.rs:40:2: 40:2 (#0), scope: scope[0] }, kind: goto -> bb28 }), is_cleanup: false }, BasicBlockData { statements: [], terminator: Some(Terminator { source_info: SourceInfo { span: src/aws.rs:40:2: 40:2 (#0), scope: scope[0] }, kind: return }), is_cleanup: false }, BasicBlockData { statements: [], terminator: Some(Terminator { source_info: SourceInfo { span: src/aws.rs:39:33: 39:34 (#277), scope: scope[4] }, kind: drop(_16) -> bb30 }), is_cleanup: true }, BasicBlockData { statements: [], terminator: Some(Terminator { source_info: SourceInfo { span: src/aws.rs:39:33: 39:34 (#278), scope: scope[2] }, kind: drop(_14) -> bb31 }), is_cleanup: true }, BasicBlockData { statements: [], terminator: Some(Terminator { source_info: SourceInfo { span: src/aws.rs:40:1: 40:2 (#0), scope: scope[1] }, kind: drop(_4) -> bb32 }), is_cleanup: true }, BasicBlockData { statements: [], terminator: Some(Terminator { source_info: SourceInfo { span: src/aws.rs:40:1: 40:2 (#0), scope: scope[0] }, kind: drop(_2) -> bb33 }), is_cleanup: true }, BasicBlockData { statements: [], terminator: Some(Terminator { source_info: SourceInfo { span: src/aws.rs:40:1: 40:2 (#0), scope: scope[0] }, kind: drop(_9) -> bb41 }), is_cleanup: true }, BasicBlockData { statements: [], terminator: Some(Terminator { source_info: SourceInfo { span: src/aws.rs:39:33: 39:34 (#277), scope: scope[2] }, kind: drop(_10) -> bb36 }), is_cleanup: true }, BasicBlockData { statements: [], terminator: Some(Terminator { source_info: SourceInfo { span: src/aws.rs:39:32: 39:33 (#0), scope: scope[2] }, kind: drop(_12) -> bb36 }), is_cleanup: true }, BasicBlockData { statements: [], terminator: Some(Terminator { source_info: SourceInfo { span: src/aws.rs:40:1: 40:2 (#0), scope: scope[1] }, kind: drop(_4) -> bb39 }), is_cleanup: true }, BasicBlockData { statements: [], terminator: Some(Terminator { source_info: SourceInfo { span: src/aws.rs:38:6: 38:7 (#0), scope: scope[1] }, kind: drop(_6) -> bb39 }), is_cleanup: true }, BasicBlockData { statements: [], terminator: Some(Terminator { source_info: SourceInfo { span: src/aws.rs:38:5: 38:6 (#0), scope: scope[1] }, kind: drop(_7) -> bb39 }), is_cleanup: true }, BasicBlockData { statements: [], terminator: Some(Terminator { source_info: SourceInfo { span: src/aws.rs:40:1: 40:2 (#0), scope: scope[0] }, kind: drop(_2) -> bb41 }), is_cleanup: true }, BasicBlockData { statements: [], terminator: Some(Terminator { source_info: SourceInfo { span: src/aws.rs:12:57: 12:58 (#0), scope: scope[0] }, kind: drop(_3) -> bb41 }), is_cleanup: true }, BasicBlockData { statements: [], terminator: Some(Terminator { source_info: SourceInfo { span: src/aws.rs:40:1: 40:2 (#0), scope: scope[0] }, kind: drop(_1) -> bb42 }), is_cleanup: true }, BasicBlockData { statements: [], terminator: Some(Terminator { source_info: SourceInfo { span: src/aws.rs:10:1: 40:2 (#0), scope: scope[0] }, kind: resume }), is_cleanup: true }], predecessor_cache: PredecessorCache { cache: OnceCell(Uninit) }, switch_source_cache: SwitchSourceCache { cache: OnceCell(Uninit) }, is_cyclic: GraphIsCyclicCache { cache: OnceCell(Uninit) }, postorder_cache: PostorderCache { cache: OnceCell(Uninit) } }, phase: Built, pass_count: 1, source: MirSource { instance: Item(WithOptConstParam { did: DefId(0:10 ~ pypi_data_scraper[38b5]::aws::check_aws_keys), const_param_did: None }), promoted: None }, source_scopes: [SourceScopeData { span: src/aws.rs:10:1: 40:2 (#0), parent_scope: None, inlined: None, inlined_parent_scope: None, local_data: Set(SourceScopeLocalData { lint_root: HirId { owner: OwnerId { def_id: DefId(0:10 ~ pypi_data_scraper[38b5]::aws::check_aws_keys) }, local_id: 0 }, safety: Safe }) }, SourceScopeData { span: src/aws.rs:12:5: 40:2 (#0), parent_scope: Some(scope[0]), inlined: None, inlined_parent_scope: None, local_data: Set(SourceScopeLocalData { lint_root: HirId { owner: OwnerId { def_id: DefId(0:10 ~ pypi_data_scraper[38b5]::aws::check_aws_keys) }, local_id: 0 }, safety: Safe }) }, SourceScopeData { span: src/aws.rs:13:5: 40:2 (#0), parent_scope: Some(scope[1]), inlined: None, inlined_parent_scope: None, local_data: Set(SourceScopeLocalData { lint_root: HirId { owner: OwnerId { def_id: DefId(0:10 ~ pypi_data_scraper[38b5]::aws::check_aws_keys) }, local_id: 0 }, safety: Safe }) }, SourceScopeData { span: src/aws.rs:39:33: 39:34 (#278), parent_scope: Some(scope[2]), inlined: None, inlined_parent_scope: None, local_data: Set(SourceScopeLocalData { lint_root: HirId { owner: OwnerId { def_id: DefId(0:10 ~ pypi_data_scraper[38b5]::aws::check_aws_keys) }, local_id: 0 }, safety: Safe }) }, SourceScopeData { span: src/aws.rs:39:33: 39:34 (#278), parent_scope: Some(scope[3]), inlined: None, inlined_parent_scope: None, local_data: Set(SourceScopeLocalData { lint_root: HirId { owner: OwnerId { def_id: DefId(0:10 ~ pypi_data_scraper[38b5]::aws::check_aws_keys) }, local_id: 330 }, safety: Safe }) }, SourceScopeData { span: src/aws.rs:39:8: 39:34 (#0), parent_scope: Some(scope[2]), inlined: None, inlined_parent_scope: None, local_data: Set(SourceScopeLocalData { lint_root: HirId { owner: OwnerId { def_id: DefId(0:10 ~ pypi_data_scraper[38b5]::aws::check_aws_keys) }, local_id: 0 }, safety: Safe }) }, SourceScopeData { span: src/aws.rs:39:8: 39:34 (#0), parent_scope: Some(scope[5]), inlined: None, inlined_parent_scope: None, local_data: Set(SourceScopeLocalData { lint_root: HirId { owner: OwnerId { def_id: DefId(0:10 ~ pypi_data_scraper[38b5]::aws::check_aws_keys) }, local_id: 321 }, safety: Safe }) }], generator: None, local_decls: [LocalDecl { mutability: Mut, local_info: None, internal: false, is_block_tail: None, ty: std::result::Result, anyhow::Error>, user_ty: None, source_info: SourceInfo { span: src/aws.rs:10:54: 10:74 (#0), scope: scope[0] } }, LocalDecl { mutability: Not, local_info: Some(User(Set(Var(VarBindingForm { binding_mode: BindByValue(Not), opt_ty_info: Some(src/aws.rs:10:32: 10:49 (#0)), opt_match_place: Some((None, src/aws.rs:10:23: 10:30 (#0))), pat_span: src/aws.rs:10:23: 10:30 (#0) })))), internal: false, is_block_tail: None, ty: std::vec::Vec, user_ty: None, source_info: SourceInfo { span: src/aws.rs:10:23: 10:30 (#0), scope: scope[0] } }, LocalDecl { mutability: Not, local_info: Some(User(Set(Var(VarBindingForm { binding_mode: BindByValue(Not), opt_ty_info: None, opt_match_place: Some((None, src/aws.rs:12:19: 12:58 (#0))), pat_span: src/aws.rs:12:9: 12:16 (#0) })))), internal: false, is_block_tail: None, ty: tokio::runtime::Runtime, user_ty: None, source_info: SourceInfo { span: src/aws.rs:12:9: 12:16 (#0), scope: scope[0] } }, LocalDecl { mutability: Mut, local_info: None, internal: false, is_block_tail: None, ty: std::result::Result, user_ty: None, source_info: SourceInfo { span: src/aws.rs:12:19: 12:49 (#0), scope: scope[0] } }, LocalDecl { mutability: Not, local_info: Some(User(Set(Var(VarBindingForm { binding_mode: BindByValue(Not), opt_ty_info: None, opt_match_place: Some((None, src/aws.rs:13:19: 38:7 (#0))), pat_span: src/aws.rs:13:9: 13:16 (#0) })))), internal: false, is_block_tail: None, ty: tokio::task::JoinHandle>, user_ty: None, source_info: SourceInfo { span: src/aws.rs:13:9: 13:16 (#0), scope: scope[1] } }, LocalDecl { mutability: Mut, local_info: None, internal: false, is_block_tail: None, ty: &tokio::runtime::Runtime, user_ty: None, source_info: SourceInfo { span: src/aws.rs:13:19: 38:7 (#0), scope: scope[0] } }, LocalDecl { mutability: Mut, local_info: None, internal: false, is_block_tail: None, ty: impl std::future::Future>, user_ty: None, source_info: SourceInfo { span: src/aws.rs:13:33: 38:6 (#0), scope: scope[0] } }, LocalDecl { mutability: Mut, local_info: None, internal: false, is_block_tail: None, ty: [static generator@src/aws.rs:13:39: 38:6], user_ty: None, source_info: SourceInfo { span: src/aws.rs:13:39: 38:6 (#0), scope: scope[0] } }, LocalDecl { mutability: Mut, local_info: Some(AggregateTemp), internal: false, is_block_tail: Some(BlockTailInfo { tail_result_is_ignored: false, span: src/aws.rs:39:5: 39:35 (#0) }), ty: std::vec::Vec, user_ty: None, source_info: SourceInfo { span: src/aws.rs:39:8: 39:34 (#0), scope: scope[0] } }, LocalDecl { mutability: Mut, local_info: None, internal: false, is_block_tail: Some(BlockTailInfo { tail_result_is_ignored: false, span: src/aws.rs:39:5: 39:35 (#0) }), ty: std::ops::ControlFlow, std::vec::Vec>, user_ty: None, source_info: SourceInfo { span: src/aws.rs:39:8: 39:34 (#277), scope: scope[0] } }, LocalDecl { mutability: Mut, local_info: None, internal: false, is_block_tail: Some(BlockTailInfo { tail_result_is_ignored: false, span: src/aws.rs:39:5: 39:35 (#0) }), ty: std::result::Result, tokio::task::JoinError>, user_ty: None, source_info: SourceInfo { span: src/aws.rs:39:8: 39:33 (#0), scope: scope[0] } }, LocalDecl { mutability: Mut, local_info: None, internal: false, is_block_tail: Some(BlockTailInfo { tail_result_is_ignored: false, span: src/aws.rs:39:5: 39:35 (#0) }), ty: &tokio::runtime::Runtime, user_ty: None, source_info: SourceInfo { span: src/aws.rs:39:8: 39:33 (#0), scope: scope[0] } }, LocalDecl { mutability: Mut, local_info: None, internal: false, is_block_tail: Some(BlockTailInfo { tail_result_is_ignored: false, span: src/aws.rs:39:5: 39:35 (#0) }), ty: tokio::task::JoinHandle>, user_ty: None, source_info: SourceInfo { span: src/aws.rs:39:25: 39:32 (#0), scope: scope[0] } }, LocalDecl { mutability: Mut, local_info: None, internal: true, is_block_tail: None, ty: isize, user_ty: None, source_info: SourceInfo { span: src/aws.rs:39:33: 39:34 (#278), scope: scope[0] } }, LocalDecl { mutability: Not, local_info: Some(User(Set(Var(VarBindingForm { binding_mode: BindByValue(Not), opt_ty_info: None, opt_match_place: Some((Some(_9), src/aws.rs:39:8: 39:34 (#277))), pat_span: src/aws.rs:39:33: 39:34 (#278) })))), internal: false, is_block_tail: None, ty: std::result::Result, user_ty: None, source_info: SourceInfo { span: src/aws.rs:39:33: 39:34 (#278), scope: scope[2] } }, LocalDecl { mutability: Mut, local_info: None, internal: false, is_block_tail: Some(BlockTailInfo { tail_result_is_ignored: false, span: src/aws.rs:39:5: 39:35 (#0) }), ty: !, user_ty: None, source_info: SourceInfo { span: src/aws.rs:39:33: 39:34 (#278), scope: scope[0] } }, LocalDecl { mutability: Mut, local_info: None, internal: false, is_block_tail: Some(BlockTailInfo { tail_result_is_ignored: false, span: src/aws.rs:39:5: 39:35 (#0) }), ty: std::result::Result, user_ty: None, source_info: SourceInfo { span: src/aws.rs:39:33: 39:34 (#278), scope: scope[0] } }, LocalDecl { mutability: Not, local_info: Some(User(Set(Var(VarBindingForm { binding_mode: BindByValue(Not), opt_ty_info: None, opt_match_place: Some((Some(_9), src/aws.rs:39:8: 39:34 (#277))), pat_span: src/aws.rs:39:8: 39:34 (#277) })))), internal: false, is_block_tail: None, ty: std::vec::Vec, user_ty: None, source_info: SourceInfo { span: src/aws.rs:39:8: 39:34 (#0), scope: scope[2] } }], user_type_annotations: [], arg_count: 1, spread_arg: None, var_debug_info: [VarDebugInfo { name: "matches", source_info: SourceInfo { span: src/aws.rs:10:23: 10:30 (#0), scope: scope[0] }, value: _1 }, VarDebugInfo { name: "runtime", source_info: SourceInfo { span: src/aws.rs:12:9: 12:16 (#0), scope: scope[1] }, value: _2 }, VarDebugInfo { name: "checker", source_info: SourceInfo { span: src/aws.rs:13:9: 13:16 (#0), scope: scope[2] }, value: _4 }, VarDebugInfo { name: "residual", source_info: SourceInfo { span: src/aws.rs:39:33: 39:34 (#278), scope: scope[3] }, value: _14 }, VarDebugInfo { name: "val", source_info: SourceInfo { span: src/aws.rs:39:8: 39:34 (#0), scope: scope[5] }, value: _17 }], span: src/aws.rs:10:1: 40:2 (#0), required_consts: [], is_polymorphic: false, tainted_by_errors: None }) }) }', compiler/rustc_query_system/src/query/plumbing.rs:658:9 ```

[Thibaut-Le-Goff]

I had an ICE with the codes in this commit :
Thibaut-Le-Goff/Runst_-unfinished-@15a51c1

with rustc --version --verbose I get:

rustc 1.66.0 (69f9c33d7 2022-12-12)
binary: rustc
commit-hash: 69f9c33d71c871fc16ac445211281c6e7a340943
commit-date: 2022-12-12
host: x86_64-unknown-linux-gnu
release: 1.66.0
LLVM version: 15.0.2

The issue occurred when I was working on the function weight_init::which_dis in the file src/runst/weight_init.rs.

Here is the txt file with the outputs.
ICE_error_Thibaut-Le-Goff_Runst_commit_15a51c1.txt

[chenyukang]

I hit this issue when working on #108938, and then git checkout master 8a73f50d875840b8077b8ec080fa41881d7ce40d, the error message is:

error: internal compiler error: encountered incremental compilation error with registered_tools(0-0)
  |
  = help: This is a known issue with the compiler. Run `cargo clean -p rust_demangler` or `cargo clean` to allow your project to compile
  = note: Please follow the instructions below to create a bug report with the provided information
  = note: See <https://github.com/rust-lang/rust/issues/84970> for more information

thread 'rustc' panicked at 'Found unstable fingerprints for registered_tools(0-0): {clippy#0, rustfmt#0}', compiler/rustc_query_system/src/query/plumbing.rs:688:9
stack backtrace:
   0: rust_begin_unwind
             at ./library/std/src/panicking.rs:579:5
   1: core::panicking::panic_fmt
             at ./library/core/src/panicking.rs:64:14
   2: rustc_query_system::query::plumbing::incremental_verify_ich_failed
             at ./compiler/rustc_query_system/src/query/plumbing.rs:688:9
   3: rustc_query_system::query::plumbing::incremental_verify_ich::<rustc_middle::ty::context::TyCtxt, &indexmap::set::IndexSet<rustc_span::symbol::Ident, core::hash::BuildHasherDefault<rustc_hash::FxHasher>>>
             at ./compiler/rustc_query_system/src/query/plumbing.rs:610:9
   4: rustc_query_system::query::plumbing::try_load_from_disk_and_cache_in_memory::<rustc_query_impl::queries::registered_tools, rustc_query_impl::plumbing::QueryCtxt>
             at ./compiler/rustc_query_system/src/query/plumbing.rs:582:5
   5: rustc_query_system::query::plumbing::execute_job::<rustc_query_impl::queries::registered_tools, rustc_query_impl::plumbing::QueryCtxt>::{closure#3}
             at ./compiler/rustc_query_system/src/query/plumbing.rs:449:13

Follow the hint can not fix it:
cargo clean -p rust_demangler or cargo clean to allow your project to compile

By the way, cargo clean -p rust_demangler should be cargo clean -p rust-demangler.

[carols10cents]

Just hit this again for evaluate_obligation with 1.69 stable, while working on InfluxDB IOx, after a long time of not seeing this problem.

rustc --version --verbose:

rustc 1.69.0 (84c898d65 2023-04-16)
binary: rustc
commit-hash: 84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc
commit-date: 2023-04-16
host: aarch64-apple-darwin
release: 1.69.0
LLVM version: 15.0.7

Backtrace

error[E0599]: the method `and_then` exists for struct `Chain<Chain<RetentionValidator<Arc<ReadThroughCache<Arc<ShardedCache<Arc<MemoryNamespaceCache>>>>>>, SchemaValidator<Arc<ReadThroughCache<...>>>>, ...>`, but its trait bounds were not satisfied
   --> router/tests/common/mod.rs:179:14
    |
176 |           let handler_stack = retention_validator
    |  _____________________________-
177 | |             .and_then(schema_validator)
178 | |             .and_then(partitioner)
179 | |             .and_then(parallel_write);
    | |             -^^^^^^^^ method cannot be called due to unsatisfied trait bounds
    | |_____________|
    |
    |
   ::: /Users/carolnichols/rust/influxdb_iox/router/src/dml_handlers/chain.rs:31:1
    |
31  |   pub struct Chain<T, U> {
    |   ----------------------
    |   |
    |   doesn't satisfy `_: DmlHandlerChainExt`
    |   doesn't satisfy `_: DmlHandler`
    |
    = note: the full type name has been written to '/Users/carolnichols/rust/influxdb_iox/target/debug/deps/http-3f8d77dc51fec4bb.long-type-10401456881853157544.txt'
    = note: the following trait bounds were not satisfied:
            `router::dml_handlers::Chain<router::dml_handlers::Chain<RetentionValidator<Arc<ReadThroughCache<Arc<ShardedCache<Arc<MemoryNamespaceCache>>>>>>, SchemaValidator<Arc<ReadThroughCache<Arc<ShardedCache<Arc<MemoryNamespaceCache>>>>>>>, Partitioner>: router::dml_handlers::DmlHandler`
            which is required by `router::dml_handlers::Chain<router::dml_handlers::Chain<RetentionValidator<Arc<ReadThroughCache<Arc<ShardedCache<Arc<MemoryNamespaceCache>>>>>>, SchemaValidator<Arc<ReadThroughCache<Arc<ShardedCache<Arc<MemoryNamespaceCache>>>>>>>, Partitioner>: DmlHandlerChainExt`
            `&router::dml_handlers::Chain<router::dml_handlers::Chain<RetentionValidator<Arc<ReadThroughCache<Arc<ShardedCache<Arc<MemoryNamespaceCache>>>>>>, SchemaValidator<Arc<ReadThroughCache<Arc<ShardedCache<Arc<MemoryNamespaceCache>>>>>>>, Partitioner>: router::dml_handlers::DmlHandler`
            which is required by `&router::dml_handlers::Chain<router::dml_handlers::Chain<RetentionValidator<Arc<ReadThroughCache<Arc<ShardedCache<Arc<MemoryNamespaceCache>>>>>>, SchemaValidator<Arc<ReadThroughCache<Arc<ShardedCache<Arc<MemoryNamespaceCache>>>>>>>, Partitioner>: DmlHandlerChainExt`
            `&mut router::dml_handlers::Chain<router::dml_handlers::Chain<RetentionValidator<Arc<ReadThroughCache<Arc<ShardedCache<Arc<MemoryNamespaceCache>>>>>>, SchemaValidator<Arc<ReadThroughCache<Arc<ShardedCache<Arc<MemoryNamespaceCache>>>>>>>, Partitioner>: router::dml_handlers::DmlHandler`
            which is required by `&mut router::dml_handlers::Chain<router::dml_handlers::Chain<RetentionValidator<Arc<ReadThroughCache<Arc<ShardedCache<Arc<MemoryNamespaceCache>>>>>>, SchemaValidator<Arc<ReadThroughCache<Arc<ShardedCache<Arc<MemoryNamespaceCache>>>>>>>, Partitioner>: DmlHandlerChainExt`

error[E0599]: the method `and_then` exists for struct `Chain<Chain<RetentionValidator<Arc<ReadThroughCache<Arc<ShardedCache<Arc<MemoryNamespaceCache>>>>>>, SchemaValidator<Arc<ReadThroughCache<...>>>>, ...>`, but its trait bounds were not satisfied
   --> router/tests/common/mod.rs:179:14
    |
176 |           let handler_stack = retention_validator
    |  _____________________________-
177 | |             .and_then(schema_validator)
178 | |             .and_then(partitioner)
179 | |             .and_then(parallel_write);
    | |             -^^^^^^^^ method cannot be called due to unsatisfied trait bounds
    | |_____________|
    |
    |
   ::: /Users/carolnichols/rust/influxdb_iox/router/src/dml_handlers/chain.rs:31:1
    |
31  |   pub struct Chain<T, U> {
    |   ----------------------
    |   |
    |   doesn't satisfy `_: DmlHandlerChainExt`
    |   doesn't satisfy `_: DmlHandler`
    |
    = note: the full type name has been written to '/Users/carolnichols/rust/influxdb_iox/target/debug/deps/grpc-3375cce1e68ab950.long-type-10547554907313205075.txt'
    = note: the following trait bounds were not satisfied:
            `router::dml_handlers::Chain<router::dml_handlers::Chain<RetentionValidator<Arc<ReadThroughCache<Arc<ShardedCache<Arc<MemoryNamespaceCache>>>>>>, SchemaValidator<Arc<ReadThroughCache<Arc<ShardedCache<Arc<MemoryNamespaceCache>>>>>>>, Partitioner>: router::dml_handlers::DmlHandler`
            which is required by `router::dml_handlers::Chain<router::dml_handlers::Chain<RetentionValidator<Arc<ReadThroughCache<Arc<ShardedCache<Arc<MemoryNamespaceCache>>>>>>, SchemaValidator<Arc<ReadThroughCache<Arc<ShardedCache<Arc<MemoryNamespaceCache>>>>>>>, Partitioner>: DmlHandlerChainExt`
            `&router::dml_handlers::Chain<router::dml_handlers::Chain<RetentionValidator<Arc<ReadThroughCache<Arc<ShardedCache<Arc<MemoryNamespaceCache>>>>>>, SchemaValidator<Arc<ReadThroughCache<Arc<ShardedCache<Arc<MemoryNamespaceCache>>>>>>>, Partitioner>: router::dml_handlers::DmlHandler`
            which is required by `&router::dml_handlers::Chain<router::dml_handlers::Chain<RetentionValidator<Arc<ReadThroughCache<Arc<ShardedCache<Arc<MemoryNamespaceCache>>>>>>, SchemaValidator<Arc<ReadThroughCache<Arc<ShardedCache<Arc<MemoryNamespaceCache>>>>>>>, Partitioner>: DmlHandlerChainExt`
            `&mut router::dml_handlers::Chain<router::dml_handlers::Chain<RetentionValidator<Arc<ReadThroughCache<Arc<ShardedCache<Arc<MemoryNamespaceCache>>>>>>, SchemaValidator<Arc<ReadThroughCache<Arc<ShardedCache<Arc<MemoryNamespaceCache>>>>>>>, Partitioner>: router::dml_handlers::DmlHandler`
            which is required by `&mut router::dml_handlers::Chain<router::dml_handlers::Chain<RetentionValidator<Arc<ReadThroughCache<Arc<ShardedCache<Arc<MemoryNamespaceCache>>>>>>, SchemaValidator<Arc<ReadThroughCache<Arc<ShardedCache<Arc<MemoryNamespaceCache>>>>>>>, Partitioner>: DmlHandlerChainExt`

error[E0599]: the method `and_then` exists for struct `Chain<Chain<InstrumentationDecorator<RetentionValidator<Arc<ReadThroughCache<Arc<InstrumentedCache<Arc<ShardedCache<Arc<...>>>>>>>>>, ...>, ...>`, but its trait bounds were not satisfied
   --> ioxd_router/src/lib.rs:343:10
    |
334 |       let handler_stack = retention_validator
    |  _________________________-
335 | |         .and_then(schema_validator)
336 | |         .and_then(partitioner)
337 | |         // Once writes have been partitioned, they are processed in parallel.
...   |
342 | |         // operation.
343 | |         .and_then(InstrumentationDecorator::new(
    | |         -^^^^^^^^ method cannot be called due to unsatisfied trait bounds
    | |_________|
    |
    |
   ::: /Users/carolnichols/rust/influxdb_iox/router/src/dml_handlers/chain.rs:31:1
    |
31  |   pub struct Chain<T, U> {
    |   ----------------------
    |   |
    |   doesn't satisfy `_: DmlHandlerChainExt`
    |   doesn't satisfy `_: DmlHandler`
    |
    = note: the full type name has been written to '/Users/carolnichols/rust/influxdb_iox/target/debug/deps/ioxd_router-28fbc6aba03968a6.long-type-9571234836598881310.txt'
    = note: the following trait bounds were not satisfied:
            `router::dml_handlers::Chain<router::dml_handlers::Chain<InstrumentationDecorator<RetentionValidator<Arc<ReadThroughCache<Arc<InstrumentedCache<Arc<ShardedCache<Arc<MemoryNamespaceCache>>>>>>>>>, InstrumentationDecorator<SchemaValidator<Arc<ReadThroughCache<Arc<InstrumentedCache<Arc<ShardedCache<Arc<MemoryNamespaceCache>>>>>>>>>>, InstrumentationDecorator<Partitioner>>: router::dml_handlers::DmlHandler`
            which is required by `router::dml_handlers::Chain<router::dml_handlers::Chain<InstrumentationDecorator<RetentionValidator<Arc<ReadThroughCache<Arc<InstrumentedCache<Arc<ShardedCache<Arc<MemoryNamespaceCache>>>>>>>>>, InstrumentationDecorator<SchemaValidator<Arc<ReadThroughCache<Arc<InstrumentedCache<Arc<ShardedCache<Arc<MemoryNamespaceCache>>>>>>>>>>, InstrumentationDecorator<Partitioner>>: DmlHandlerChainExt`
            `&router::dml_handlers::Chain<router::dml_handlers::Chain<InstrumentationDecorator<RetentionValidator<Arc<ReadThroughCache<Arc<InstrumentedCache<Arc<ShardedCache<Arc<MemoryNamespaceCache>>>>>>>>>, InstrumentationDecorator<SchemaValidator<Arc<ReadThroughCache<Arc<InstrumentedCache<Arc<ShardedCache<Arc<MemoryNamespaceCache>>>>>>>>>>, InstrumentationDecorator<Partitioner>>: router::dml_handlers::DmlHandler`
            which is required by `&router::dml_handlers::Chain<router::dml_handlers::Chain<InstrumentationDecorator<RetentionValidator<Arc<ReadThroughCache<Arc<InstrumentedCache<Arc<ShardedCache<Arc<MemoryNamespaceCache>>>>>>>>>, InstrumentationDecorator<SchemaValidator<Arc<ReadThroughCache<Arc<InstrumentedCache<Arc<ShardedCache<Arc<MemoryNamespaceCache>>>>>>>>>>, InstrumentationDecorator<Partitioner>>: DmlHandlerChainExt`
            `&mut router::dml_handlers::Chain<router::dml_handlers::Chain<InstrumentationDecorator<RetentionValidator<Arc<ReadThroughCache<Arc<InstrumentedCache<Arc<ShardedCache<Arc<MemoryNamespaceCache>>>>>>>>>, InstrumentationDecorator<SchemaValidator<Arc<ReadThroughCache<Arc<InstrumentedCache<Arc<ShardedCache<Arc<MemoryNamespaceCache>>>>>>>>>>, InstrumentationDecorator<Partitioner>>: router::dml_handlers::DmlHandler`
            which is required by `&mut router::dml_handlers::Chain<router::dml_handlers::Chain<InstrumentationDecorator<RetentionValidator<Arc<ReadThroughCache<Arc<InstrumentedCache<Arc<ShardedCache<Arc<MemoryNamespaceCache>>>>>>>>>, InstrumentationDecorator<SchemaValidator<Arc<ReadThroughCache<Arc<InstrumentedCache<Arc<ShardedCache<Arc<MemoryNamespaceCache>>>>>>>>>>, InstrumentationDecorator<Partitioner>>: DmlHandlerChainExt`

error: internal compiler error: encountered incremental compilation error with evaluate_obligation(c87e8d7be8b35fff-23a633ac04958b7e)
  |
  = help: This is a known issue with the compiler. Run `cargo clean -p http` or `cargo clean` to allow your project to compile
  = note: Please follow the instructions below to create a bug report with the provided information
  = note: See <https://github.com/rust-lang/rust/issues/84970> for more information

thread 'rustc' panicked at 'Found unstable fingerprints for evaluate_obligation(c87e8d7be8b35fff-23a633ac04958b7e): Ok(EvaluatedToErr)', compiler/rustc_query_system/src/query/plumbing.rs:677:9
stack backtrace:
error: internal compiler error: encountered incremental compilation error with evaluate_obligation(c87e8d7be8b35fff-23a633ac04958b7e)
  |
  = help: This is a known issue with the compiler. Run `cargo clean -p grpc` or `cargo clean` to allow your project to compile
  = note: Please follow the instructions below to create a bug report with the provided information
  = note: See <https://github.com/rust-lang/rust/issues/84970> for more information

thread 'rustc' panicked at 'Found unstable fingerprints for evaluate_obligation(c87e8d7be8b35fff-23a633ac04958b7e): Ok(EvaluatedToErr)', compiler/rustc_query_system/src/query/plumbing.rs:677:9
stack backtrace:
error[E0599]: the method `and_then` exists for struct `Chain<Chain<InstrumentationDecorator<RetentionValidator<Arc<ReadThroughCache<Arc<InstrumentedCache<Arc<ShardedCache<Arc<...>>>>>>>>>, ...>, ...>`, but its trait bounds were not satisfied
   --> ioxd_router/src/lib.rs:343:10
    |
334 |       let handler_stack = retention_validator
    |  _________________________-
335 | |         .and_then(schema_validator)
336 | |         .and_then(partitioner)
337 | |         // Once writes have been partitioned, they are processed in parallel.
...   |
342 | |         // operation.
343 | |         .and_then(InstrumentationDecorator::new(
    | |         -^^^^^^^^ method cannot be called due to unsatisfied trait bounds
    | |_________|
    |
    |
   ::: /Users/carolnichols/rust/influxdb_iox/router/src/dml_handlers/chain.rs:31:1
    |
31  |   pub struct Chain<T, U> {
    |   ----------------------
    |   |
    |   doesn't satisfy `_: DmlHandlerChainExt`
    |   doesn't satisfy `_: DmlHandler`
    |
    = note: the full type name has been written to '/Users/carolnichols/rust/influxdb_iox/target/debug/deps/ioxd_router-5d1cf8648587517f.long-type-4254684705345345092.txt'
    = note: the following trait bounds were not satisfied:
            `router::dml_handlers::Chain<router::dml_handlers::Chain<InstrumentationDecorator<RetentionValidator<std::sync::Arc<ReadThroughCache<std::sync::Arc<InstrumentedCache<std::sync::Arc<ShardedCache<std::sync::Arc<router::namespace_cache::MemoryNamespaceCache>>>>>>>>>, InstrumentationDecorator<SchemaValidator<std::sync::Arc<ReadThroughCache<std::sync::Arc<InstrumentedCache<std::sync::Arc<ShardedCache<std::sync::Arc<router::namespace_cache::MemoryNamespaceCache>>>>>>>>>>, InstrumentationDecorator<Partitioner>>: router::dml_handlers::DmlHandler`
            which is required by `router::dml_handlers::Chain<router::dml_handlers::Chain<InstrumentationDecorator<RetentionValidator<std::sync::Arc<ReadThroughCache<std::sync::Arc<InstrumentedCache<std::sync::Arc<ShardedCache<std::sync::Arc<router::namespace_cache::MemoryNamespaceCache>>>>>>>>>, InstrumentationDecorator<SchemaValidator<std::sync::Arc<ReadThroughCache<std::sync::Arc<InstrumentedCache<std::sync::Arc<ShardedCache<std::sync::Arc<router::namespace_cache::MemoryNamespaceCache>>>>>>>>>>, InstrumentationDecorator<Partitioner>>: DmlHandlerChainExt`
            `&router::dml_handlers::Chain<router::dml_handlers::Chain<InstrumentationDecorator<RetentionValidator<std::sync::Arc<ReadThroughCache<std::sync::Arc<InstrumentedCache<std::sync::Arc<ShardedCache<std::sync::Arc<router::namespace_cache::MemoryNamespaceCache>>>>>>>>>, InstrumentationDecorator<SchemaValidator<std::sync::Arc<ReadThroughCache<std::sync::Arc<InstrumentedCache<std::sync::Arc<ShardedCache<std::sync::Arc<router::namespace_cache::MemoryNamespaceCache>>>>>>>>>>, InstrumentationDecorator<Partitioner>>: router::dml_handlers::DmlHandler`
            which is required by `&router::dml_handlers::Chain<router::dml_handlers::Chain<InstrumentationDecorator<RetentionValidator<std::sync::Arc<ReadThroughCache<std::sync::Arc<InstrumentedCache<std::sync::Arc<ShardedCache<std::sync::Arc<router::namespace_cache::MemoryNamespaceCache>>>>>>>>>, InstrumentationDecorator<SchemaValidator<std::sync::Arc<ReadThroughCache<std::sync::Arc<InstrumentedCache<std::sync::Arc<ShardedCache<std::sync::Arc<router::namespace_cache::MemoryNamespaceCache>>>>>>>>>>, InstrumentationDecorator<Partitioner>>: DmlHandlerChainExt`
            `&mut router::dml_handlers::Chain<router::dml_handlers::Chain<InstrumentationDecorator<RetentionValidator<std::sync::Arc<ReadThroughCache<std::sync::Arc<InstrumentedCache<std::sync::Arc<ShardedCache<std::sync::Arc<router::namespace_cache::MemoryNamespaceCache>>>>>>>>>, InstrumentationDecorator<SchemaValidator<std::sync::Arc<ReadThroughCache<std::sync::Arc<InstrumentedCache<std::sync::Arc<ShardedCache<std::sync::Arc<router::namespace_cache::MemoryNamespaceCache>>>>>>>>>>, InstrumentationDecorator<Partitioner>>: router::dml_handlers::DmlHandler`
            which is required by `&mut router::dml_handlers::Chain<router::dml_handlers::Chain<InstrumentationDecorator<RetentionValidator<std::sync::Arc<ReadThroughCache<std::sync::Arc<InstrumentedCache<std::sync::Arc<ShardedCache<std::sync::Arc<router::namespace_cache::MemoryNamespaceCache>>>>>>>>>, InstrumentationDecorator<SchemaValidator<std::sync::Arc<ReadThroughCache<std::sync::Arc<InstrumentedCache<std::sync::Arc<ShardedCache<std::sync::Arc<router::namespace_cache::MemoryNamespaceCache>>>>>>>>>>, InstrumentationDecorator<Partitioner>>: DmlHandlerChainExt`

   0:        0x102f40468 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h105074e3d85f800b
   0:        0x102cec468 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h105074e3d85f800b
   1:        0x102f93388 - core::fmt::write::h34766cf8fff7af1e
   1:        0x102d3f388 - core::fmt::write::h34766cf8fff7af1e
   2:        0x102f361e0 - std::io::Write::write_fmt::hd64c4cf6e7adea59
   2:        0x102ce21e0 - std::io::Write::write_fmt::hd64c4cf6e7adea59
   3:        0x102f4027c - std::sys_common::backtrace::print::h2a6828a537036cf9
   3:        0x102cec27c - std::sys_common::backtrace::print::h2a6828a537036cf9
   4:        0x102ceed04 - std::panicking::default_hook::{{closure}}::h4e82ce6ccef941b2
   4:        0x102f42d04 - std::panicking::default_hook::{{closure}}::h4e82ce6ccef941b2
   5:        0x102ceea5c - std::panicking::default_hook::h29f62f8795c5cb00
   5:        0x102f42a5c - std::panicking::default_hook::h29f62f8795c5cb00
For more information about this error, try `rustc --explain E0599`.
   6:        0x10adc174c - rustc_driver_impl[4da67b3d18d5a97a]::DEFAULT_HOOK::{closure#0}::{closure#0}
   6:        0x10b01574c - rustc_driver_impl[4da67b3d18d5a97a]::DEFAULT_HOOK::{closure#0}::{closure#0}
   7:        0x102cef3fc - std::panicking::rust_panic_with_hook::h19862cbd0fbda7ba
   7:        0x102f433fc - std::panicking::rust_panic_with_hook::h19862cbd0fbda7ba
   8:        0x102f431f4 - std::panicking::begin_panic_handler::{{closure}}::h3f3626935e1669fe
   8:        0x102cef1f4 - std::panicking::begin_panic_handler::{{closure}}::h3f3626935e1669fe
   9:        0x102cec888 - std::sys_common::backtrace::__rust_end_short_backtrace::h5054ef52bd507d0a
   9:        0x102f40888 - std::sys_common::backtrace::__rust_end_short_backtrace::h5054ef52bd507d0a
  10:        0x102ceef50 - _rust_begin_unwind
  10:        0x102f42f50 - _rust_begin_unwind
  11:        0x102fbe608 - core::panicking::panic_fmt::h7e47e10600a90221
  12:        0x10f29de24 - rustc_query_system[4ec8b7e6ed7d2b80]::query::plumbing::incremental_verify_ich_failed
  13:        0x10e02a20c - rustc_query_system[4ec8b7e6ed7d2b80]::query::plumbing::try_execute_query::<rustc_query_impl[4d60d89ac47f5f83]::queries::evaluate_obligation, rustc_query_impl[4d60d89ac47f5f83]::plumbing::QueryCtxt>
  11:        0x102d6a608 - core::panicking::panic_fmt::h7e47e10600a90221
  14:        0x10e17fa24 - <rustc_query_impl[4d60d89ac47f5f83]::Queries as rustc_middle[a12d4e31e9c7c7b2]::ty::query::QueryEngine>::evaluate_obligation
  12:        0x10f049e24 - rustc_query_system[4ec8b7e6ed7d2b80]::query::plumbing::incremental_verify_ich_failed
  13:        0x10ddd620c - rustc_query_system[4ec8b7e6ed7d2b80]::query::plumbing::try_execute_query::<rustc_query_impl[4d60d89ac47f5f83]::queries::evaluate_obligation, rustc_query_impl[4d60d89ac47f5f83]::plumbing::QueryCtxt>
  14:        0x10df2ba24 - <rustc_query_impl[4d60d89ac47f5f83]::Queries as rustc_middle[a12d4e31e9c7c7b2]::ty::query::QueryEngine>::evaluate_obligation
  15:        0x10eae8d60 - <rustc_infer[9cb72f641b6c399a]::infer::InferCtxt as rustc_trait_selection[9b8c5d34047d54d3]::traits::query::evaluate_obligation::InferCtxtExt>::evaluate_obligation
  15:        0x10e894d60 - <rustc_infer[9cb72f641b6c399a]::infer::InferCtxt as rustc_trait_selection[9b8c5d34047d54d3]::traits::query::evaluate_obligation::InferCtxtExt>::evaluate_obligation
  16:        0x10eae8ee0 - <rustc_infer[9cb72f641b6c399a]::infer::InferCtxt as rustc_trait_selection[9b8c5d34047d54d3]::traits::query::evaluate_obligation::InferCtxtExt>::evaluate_obligation_no_overflow
  16:        0x10e894ee0 - <rustc_infer[9cb72f641b6c399a]::infer::InferCtxt as rustc_trait_selection[9b8c5d34047d54d3]::traits::query::evaluate_obligation::InferCtxtExt>::evaluate_obligation_no_overflow
  17:        0x10eae8a44 - <rustc_infer[9cb72f641b6c399a]::infer::InferCtxt as rustc_trait_selection[9b8c5d34047d54d3]::traits::query::evaluate_obligation::InferCtxtExt>::predicate_may_hold
  17:        0x10e894a44 - <rustc_infer[9cb72f641b6c399a]::infer::InferCtxt as rustc_trait_selection[9b8c5d34047d54d3]::traits::query::evaluate_obligation::InferCtxtExt>::predicate_may_hold
  18:        0x10d31a524 - <rustc_infer[9cb72f641b6c399a]::infer::InferCtxt>::probe::<rustc_hir_typeck[4ee6bcd98d40a6cc]::method::probe::ProbeResult, <rustc_hir_typeck[4ee6bcd98d40a6cc]::method::probe::ProbeContext>::consider_probe::{closure#0}>
  18:        0x10d0c6524 - <rustc_infer[9cb72f641b6c399a]::infer::InferCtxt>::probe::<rustc_hir_typeck[4ee6bcd98d40a6cc]::method::probe::ProbeResult, <rustc_hir_typeck[4ee6bcd98d40a6cc]::method::probe::ProbeContext>::consider_probe::{closure#0}>
  19:        0x10d3799b4 - <alloc[f251364f33aae5a4]::vec::Vec<(&rustc_hir_typeck[4ee6bcd98d40a6cc]::method::probe::Candidate, rustc_hir_typeck[4ee6bcd98d40a6cc]::method::probe::ProbeResult)> as alloc[f251364f33aae5a4]::vec::spec_from_iter::SpecFromIter<(&rustc_hir_typeck[4ee6bcd98d40a6cc]::method::probe::Candidate, rustc_hir_typeck[4ee6bcd98d40a6cc]::method::probe::ProbeResult), core[446d629763b4aa]::iter::adapters::filter::Filter<core[446d629763b4aa]::iter::adapters::map::Map<core[446d629763b4aa]::slice::iter::Iter<rustc_hir_typeck[4ee6bcd98d40a6cc]::method::probe::Candidate>, <rustc_hir_typeck[4ee6bcd98d40a6cc]::method::probe::ProbeContext>::consider_candidates::{closure#0}>, <rustc_hir_typeck[4ee6bcd98d40a6cc]::method::probe::ProbeContext>::consider_candidates::{closure#1}>>>::from_iter
  20:        0x10d3e5ba0 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::method::probe::ProbeContext>::pick_method
  21:        0x10d3e5630 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::method::probe::ProbeContext>::pick_all_method
  22:        0x10d3e5230 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::method::probe::ProbeContext>::pick_core
  23:        0x10d3e4b50 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::method::probe::ProbeContext>::pick
  19:        0x10d1259b4 - <alloc[f251364f33aae5a4]::vec::Vec<(&rustc_hir_typeck[4ee6bcd98d40a6cc]::method::probe::Candidate, rustc_hir_typeck[4ee6bcd98d40a6cc]::method::probe::ProbeResult)> as alloc[f251364f33aae5a4]::vec::spec_from_iter::SpecFromIter<(&rustc_hir_typeck[4ee6bcd98d40a6cc]::method::probe::Candidate, rustc_hir_typeck[4ee6bcd98d40a6cc]::method::probe::ProbeResult), core[446d629763b4aa]::iter::adapters::filter::Filter<core[446d629763b4aa]::iter::adapters::map::Map<core[446d629763b4aa]::slice::iter::Iter<rustc_hir_typeck[4ee6bcd98d40a6cc]::method::probe::Candidate>, <rustc_hir_typeck[4ee6bcd98d40a6cc]::method::probe::ProbeContext>::consider_candidates::{closure#0}>, <rustc_hir_typeck[4ee6bcd98d40a6cc]::method::probe::ProbeContext>::consider_candidates::{closure#1}>>>::from_iter
  20:        0x10d191ba0 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::method::probe::ProbeContext>::pick_method
  21:        0x10d191630 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::method::probe::ProbeContext>::pick_all_method
  22:        0x10d191230 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::method::probe::ProbeContext>::pick_core
  24:        0x10d319634 - <rustc_infer[9cb72f641b6c399a]::infer::InferCtxt>::probe::<core[446d629763b4aa]::result::Result<rustc_hir_typeck[4ee6bcd98d40a6cc]::method::probe::Pick, rustc_hir_typeck[4ee6bcd98d40a6cc]::method::MethodError>, <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::probe_op<<rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::probe_for_name::{closure#0}, rustc_hir_typeck[4ee6bcd98d40a6cc]::method::probe::Pick>::{closure#4}>
  25:        0x10d2f1378 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::probe_for_name
  26:        0x10d2f2e5c - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::lookup_method
  27:        0x10d2e4300 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_expr_kind
  28:        0x10d29fd60 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_expr_with_expectation_and_args
  29:        0x10d2aec8c - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_argument_types
  23:        0x10d190b50 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::method::probe::ProbeContext>::pick
  30:        0x10d290b10 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::confirm_builtin_call
  24:        0x10d0c5634 - <rustc_infer[9cb72f641b6c399a]::infer::InferCtxt>::probe::<core[446d629763b4aa]::result::Result<rustc_hir_typeck[4ee6bcd98d40a6cc]::method::probe::Pick, rustc_hir_typeck[4ee6bcd98d40a6cc]::method::MethodError>, <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::probe_op<<rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::probe_for_name::{closure#0}, rustc_hir_typeck[4ee6bcd98d40a6cc]::method::probe::Pick>::{closure#4}>
  25:        0x10d09d378 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::probe_for_name
  26:        0x10d09ee5c - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::lookup_method
  31:        0x10d28f9a0 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_call
  27:        0x10d090300 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_expr_kind
  32:        0x10d2e4744 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_expr_kind
  33:        0x10d29fd60 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_expr_with_expectation_and_args
  28:        0x10d04bd60 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_expr_with_expectation_and_args
  29:        0x10d05ac8c - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_argument_types
  30:        0x10d03cb10 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::confirm_builtin_call
  34:        0x10d2dfeb0 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_match::{closure#0}
  31:        0x10d03b9a0 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_call
  32:        0x10d090744 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_expr_kind
  35:        0x10d2e4720 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_expr_kind
  33:        0x10d04bd60 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_expr_with_expectation_and_args
  36:        0x10d29fd60 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_expr_with_expectation_and_args
  37:        0x10d2b5304 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_block_with_expected
  38:        0x10d2e4940 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_expr_kind
  39:        0x10d29fd60 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_expr_with_expectation_and_args
  34:        0x10d08beb0 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_match::{closure#0}
  35:        0x10d090720 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_expr_kind
  40:        0x10d2e4284 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_expr_kind
  41:        0x10d29fd60 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_expr_with_expectation_and_args
  36:        0x10d04bd60 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_expr_with_expectation_and_args
  42:        0x10d2b5304 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_block_with_expected
  37:        0x10d061304 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_block_with_expected
  43:        0x10d2e4940 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_expr_kind
  38:        0x10d090940 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_expr_kind
  44:        0x10d29fd60 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_expr_with_expectation_and_args
  39:        0x10d04bd60 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_expr_with_expectation_and_args
  45:        0x10d2a0f90 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_return_expr
  46:        0x10d39e180 - rustc_hir_typeck[4ee6bcd98d40a6cc]::check::check_fn
  47:        0x10d2e1efc - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_expr_closure
  48:        0x10d2e4240 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_expr_kind
  40:        0x10d090284 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_expr_kind
  49:        0x10d29fd60 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_expr_with_expectation_and_args
  41:        0x10d04bd60 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_expr_with_expectation_and_args
  42:        0x10d061304 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_block_with_expected
  50:        0x10d2aec8c - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_argument_types
  43:        0x10d090940 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_expr_kind
  51:        0x10d290b10 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::confirm_builtin_call
  52:        0x10d28f9a0 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_call
  53:        0x10d2e4744 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_expr_kind
  54:        0x10d29fd60 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_expr_with_expectation_and_args
  55:        0x10d2a0f90 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_return_expr
  44:        0x10d04bd60 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_expr_with_expectation_and_args
  45:        0x10d04cf90 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_return_expr
  56:        0x10d39e180 - rustc_hir_typeck[4ee6bcd98d40a6cc]::check::check_fn
  57:        0x10d3c2470 - rustc_hir_typeck[4ee6bcd98d40a6cc]::typeck
  46:        0x10d14a180 - rustc_hir_typeck[4ee6bcd98d40a6cc]::check::check_fn
  58:        0x10e143a24 - <rustc_middle[a12d4e31e9c7c7b2]::dep_graph::dep_node::DepKind as rustc_query_system[4ec8b7e6ed7d2b80]::dep_graph::DepKind>::with_deps::<<rustc_query_system[4ec8b7e6ed7d2b80]::dep_graph::graph::DepGraph<rustc_middle[a12d4e31e9c7c7b2]::dep_graph::dep_node::DepKind>>::with_task_impl<rustc_query_impl[4d60d89ac47f5f83]::plumbing::QueryCtxt, rustc_span[9bec1e8d1a20d9df]::def_id::LocalDefId, &rustc_middle[a12d4e31e9c7c7b2]::ty::typeck_results::TypeckResults>::{closure#0}, &rustc_middle[a12d4e31e9c7c7b2]::ty::typeck_results::TypeckResults>
  47:        0x10d08defc - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_expr_closure
  48:        0x10d090240 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_expr_kind
  59:        0x10e0b0114 - rustc_query_system[4ec8b7e6ed7d2b80]::query::plumbing::try_execute_query::<rustc_query_impl[4d60d89ac47f5f83]::queries::typeck, rustc_query_impl[4d60d89ac47f5f83]::plumbing::QueryCtxt>
  49:        0x10d04bd60 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_expr_with_expectation_and_args
  50:        0x10d05ac8c - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_argument_types
  51:        0x10d03cb10 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::confirm_builtin_call
  52:        0x10d03b9a0 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_call
  60:        0x10df98550 - rustc_query_system[4ec8b7e6ed7d2b80]::query::plumbing::force_query::<rustc_query_impl[4d60d89ac47f5f83]::queries::typeck, rustc_query_impl[4d60d89ac47f5f83]::plumbing::QueryCtxt, rustc_middle[a12d4e31e9c7c7b2]::dep_graph::dep_node::DepKind>
  61:        0x10e1f645c - rustc_query_impl[4d60d89ac47f5f83]::plumbing::force_from_dep_node::<rustc_query_impl[4d60d89ac47f5f83]::queries::typeck>
  62:        0x10e1c613c - <rustc_query_system[4ec8b7e6ed7d2b80]::dep_graph::graph::DepGraph<rustc_middle[a12d4e31e9c7c7b2]::dep_graph::dep_node::DepKind>>::try_mark_previous_green::<rustc_query_impl[4d60d89ac47f5f83]::plumbing::QueryCtxt>
  63:        0x10e1c6184 - <rustc_query_system[4ec8b7e6ed7d2b80]::dep_graph::graph::DepGraph<rustc_middle[a12d4e31e9c7c7b2]::dep_graph::dep_node::DepKind>>::try_mark_previous_green::<rustc_query_impl[4d60d89ac47f5f83]::plumbing::QueryCtxt>
  53:        0x10d090744 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_expr_kind
  54:        0x10d04bd60 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_expr_with_expectation_and_args
  55:        0x10d04cf90 - <rustc_hir_typeck[4ee6bcd98d40a6cc]::fn_ctxt::FnCtxt>::check_return_expr
  56:        0x10d14a180 - rustc_hir_typeck[4ee6bcd98d40a6cc]::check::check_fn
  57:        0x10d16e470 - rustc_hir_typeck[4ee6bcd98d40a6cc]::typeck
  64:        0x10e1c6184 - <rustc_query_system[4ec8b7e6ed7d2b80]::dep_graph::graph::DepGraph<rustc_middle[a12d4e31e9c7c7b2]::dep_graph::dep_node::DepKind>>::try_mark_previous_green::<rustc_query_impl[4d60d89ac47f5f83]::plumbing::QueryCtxt>
  65:        0x10e1c6184 - <rustc_query_system[4ec8b7e6ed7d2b80]::dep_graph::graph::DepGraph<rustc_middle[a12d4e31e9c7c7b2]::dep_graph::dep_node::DepKind>>::try_mark_previous_green::<rustc_query_impl[4d60d89ac47f5f83]::plumbing::QueryCtxt>
  66:        0x10e1c6184 - <rustc_query_system[4ec8b7e6ed7d2b80]::dep_graph::graph::DepGraph<rustc_middle[a12d4e31e9c7c7b2]::dep_graph::dep_node::DepKind>>::try_mark_previous_green::<rustc_query_impl[4d60d89ac47f5f83]::plumbing::QueryCtxt>
  67:        0x10e1c6184 - <rustc_query_system[4ec8b7e6ed7d2b80]::dep_graph::graph::DepGraph<rustc_middle[a12d4e31e9c7c7b2]::dep_graph::dep_node::DepKind>>::try_mark_previous_green::<rustc_query_impl[4d60d89ac47f5f83]::plumbing::QueryCtxt>
  68:        0x10e1c6184 - <rustc_query_system[4ec8b7e6ed7d2b80]::dep_graph::graph::DepGraph<rustc_middle[a12d4e31e9c7c7b2]::dep_graph::dep_node::DepKind>>::try_mark_previous_green::<rustc_query_impl[4d60d89ac47f5f83]::plumbing::QueryCtxt>
  58:        0x10deefa24 - <rustc_middle[a12d4e31e9c7c7b2]::dep_graph::dep_node::DepKind as rustc_query_system[4ec8b7e6ed7d2b80]::dep_graph::DepKind>::with_deps::<<rustc_query_system[4ec8b7e6ed7d2b80]::dep_graph::graph::DepGraph<rustc_middle[a12d4e31e9c7c7b2]::dep_graph::dep_node::DepKind>>::with_task_impl<rustc_query_impl[4d60d89ac47f5f83]::plumbing::QueryCtxt, rustc_span[9bec1e8d1a20d9df]::def_id::LocalDefId, &rustc_middle[a12d4e31e9c7c7b2]::ty::typeck_results::TypeckResults>::{closure#0}, &rustc_middle[a12d4e31e9c7c7b2]::ty::typeck_results::TypeckResults>
  59:        0x10de5c114 - rustc_query_system[4ec8b7e6ed7d2b80]::query::plumbing::try_execute_query::<rustc_query_impl[4d60d89ac47f5f83]::queries::typeck, rustc_query_impl[4d60d89ac47f5f83]::plumbing::QueryCtxt>
  60:        0x10dd44550 - rustc_query_system[4ec8b7e6ed7d2b80]::query::plumbing::force_query::<rustc_query_impl[4d60d89ac47f5f83]::queries::typeck, rustc_query_impl[4d60d89ac47f5f83]::plumbing::QueryCtxt, rustc_middle[a12d4e31e9c7c7b2]::dep_graph::dep_node::DepKind>
  69:        0x10e1c4bb8 - <rustc_query_system[4ec8b7e6ed7d2b80]::dep_graph::graph::DepGraph<rustc_middle[a12d4e31e9c7c7b2]::dep_graph::dep_node::DepKind>>::try_mark_green::<rustc_query_impl[4d60d89ac47f5f83]::plumbing::QueryCtxt>
  70:        0x10e0b1c80 - rustc_query_system[4ec8b7e6ed7d2b80]::query::plumbing::try_execute_query::<rustc_query_impl[4d60d89ac47f5f83]::queries::type_of, rustc_query_impl[4d60d89ac47f5f83]::plumbing::QueryCtxt>
  71:        0x10e167350 - <rustc_query_impl[4d60d89ac47f5f83]::Queries as rustc_middle[a12d4e31e9c7c7b2]::ty::query::QueryEngine>::type_of
  72:        0x10d43bcf8 - rustc_hir_analysis[5e14f7d1cb611e1f]::check::check::check_item_type
  73:        0x10d444680 - rustc_hir_analysis[5e14f7d1cb611e1f]::check::check::check_mod_item_types
  61:        0x10dfa245c - rustc_query_impl[4d60d89ac47f5f83]::plumbing::force_from_dep_node::<rustc_query_impl[4d60d89ac47f5f83]::queries::typeck>
  62:        0x10df7213c - <rustc_query_system[4ec8b7e6ed7d2b80]::dep_graph::graph::DepGraph<rustc_middle[a12d4e31e9c7c7b2]::dep_graph::dep_node::DepKind>>::try_mark_previous_green::<rustc_query_impl[4d60d89ac47f5f83]::plumbing::QueryCtxt>
  63:        0x10df72184 - <rustc_query_system[4ec8b7e6ed7d2b80]::dep_graph::graph::DepGraph<rustc_middle[a12d4e31e9c7c7b2]::dep_graph::dep_node::DepKind>>::try_mark_previous_green::<rustc_query_impl[4d60d89ac47f5f83]::plumbing::QueryCtxt>
  64:        0x10df72184 - <rustc_query_system[4ec8b7e6ed7d2b80]::dep_graph::graph::DepGraph<rustc_middle[a12d4e31e9c7c7b2]::dep_graph::dep_node::DepKind>>::try_mark_previous_green::<rustc_query_impl[4d60d89ac47f5f83]::plumbing::QueryCtxt>
error: could not compile `ioxd_router` due to previous error
  74:        0x10e1426bc - <rustc_middle[a12d4e31e9c7c7b2]::dep_graph::dep_node::DepKind as rustc_query_system[4ec8b7e6ed7d2b80]::dep_graph::DepKind>::with_deps::<<rustc_query_system[4ec8b7e6ed7d2b80]::dep_graph::graph::DepGraph<rustc_middle[a12d4e31e9c7c7b2]::dep_graph::dep_node::DepKind>>::with_task_impl<rustc_query_impl[4d60d89ac47f5f83]::plumbing::QueryCtxt, core[446d629763b4aa]::option::Option<rustc_span[9bec1e8d1a20d9df]::symbol::Symbol>, ()>::{closure#0}, ()>
  65:        0x10df72184 - <rustc_query_system[4ec8b7e6ed7d2b80]::dep_graph::graph::DepGraph<rustc_middle[a12d4e31e9c7c7b2]::dep_graph::dep_node::DepKind>>::try_mark_previous_green::<rustc_query_impl[4d60d89ac47f5f83]::plumbing::QueryCtxt>
  66:        0x10df72184 - <rustc_query_system[4ec8b7e6ed7d2b80]::dep_graph::graph::DepGraph<rustc_middle[a12d4e31e9c7c7b2]::dep_graph::dep_node::DepKind>>::try_mark_previous_green::<rustc_query_impl[4d60d89ac47f5f83]::plumbing::QueryCtxt>
  67:        0x10df72184 - <rustc_query_system[4ec8b7e6ed7d2b80]::dep_graph::graph::DepGraph<rustc_middle[a12d4e31e9c7c7b2]::dep_graph::dep_node::DepKind>>::try_mark_previous_green::<rustc_query_impl[4d60d89ac47f5f83]::plumbing::QueryCtxt>
  75:        0x10e0363bc - rustc_query_system[4ec8b7e6ed7d2b80]::query::plumbing::try_execute_query::<rustc_query_impl[4d60d89ac47f5f83]::queries::check_mod_item_types, rustc_query_impl[4d60d89ac47f5f83]::plumbing::QueryCtxt>
  76:        0x10e16f964 - <rustc_query_impl[4d60d89ac47f5f83]::Queries as rustc_middle[a12d4e31e9c7c7b2]::ty::query::QueryEngine>::check_mod_item_types
  68:        0x10df72184 - <rustc_query_system[4ec8b7e6ed7d2b80]::dep_graph::graph::DepGraph<rustc_middle[a12d4e31e9c7c7b2]::dep_graph::dep_node::DepKind>>::try_mark_previous_green::<rustc_query_impl[4d60d89ac47f5f83]::plumbing::QueryCtxt>
  77:        0x10d49ba4c - <rustc_middle[a12d4e31e9c7c7b2]::hir::map::Map>::for_each_module::<rustc_hir_analysis[5e14f7d1cb611e1f]::check_crate::{closure#6}::{closure#0}>
  69:        0x10df70bb8 - <rustc_query_system[4ec8b7e6ed7d2b80]::dep_graph::graph::DepGraph<rustc_middle[a12d4e31e9c7c7b2]::dep_graph::dep_node::DepKind>>::try_mark_green::<rustc_query_impl[4d60d89ac47f5f83]::plumbing::QueryCtxt>
  70:        0x10de5dc80 - rustc_query_system[4ec8b7e6ed7d2b80]::query::plumbing::try_execute_query::<rustc_query_impl[4d60d89ac47f5f83]::queries::type_of, rustc_query_impl[4d60d89ac47f5f83]::plumbing::QueryCtxt>
  78:        0x10d4bba08 - <rustc_session[f1d766c083bcad2f]::session::Session>::time::<(), rustc_hir_analysis[5e14f7d1cb611e1f]::check_crate::{closure#6}>
  71:        0x10df13350 - <rustc_query_impl[4d60d89ac47f5f83]::Queries as rustc_middle[a12d4e31e9c7c7b2]::ty::query::QueryEngine>::type_of
  79:        0x10d500d50 - rustc_hir_analysis[5e14f7d1cb611e1f]::check_crate
  72:        0x10d1e7cf8 - rustc_hir_analysis[5e14f7d1cb611e1f]::check::check::check_item_type
  80:        0x10b0b266c - rustc_interface[812bb1defcb7896d]::passes::analysis
  73:        0x10d1f0680 - rustc_hir_analysis[5e14f7d1cb611e1f]::check::check::check_mod_item_types
  74:        0x10deee6bc - <rustc_middle[a12d4e31e9c7c7b2]::dep_graph::dep_node::DepKind as rustc_query_system[4ec8b7e6ed7d2b80]::dep_graph::DepKind>::with_deps::<<rustc_query_system[4ec8b7e6ed7d2b80]::dep_graph::graph::DepGraph<rustc_middle[a12d4e31e9c7c7b2]::dep_graph::dep_node::DepKind>>::with_task_impl<rustc_query_impl[4d60d89ac47f5f83]::plumbing::QueryCtxt, core[446d629763b4aa]::option::Option<rustc_span[9bec1e8d1a20d9df]::symbol::Symbol>, ()>::{closure#0}, ()>
  81:        0x10e147f64 - <rustc_middle[a12d4e31e9c7c7b2]::dep_graph::dep_node::DepKind as rustc_query_system[4ec8b7e6ed7d2b80]::dep_graph::DepKind>::with_deps::<<rustc_query_system[4ec8b7e6ed7d2b80]::dep_graph::graph::DepGraph<rustc_middle[a12d4e31e9c7c7b2]::dep_graph::dep_node::DepKind>>::with_task_impl<rustc_query_impl[4d60d89ac47f5f83]::plumbing::QueryCtxt, (), core[446d629763b4aa]::result::Result<(), rustc_span[9bec1e8d1a20d9df]::ErrorGuaranteed>>::{closure#0}, core[446d629763b4aa]::result::Result<(), rustc_span[9bec1e8d1a20d9df]::ErrorGuaranteed>>
  75:        0x10dde23bc - rustc_query_system[4ec8b7e6ed7d2b80]::query::plumbing::try_execute_query::<rustc_query_impl[4d60d89ac47f5f83]::queries::check_mod_item_types, rustc_query_impl[4d60d89ac47f5f83]::plumbing::QueryCtxt>
  82:        0x10e0b2df4 - rustc_query_system[4ec8b7e6ed7d2b80]::query::plumbing::try_execute_query::<rustc_query_impl[4d60d89ac47f5f83]::queries::analysis, rustc_query_impl[4d60d89ac47f5f83]::plumbing::QueryCtxt>
  76:        0x10df1b964 - <rustc_query_impl[4d60d89ac47f5f83]::Queries as rustc_middle[a12d4e31e9c7c7b2]::ty::query::QueryEngine>::check_mod_item_types
  83:        0x10e16793c - <rustc_query_impl[4d60d89ac47f5f83]::Queries as rustc_middle[a12d4e31e9c7c7b2]::ty::query::QueryEngine>::analysis
  84:        0x10b0625c0 - <rustc_middle[a12d4e31e9c7c7b2]::ty::context::GlobalCtxt>::enter::<rustc_driver_impl[4da67b3d18d5a97a]::run_compiler::{closure#1}::{closure#2}::{closure#4}, core[446d629763b4aa]::result::Result<(), rustc_span[9bec1e8d1a20d9df]::ErrorGuaranteed>>
  77:        0x10d247a4c - <rustc_middle[a12d4e31e9c7c7b2]::hir::map::Map>::for_each_module::<rustc_hir_analysis[5e14f7d1cb611e1f]::check_crate::{closure#6}::{closure#0}>
  78:        0x10d267a08 - <rustc_session[f1d766c083bcad2f]::session::Session>::time::<(), rustc_hir_analysis[5e14f7d1cb611e1f]::check_crate::{closure#6}>
  79:        0x10d2acd50 - rustc_hir_analysis[5e14f7d1cb611e1f]::check_crate
error: could not compile `ioxd_router` due to previous error
  85:        0x10b061b20 - rustc_span[9bec1e8d1a20d9df]::with_source_map::<core[446d629763b4aa]::result::Result<(), rustc_span[9bec1e8d1a20d9df]::ErrorGuaranteed>, rustc_interface[812bb1defcb7896d]::interface::run_compiler<core[446d629763b4aa]::result::Result<(), rustc_span[9bec1e8d1a20d9df]::ErrorGuaranteed>, rustc_driver_impl[4da67b3d18d5a97a]::run_compiler::{closure#1}>::{closure#0}::{closure#0}>
  80:        0x10ae5e66c - rustc_interface[812bb1defcb7896d]::passes::analysis
  81:        0x10def3f64 - <rustc_middle[a12d4e31e9c7c7b2]::dep_graph::dep_node::DepKind as rustc_query_system[4ec8b7e6ed7d2b80]::dep_graph::DepKind>::with_deps::<<rustc_query_system[4ec8b7e6ed7d2b80]::dep_graph::graph::DepGraph<rustc_middle[a12d4e31e9c7c7b2]::dep_graph::dep_node::DepKind>>::with_task_impl<rustc_query_impl[4d60d89ac47f5f83]::plumbing::QueryCtxt, (), core[446d629763b4aa]::result::Result<(), rustc_span[9bec1e8d1a20d9df]::ErrorGuaranteed>>::{closure#0}, core[446d629763b4aa]::result::Result<(), rustc_span[9bec1e8d1a20d9df]::ErrorGuaranteed>>
  86:        0x10b023f84 - std[57ecbc5b2c1792f]::sys_common::backtrace::__rust_begin_short_backtrace::<rustc_interface[812bb1defcb7896d]::util::run_in_thread_pool_with_globals<rustc_interface[812bb1defcb7896d]::interface::run_compiler<core[446d629763b4aa]::result::Result<(), rustc_span[9bec1e8d1a20d9df]::ErrorGuaranteed>, rustc_driver_impl[4da67b3d18d5a97a]::run_compiler::{closure#1}>::{closure#0}, core[446d629763b4aa]::result::Result<(), rustc_span[9bec1e8d1a20d9df]::ErrorGuaranteed>>::{closure#0}::{closure#0}, core[446d629763b4aa]::result::Result<(), rustc_span[9bec1e8d1a20d9df]::ErrorGuaranteed>>
  82:        0x10de5edf4 - rustc_query_system[4ec8b7e6ed7d2b80]::query::plumbing::try_execute_query::<rustc_query_impl[4d60d89ac47f5f83]::queries::analysis, rustc_query_impl[4d60d89ac47f5f83]::plumbing::QueryCtxt>
  83:        0x10df1393c - <rustc_query_impl[4d60d89ac47f5f83]::Queries as rustc_middle[a12d4e31e9c7c7b2]::ty::query::QueryEngine>::analysis
  84:        0x10ae0e5c0 - <rustc_middle[a12d4e31e9c7c7b2]::ty::context::GlobalCtxt>::enter::<rustc_driver_impl[4da67b3d18d5a97a]::run_compiler::{closure#1}::{closure#2}::{closure#4}, core[446d629763b4aa]::result::Result<(), rustc_span[9bec1e8d1a20d9df]::ErrorGuaranteed>>
  85:        0x10ae0db20 - rustc_span[9bec1e8d1a20d9df]::with_source_map::<core[446d629763b4aa]::result::Result<(), rustc_span[9bec1e8d1a20d9df]::ErrorGuaranteed>, rustc_interface[812bb1defcb7896d]::interface::run_compiler<core[446d629763b4aa]::result::Result<(), rustc_span[9bec1e8d1a20d9df]::ErrorGuaranteed>, rustc_driver_impl[4da67b3d18d5a97a]::run_compiler::{closure#1}>::{closure#0}::{closure#0}>
  87:        0x10b025d20 - <<std[57ecbc5b2c1792f]::thread::Builder>::spawn_unchecked_<rustc_interface[812bb1defcb7896d]::util::run_in_thread_pool_with_globals<rustc_interface[812bb1defcb7896d]::interface::run_compiler<core[446d629763b4aa]::result::Result<(), rustc_span[9bec1e8d1a20d9df]::ErrorGuaranteed>, rustc_driver_impl[4da67b3d18d5a97a]::run_compiler::{closure#1}>::{closure#0}, core[446d629763b4aa]::result::Result<(), rustc_span[9bec1e8d1a20d9df]::ErrorGuaranteed>>::{closure#0}::{closure#0}, core[446d629763b4aa]::result::Result<(), rustc_span[9bec1e8d1a20d9df]::ErrorGuaranteed>>::{closure#1} as core[446d629763b4aa]::ops::function::FnOnce<()>>::call_once::{shim:vtable#0}
  88:        0x102f4ba24 - std::sys::unix::thread::Thread::new::thread_start::h5db5d66cd91ffb33
  86:        0x10adcff84 - std[57ecbc5b2c1792f]::sys_common::backtrace::__rust_begin_short_backtrace::<rustc_interface[812bb1defcb7896d]::util::run_in_thread_pool_with_globals<rustc_interface[812bb1defcb7896d]::interface::run_compiler<core[446d629763b4aa]::result::Result<(), rustc_span[9bec1e8d1a20d9df]::ErrorGuaranteed>, rustc_driver_impl[4da67b3d18d5a97a]::run_compiler::{closure#1}>::{closure#0}, core[446d629763b4aa]::result::Result<(), rustc_span[9bec1e8d1a20d9df]::ErrorGuaranteed>>::{closure#0}::{closure#0}, core[446d629763b4aa]::result::Result<(), rustc_span[9bec1e8d1a20d9df]::ErrorGuaranteed>>
  87:        0x10add1d20 - <<std[57ecbc5b2c1792f]::thread::Builder>::spawn_unchecked_<rustc_interface[812bb1defcb7896d]::util::run_in_thread_pool_with_globals<rustc_interface[812bb1defcb7896d]::interface::run_compiler<core[446d629763b4aa]::result::Result<(), rustc_span[9bec1e8d1a20d9df]::ErrorGuaranteed>, rustc_driver_impl[4da67b3d18d5a97a]::run_compiler::{closure#1}>::{closure#0}, core[446d629763b4aa]::result::Result<(), rustc_span[9bec1e8d1a20d9df]::ErrorGuaranteed>>::{closure#0}::{closure#0}, core[446d629763b4aa]::result::Result<(), rustc_span[9bec1e8d1a20d9df]::ErrorGuaranteed>>::{closure#1} as core[446d629763b4aa]::ops::function::FnOnce<()>>::call_once::{shim:vtable#0}
  88:        0x102cf7a24 - std::sys::unix::thread::Thread::new::thread_start::h5db5d66cd91ffb33
  89:        0x1990f3fa8 - __pthread_joiner_wake

  89:        0x1990f3fa8 - __pthread_joiner_wake

error: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: rustc 1.69.0 (84c898d65 2023-04-16) running on aarch64-apple-darwin
error: the compiler unexpectedly panicked. this is a bug.

note: compiler flags: -C embed-bitcode=no -C split-debuginfo=unpacked -C debuginfo=2 -C incremental=[REDACTED]

note: some of the compiler flags provided by cargo are hidden

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

query stack during panic:

note: rustc 1.69.0 (84c898d65 2023-04-16) running on aarch64-apple-darwin

note: compiler flags: -C embed-bitcode=no -C split-debuginfo=unpacked -C debuginfo=2 -C incremental=[REDACTED]

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
#0 [evaluate_obligation] evaluating trait selection obligation `router::dml_handlers::instrumentation::InstrumentationDecorator<router::dml_handlers::chain::Chain<router::dml_handlers::chain::Chain<router::dml_handlers::chain::Chain<router::dml_handlers::retention_validation::RetentionValidator<alloc::sync::Arc<router::namespace_cache::read_through_cache::ReadThroughCache<alloc::sync::Arc<router::namespace_cache::sharded_cache::ShardedCache<alloc::sync::Arc<router::namespace_cache::memory::MemoryNamespaceCache>>>>>>, router::dml_handlers::schema_validation::SchemaValidator<alloc::sync::Arc<router::namespace_cache::read_through_cache::ReadThroughCache<alloc::sync::Arc<router::namespace_cache::sharded_cache::ShardedCache<alloc::sync::Arc<router::namespace_cache::memory::MemoryNamespaceCache>>>>>>>, router::dml_handlers::partitioner::Partitioner>, router::dml_handlers::fan_out::FanOutAdaptor<router::dml_handlers::rpc_write::RpcWrite<alloc::sync::Arc<router::dml_handlers::rpc_write::client::mock::MockWriteClient>>, alloc::vec::Vec<router::dml_handlers::partitioner::Partitioned<hashbrown::map::HashMap<data_types::TableId, (alloc::string::String, mutable_batch::MutableBatch)>>>>>>: router::dml_handlers::r#trait::DmlHandler`
#0 [evaluate_obligation] evaluating trait selection obligation `router::dml_handlers::instrumentation::InstrumentationDecorator<router::dml_handlers::chain::Chain<router::dml_handlers::chain::Chain<router::dml_handlers::chain::Chain<router::dml_handlers::retention_validation::RetentionValidator<alloc::sync::Arc<router::namespace_cache::read_through_cache::ReadThroughCache<alloc::sync::Arc<router::namespace_cache::sharded_cache::ShardedCache<alloc::sync::Arc<router::namespace_cache::memory::MemoryNamespaceCache>>>>>>, router::dml_handlers::schema_validation::SchemaValidator<alloc::sync::Arc<router::namespace_cache::read_through_cache::ReadThroughCache<alloc::sync::Arc<router::namespace_cache::sharded_cache::ShardedCache<alloc::sync::Arc<router::namespace_cache::memory::MemoryNamespaceCache>>>>>>>, router::dml_handlers::partitioner::Partitioner>, router::dml_handlers::fan_out::FanOutAdaptor<router::dml_handlers::rpc_write::RpcWrite<alloc::sync::Arc<router::dml_handlers::rpc_write::client::mock::MockWriteClient>>, alloc::vec::Vec<router::dml_handlers::partitioner::Partitioned<hashbrown::map::HashMap<data_types::TableId, (alloc::string::String, mutable_batch::MutableBatch)>>>>>>: router::dml_handlers::r#trait::DmlHandler`
#1 [typeck] type-checking `common::<impl at router/tests/common/mod.rs:140:1: 140:17>::write_lp`
#2 [type_of] computing type of `common::<impl at router/tests/common/mod.rs:140:1: 140:17>::write_lp::{opaque#1}`
#1 [typeck] type-checking `common::<impl at router/tests/common/mod.rs:140:1: 140:17>::write_lp`
#3 [check_mod_item_types] checking item types in module `common`
#2 [type_of] computing type of `common::<impl at router/tests/common/mod.rs:140:1: 140:17>::write_lp::{opaque#1}`
#4 [analysis] running analysis passes on this crate
end of query stack
#3 [check_mod_item_types] checking item types in module `common`
#4 [analysis] running analysis passes on this crate
end of query stack
encountered while trying to mark dependency green: typeck(b86577fc7f83c507-318c169493102de0)
encountered while trying to mark dependency green: thir_body(93b93ee5ce30ce31-fcf1e3c9721c3b7)
encountered while trying to mark dependency green: mir_built(93b93ee5ce30ce31-fcf1e3c9721c3b7)
encountered while trying to mark dependency green: typeck(f4e722dc2030f5ed-4544494f74a2894f)
encountered while trying to mark dependency green: unsafety_check_result(b86577fc7f83c507-318c169493102de0)
encountered while trying to mark dependency green: mir_const(93b93ee5ce30ce31-fcf1e3c9721c3b7)
encountered while trying to mark dependency green: mir_promoted(93b93ee5ce30ce31-fcf1e3c9721c3b7)
encountered while trying to mark dependency green: thir_body(768a65407ef8cde8-4144889007a5e618)
encountered while trying to mark dependency green: mir_borrowck(b86577fc7f83c507-318c169493102de0)
encountered while trying to mark dependency green: mir_built(768a65407ef8cde8-4144889007a5e618)
encountered while trying to mark dependency green: type_of(b86577fc7f83c507-c9f6974a71c7c612)
encountered while trying to mark dependency green: unsafety_check_result(f4e722dc2030f5ed-4544494f74a2894f)
encountered while trying to mark dependency green: mir_const(768a65407ef8cde8-4144889007a5e618)
encountered while trying to mark dependency green: mir_promoted(768a65407ef8cde8-4144889007a5e618)
encountered while trying to mark dependency green: mir_borrowck(f4e722dc2030f5ed-4544494f74a2894f)
encountered while trying to mark dependency green: type_of(f4e722dc2030f5ed-d996c3ba71858dac)
error: could not compile `router` due to 2 previous errors
error: could not compile `router` due to 2 previous errors
Some errors have detailed explanations: E0050, E0061.
For more information about an error, try `rustc --explain E0050`.
error: could not compile `router` due to 29 previous errors

[xxchan]

Hit this problem recently. And I can reproduce it stably:

git clone https://github.com/risingwavelabs/risingwave

git checkout 487c4c60c584e1e8d603b0a6d2af4632aa869a47 # this commit updated protobuf
cargo build -p risingwave_pb 

git checkout HEAD~
cargo build -p risingwave_pb  # boom

rustc 1.71.0-nightly (c373194cb 2023-05-24)
binary: rustc
commit-hash: c373194cb6d882dc455a588bcc29c92a96b50252
commit-date: 2023-05-24
host: aarch64-apple-darwin
release: 1.71.0-nightly
LLVM version: 16.0.4

error: internal compiler error: encountered incremental compilation error with mir_built(risingwave_pb[4467]::meta::subscribe_response::{impl#17}::merge::{closure#8})
  |
  = help: This is a known issue with the compiler. Run `cargo clean -p risingwave_pb` or `cargo clean` to allow your project to compile
  = note: Please follow the instructions below to create a bug report with the provided information
  = note: See <https://github.com/rust-lang/rust/issues/84970> for more information

thread 'rustc' panicked at 'Found unstable fingerprints for mir_built(risingwave_pb[4467]::meta::subscribe_response::{impl#17}::merge::{closure#8}): Steal { value: RwLock(RefCell { value: Some(Body { basic_blocks: BasicBlocks { basic_blocks: [BasicBlockData { statements: [StorageLive(_3), StorageLive(_4), StorageLive(_5), _5 = move (_1.1: meta::MetaSnapshot), _4 = meta::subscribe_response::Info::Snapshot(move _5)], terminator: Some(Terminator { source_info: SourceInfo { span: src/prost/src/meta.rs:613:45: 613:46 (#3105), scope: scope[0] }, kind: drop(_5) -> [return: bb1, unwind: bb8] }), is_cleanup: false }, BasicBlockData { statements: [StorageDead(_5), _3 = std::option::Option::<meta::subscribe_response::Info>::Some(move _4)], terminator: Some(Terminator { source_info: SourceInfo { span: src/prost/src/meta.rs:613:45: 613:46 (#3105), scope: scope[0] }, kind: drop(_4) -> [return: bb2, unwind: bb7] }), is_cleanup: false }, BasicBlockData { statements: [StorageDead(_4)], terminator: Some(Terminator { source_info: SourceInfo { span: src/prost/src/meta.rs:613:32: 613:46 (#28903), scope: scope[0] }, kind: drop((*(_1.0: &mut std::option::Option<meta::subscribe_response::Info>))) -> [return: bb3, unwind: bb4] }), is_cleanup: false }, BasicBlockData { statements: [(*(_1.0: &mut std::option::Option<meta::subscribe_response::Info>)) = move _3, _0 = const ()], terminator: Some(Terminator { source_info: SourceInfo { span: src/prost/src/meta.rs:613:45: 613:46 (#3105), scope: scope[0] }, kind: drop(_3) -> [return: bb5, unwind: bb9] }), is_cleanup: false }, BasicBlockData { statements: [(*(_1.0: &mut std::option::Option<meta::subscribe_response::Info>)) = move _3], terminator: Some(Terminator { source_info: SourceInfo { span: src/prost/src/meta.rs:613:32: 613:46 (#28903), scope: scope[0] }, kind: goto -> bb7 }), is_cleanup: true }, BasicBlockData { statements: [StorageDead(_3)], terminator: Some(Terminator { source_info: SourceInfo { span: src/prost/src/meta.rs:613:45: 613:46 (#3105), scope: scope[0] }, kind: drop(_1) -> [return: bb6, unwind: bb10] }), is_cleanup: false }, BasicBlockData { statements: [], terminator: Some(Terminator { source_info: SourceInfo { span: src/prost/src/meta.rs:613:46: 613:46 (#3105), scope: scope[0] }, kind: return }), is_cleanup: false }, BasicBlockData { statements: [], terminator: Some(Terminator { source_info: SourceInfo { span: src/prost/src/meta.rs:613:45: 613:46 (#3105), scope: scope[0] }, kind: drop(_3) -> [return: bb9, unwind terminate] }), is_cleanup: true }, BasicBlockData { statements: [], terminator: Some(Terminator { source_info: SourceInfo { span: src/prost/src/meta.rs:613:45: 613:46 (#3105), scope: scope[0] }, kind: drop(_4) -> [return: bb9, unwind terminate] }), is_cleanup: true }, BasicBlockData { statements: [], terminator: Some(Terminator { source_info: SourceInfo { span: src/prost/src/meta.rs:613:45: 613:46 (#3105), scope: scope[0] }, kind: drop(_1) -> [return: bb10, unwind terminate] }), is_cleanup: true }, BasicBlockData { statements: [], terminator: Some(Terminator { source_info: SourceInfo { span: src/prost/src/meta.rs:613:32: 613:46 (#3105), scope: scope[0] }, kind: resume }), is_cleanup: true }], cache: Cache { predecessors: OnceCell(Uninit), switch_sources: OnceCell(Uninit), is_cyclic: OnceCell(Uninit), postorder: OnceCell(Uninit), dominators: OnceCell(Uninit) } }, phase: Built, pass_count: 0, source: MirSource { instance: Item(DefId(0:8602 ~ risingwave_pb[4467]::meta::subscribe_response::{impl#17}::merge::{closure#8})), promoted: None }, source_scopes: [SourceScopeData { span: src/prost/src/meta.rs:613:32: 613:46 (#3105), parent_scope: None, inlined: None, inlined_parent_scope: None, local_data: Set(SourceScopeLocalData { lint_root: HirId(DefId(0:8592 ~ risingwave_pb[4467]::meta::subscribe_response::{impl#17}::merge).837), safety: Safe }) }], generator: None, local_decls: [LocalDecl { mutability: Mut, local_info: Set(Boring), internal: false, ty: (), user_ty: None, source_info: SourceInfo { span: src/prost/src/meta.rs:613:32: 613:32 (#3105), scope: scope[0] } }, LocalDecl { mutability: Mut, local_info: Set(Boring), internal: false, ty: [closure@src/prost/src/meta.rs:613:32: 613:46], user_ty: None, source_info: SourceInfo { span: src/prost/src/meta.rs:613:32: 613:46 (#3105), scope: scope[0] } }, LocalDecl { mutability: Mut, local_info: Set(Boring), internal: false, ty: (), user_ty: None, source_info: SourceInfo { span: src/prost/src/meta.rs:613:32: 613:46 (#3105), scope: scope[0] } }, LocalDecl { mutability: Mut, local_info: Set(Boring), internal: false, ty: std::option::Option<meta::subscribe_response::Info>, user_ty: None, source_info: SourceInfo { span: src/prost/src/meta.rs:613:32: 613:46 (#3105), scope: scope[0] } }, LocalDecl { mutability: Mut, local_info: Set(AggregateTemp), internal: false, ty: meta::subscribe_response::Info, user_ty: None, source_info: SourceInfo { span: src/prost/src/meta.rs:613:32: 613:46 (#3105), scope: scope[0] } }, LocalDecl { mutability: Mut, local_info: Set(AggregateTemp), internal: false, ty: meta::MetaSnapshot, user_ty: None, source_info: SourceInfo { span: src/prost/src/meta.rs:613:32: 613:46 (#3105), scope: scope[0] } }], user_type_annotations: [], arg_count: 2, spread_arg: None, var_debug_info: [VarDebugInfo { name: "field", source_info: SourceInfo { span: src/prost/src/meta.rs:613:32: 613:46 (#26548), scope: scope[0] }, value: (*(_1.0: &mut std::option::Option<meta::subscribe_response::Info>)), argument_index: None, references: 0 }, VarDebugInfo { name: "owned_value", source_info: SourceInfo { span: src/prost/src/meta.rs:613:32: 613:46 (#26548), scope: scope[0] }, value: (_1.1: meta::MetaSnapshot), argument_index: None, references: 0 }], span: src/prost/src/meta.rs:613:32: 613:46 (#3105), required_consts: [], is_polymorphic: true, injection_phase: None, tainted_by_errors: None }) }) }', /rustc/c373194cb6d882dc455a588bcc29c92a96b50252/compiler/rustc_query_system/src/query/plumbing.rs:735:9
stack backtrace:
   0: _rust_begin_unwind
   1: core::panicking::panic_fmt
   2: rustc_query_system::query::plumbing::incremental_verify_ich_failed::<rustc_middle::ty::context::TyCtxt>
   3: rustc_query_system::query::plumbing::incremental_verify_ich::<rustc_middle::ty::context::TyCtxt, rustc_middle::query::erase::Erased<[u8; 8]>>
   4: rustc_query_system::query::plumbing::try_execute_query::<rustc_query_impl::DynamicConfig<rustc_query_system::query::caches::VecCache<rustc_span::def_id::LocalDefId, rustc_middle::query::erase::Erased<[u8; 8]>>, false, false, false>, rustc_query_impl::plumbing::QueryCtxt, true>
   5: rustc_mir_transform::mir_const
   6: rustc_mir_transform::mir_promoted
   7: rustc_borrowck::mir_borrowck
   8: <core::panic::unwind_safe::AssertUnwindSafe<rustc_data_structures::sync::par_for_each_in<&[rustc_span::def_id::LocalDefId], <rustc_middle::hir::map::Map>::par_body_owners<rustc_interface::passes::analysis::{closure#1}::{closure#0}>::{closure#0}>::{closure#0}::{closure#0}> as core::ops::function::FnOnce<()>>::call_once
   9: rustc_data_structures::sync::par_for_each_in::<&[rustc_span::def_id::LocalDefId], <rustc_middle::hir::map::Map>::par_body_owners<rustc_interface::passes::analysis::{closure#1}::{closure#0}>::{closure#0}>
  10: rustc_interface::passes::analysis
  11: <rustc_interface::queries::QueryResult<&rustc_middle::ty::context::GlobalCtxt>>::enter::<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_driver_impl::run_compiler::{closure#1}::{closure#2}::{closure#4}>
  12: <rustc_interface::interface::Compiler>::enter::<rustc_driver_impl::run_compiler::{closure#1}::{closure#2}, core::result::Result<core::option::Option<rustc_interface::queries::Linker>, rustc_span::ErrorGuaranteed>>
  13: rustc_span::set_source_map::<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_interface::interface::run_compiler<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_driver_impl::run_compiler::{closure#1}>::{closure#0}::{closure#0}>
  14: <scoped_tls::ScopedKey<rustc_span::SessionGlobals>>::set::<rustc_interface::interface::run_compiler<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_driver_impl::run_compiler::{closure#1}>::{closure#0}, core::result::Result<(), rustc_span::ErrorGuaranteed>>
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

error: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: rustc 1.71.0-nightly (c373194cb 2023-05-24) running on aarch64-apple-darwin

note: compiler flags: --crate-type lib -C lto=off -C embed-bitcode=no -C split-debuginfo=unpacked -C debuginfo=2 -C incremental=[REDACTED] -C link-arg=-fuse-ld=/opt/homebrew/opt/llvm/bin/ld64.lld

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
#0 [mir_built] building MIR for `meta::subscribe_response::<impl at src/prost/src/meta.rs:613:32: 613:46>::merge::{closure#8}`
#1 [mir_const] preparing `meta::subscribe_response::<impl at src/prost/src/meta.rs:613:32: 613:46>::merge::{closure#8}` for borrow checking
#2 [mir_promoted] promoting constants in MIR for `meta::subscribe_response::<impl at src/prost/src/meta.rs:613:32: 613:46>::merge::{closure#8}`
#3 [mir_borrowck] borrow-checking `meta::subscribe_response::<impl at src/prost/src/meta.rs:613:32: 613:46>::merge::{closure#8}`
#4 [analysis] running analysis passes on this crate
end of query stack
there was a panic while trying to force a dep node
try_mark_green dep node stack:
#0 mir_borrowck(risingwave_pb[4467]::meta::subscribe_response::{impl#17}::merge)
end of try_mark_green dep node stack
error: internal compiler error: reentrant incremental verify failure, suppressing message

error: could not compile `risingwave_pb` (lib) due to 2 previous errors

[bjorn3]

Does running the first build command with RUSTFLAGS="-Zincremental-verify-ich=yes" given an ICE?

[xxchan]

Does running the first build command with RUSTFLAGS="-Zincremental-verify-ich=yes" given an ICE?

No

[xxchan]

I did a bisect, and found the regression :

#!/bin/sh

set -ex

git checkout 487c4c60c584e1e8d603b0a6d2af4632aa869a47 # this commit updated protobuf
cargo clean -p risingwave_pb
cargo build -p risingwave_pb 

git checkout HEAD~
cargo build -p risingwave_pb  # boom

cargo bisect-rustc --start 2023-03-01 --end 2023-04-07 --script ./test.sh

searched nightlies: from nightly-2023-03-01 to nightly-2023-04-07
regressed nightly: nightly-2023-03-06
searched commit range: f15f0ea...7820b62
regressed commit: 14c54b6

[jyn514]

cc @zeegomo @cjgillot

[xxchan]

My issue is already fixed in latest nightly.

[jonathanhood]

I'm now seeing error messages related to this issue in 1.70.0 that did not appear in 1.69.0. I originally added info in #112479 before realizing the error message contained a link here 😅 . More context can be found in that (linked) issue.

[qredek]

I also did not realise that there was a tracking issue and added details in #113062.

[temportalflux]

I frequently hit this issue with mir_built while compiling my WASM web app using Yew & Web-Sys (among many other deps). Its possible this may be addressed already by #112137, but I'm using stable not nightly, so I wont know until that is stablized. Just adding this report+log here on the off chance its useful.

Found unstable fingerprints for mir_built(11a1f347035615da-af8058392ca9b038)

error: internal compiler error: encountered incremental compilation error with mir_built(11a1f347035615da-af8058392ca9b038)
  |
  = help: This is a known issue with the compiler. Run `cargo clean -p integro_tabletop` or `cargo clean` to allow your project to compile
  = note: Please follow the instructions below to create a bug report with the provided information
  = note: See <https://github.com/rust-lang/rust/issues/84970> for more information

thread 'rustc' panicked at 'Found unstable fingerprints for mir_built(11a1f347035615da-af8058392ca9b038): Steal { value: RwLock(RefCell { value: Some(Body { basic_blocks: BasicBlocks { basic_blocks: [BasicBlockData { statements: [StorageLive(_4), StorageLive(_5), StorageLive(_6), _6 = move _3], terminator: Some(Terminator { source_info: SourceInfo { span: src\components\annotated_number.rs:48:28: 48:38 (#342), scope: scope[0] }, kind: _5 = <impl ::yew::html::IntoPropValue<ChildrenWithProps<AnnotatedNumber>> as IntoPropValue<ChildrenRenderer<VChild<annotated_number::AnnotatedNumber>>>>::into_prop_value(move _6) -> [return: bb1, unwind: bb9] }), is_cleanup: false }, BasicBlockData { statements: [StorageDead(_6), _4 = std::option::Option::<ChildrenRenderer<VChild<annotated_number::AnnotatedNumber>>>::Some(move _5)], terminator: Some(Terminator { source_info: SourceInfo { span: src\components\annotated_number.rs:48:37: 48:38 (#342), scope: scope[0] }, kind: drop(_5) -> [return: bb2, unwind: 
bb8] }), is_cleanup: false }, BasicBlockData { statements: [StorageDead(_5)], terminator: Some(Terminator { source_info: SourceInfo { span: src\components\annotated_number.rs:48:28: 48:38 (#16570), scope: scope[0] }, kind: drop(((*((*_1).0: std::boxed::Box<components::annotated_number::AnnotatedNumberCardPropsWrapper>)).3: std::option::Option<yew::html::ChildrenRenderer<yew::virtual_dom::VChild<components::annotated_number::AnnotatedNumber>>>)) -> [return: bb3, unwind: bb4] }), is_cleanup: false 
}, BasicBlockData { statements: [((*((*_1).0: std::boxed::Box<components::annotated_number::AnnotatedNumberCardPropsWrapper>)).3: std::option::Option<yew::html::ChildrenRenderer<yew::virtual_dom::VChild<components::annotated_number::AnnotatedNumber>>>) = move _4], terminator: Some(Terminator { source_info: SourceInfo { span: src\components\annotated_number.rs:48:37: 48:38 (#342), scope: scope[0] }, kind: drop(_4) -> [return: bb5, unwind: bb10] }), is_cleanup: false }, BasicBlockData { statements: [((*((*_1).0: std::boxed::Box<components::annotated_number::AnnotatedNumberCardPropsWrapper>)).3: std::option::Option<yew::html::ChildrenRenderer<yew::virtual_dom::VChild<components::annotated_number::AnnotatedNumber>>>) = move _4], terminator: Some(Terminator { source_info: SourceInfo { span: src\components\annotated_number.rs:48:28: 48:38 (#16570), scope: scope[0] }, kind: goto -> bb8 }), is_cleanup: true }, BasicBlockData { statements: [StorageDead(_4), StorageLive(_7), _7 = PhantomData::<__YewTokenTy>, _0 = HasAnnotatedNumberCardPropschildren::<__YewTokenTy>(move _7), StorageDead(_7)], terminator: Some(Terminator { source_info: SourceInfo { span: src\components\annotated_number.rs:48:37: 48:38 (#342), scope: scope[0] }, kind: drop(_3) -> [return: bb6, unwind: bb11] }), is_cleanup: false }, BasicBlockData { statements: 
[], terminator: Some(Terminator { source_info: SourceInfo { span: src\components\annotated_number.rs:48:37: 48:38 (#342), scope: scope[0] }, kind: drop(_2) -> [return: bb7, unwind: bb12] }), is_cleanup: false }, BasicBlockData { statements: [], terminator: Some(Terminator { source_info: SourceInfo { span: src\components\annotated_number.rs:48:38: 48:38 (#342), scope: scope[0] }, kind: return }), is_cleanup: false }, BasicBlockData { statements: [], terminator: Some(Terminator { source_info: SourceInfo { span: src\components\annotated_number.rs:48:37: 48:38 (#342), scope: scope[0] }, kind: drop(_4) -> [return: bb10, unwind terminate] }), is_cleanup: true }, BasicBlockData { statements: [], terminator: Some(Terminator { source_info: SourceInfo { span: src\components\annotated_number.rs:48:37: 48:38 (#342), scope: scope[0] }, 
kind: drop(_6) -> [return: bb10, unwind terminate] }), is_cleanup: true }, BasicBlockData { statements: [], terminator: Some(Terminator { source_info: SourceInfo { span: src\components\annotated_number.rs:48:37: 48:38 (#342), scope: scope[0] }, kind: drop(_3) -> [return: bb11, unwind terminate] }), is_cleanup: true }, BasicBlockData { statements: [], terminator: Some(Terminator { source_info: SourceInfo { span: src\components\annotated_number.rs:48:37: 48:38 (#342), scope: scope[0] }, kind: drop(_2) -> [return: bb12, unwind terminate] }), is_cleanup: true }, BasicBlockData { statements: [], terminator: Some(Terminator { source_info: SourceInfo { span: src\components\annotated_number.rs:48:28: 48:38 (#342), scope: scope[0] }, kind: resume }), is_cleanup: true }], cache: Cache { predecessors: OnceCell(Uninit), switch_sources: OnceCell(Uninit), is_cyclic: OnceCell(Uninit), postorder: OnceCell(Uninit) } }, phase: Built, pass_count: 0, source: MirSource { instance: Item(WithOptConstParam { did: DefId(0:6677 ~ integro_tabletop[e218]::components::annotated_number::{impl#18}::children), const_param_did: None }), promoted: None }, source_scopes: [SourceScopeData { span: src\components\annotated_number.rs:48:28: 48:38 (#342), parent_scope: None, inlined: None, inlined_parent_scope: None, local_data: Set(SourceScopeLocalData 
{ lint_root: HirId(DefId(0:6677 ~ integro_tabletop[e218]::components::annotated_number::{impl#18}::children).0), safety: Safe }) }], generator: None, local_decls: [LocalDecl { mutability: Mut, local_info: Set(Boring), internal: false, ty: components::annotated_number::HasAnnotatedNumberCardPropschildren<__YewTokenTy>, user_ty: None, source_info: SourceInfo { span: src\components\annotated_number.rs:48:28: 48:38 (#343), scope: scope[0] } }, LocalDecl { mutability: Not, local_info: Set(User(ImplicitSelf(MutRef))), internal: false, ty: &mut components::annotated_number::AnnotatedNumberCardPropsBuilder, user_ty: None, source_info: SourceInfo { span: src\components\annotated_number.rs:48:28: 48:38 (#342), scope: scope[0] } }, LocalDecl { mutability: Not, local_info: Set(User(Var(VarBindingForm { binding_mode: BindByValue(Not), opt_ty_info: Some(src\components\annotated_number.rs:48:28: 48:38 (#343)), opt_match_place: Some((None, src\components\annotated_number.rs:48:28: 48:38 (#342))), pat_span: src\components\annotated_number.rs:48:28: 48:38 (#342) }))), internal: false, ty: __YewTokenTy, user_ty: None, source_info: SourceInfo { span: src\components\annotated_number.rs:48:28: 48:38 (#342), scope: scope[0] } }, LocalDecl { mutability: Not, local_info: Set(User(Var(VarBindingForm { binding_mode: BindByValue(Not), opt_ty_info: None, opt_match_place: Some((None, src\components\annotated_number.rs:48:28: 48:38 (#342))), pat_span: src\components\annotated_number.rs:48:28: 48:38 (#342) }))), internal: false, ty: impl ::yew::html::IntoPropValue<ChildrenWithProps<AnnotatedNumber>>, user_ty: None, source_info: SourceInfo { span: src\components\annotated_number.rs:48:28: 48:38 (#342), scope: scope[0] } }, LocalDecl { mutability: Mut, local_info: Set(Boring), internal: false, ty: std::option::Option<yew::html::ChildrenRenderer<yew::virtual_dom::VChild<components::annotated_number::AnnotatedNumber>>>, user_ty: None, source_info: SourceInfo { span: src\components\annotated_number.rs:48:28: 48:38 (#342), scope: scope[0] } }, LocalDecl { mutability: Mut, local_info: Set(AggregateTemp), internal: false, ty: yew::html::ChildrenRenderer<yew::virtual_dom::VChild<components::annotated_number::AnnotatedNumber>>, user_ty: None, source_info: SourceInfo { span: src\components\annotated_number.rs:48:28: 48:38 (#342), scope: scope[0] } }, LocalDecl { mutability: Mut, local_info: Set(Boring), internal: false, ty: impl ::yew::html::IntoPropValue<ChildrenWithProps<AnnotatedNumber>>, user_ty: None, source_info: SourceInfo { span: src\components\annotated_number.rs:48:28: 48:38 (#342), scope: scope[0] } }, LocalDecl { mutability: Mut, local_info: Set(AggregateTemp), internal: false, ty: std::marker::PhantomData<__YewTokenTy>, user_ty: None, source_info: SourceInfo { span: src\components\annotated_number.rs:48:28: 48:38 (#342), scope: scope[0] } }], user_type_annotations: [], arg_count: 3, spread_arg: None, var_debug_info: [VarDebugInfo { name: "self", source_info: SourceInfo { span: src\components\annotated_number.rs:48:28: 48:38 (#342), scope: scope[0] }, value: _1, argument_index: Some(1) }, VarDebugInfo { name: "token", source_info: SourceInfo { span: src\components\annotated_number.rs:48:28: 48:38 (#342), scope: scope[0] }, value: _2, argument_index: Some(2) }, VarDebugInfo { name: "value", source_info: SourceInfo { span: src\components\annotated_number.rs:48:28: 48:38 (#342), scope: scope[0] }, value: _3, argument_index: Some(3) }], span: src\components\annotated_number.rs:48:28: 48:38 (#342), required_consts: [], is_polymorphic: true, injection_phase: None, tainted_by_errors: None }) }) }', /rustc/90c541806f23a127002de5b4038be731ba1458ca\compiler\rustc_query_system\src\query\plumbing.rs:715:9
stack backtrace:
   0:     0x7ff9c59b6c02 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::he4dbe7f5385cd1e2
   1:     0x7ff9c59f4f5b - core::fmt::write::h72b30843329f2c20
   2:     0x7ff9c59abe6a - <std::io::IoSliceMut as core::fmt::Debug>::fmt::h44d7f217ec8c08a4
   3:     0x7ff9c59b694b - std::sys::common::alloc::realloc_fallback::h25dd000eefb1a25f
   4:     0x7ff9c59ba2da - std::panicking::default_hook::h8d2d551f7739f430
   5:     0x7ff9c59b9f40 - std::panicking::default_hook::h8d2d551f7739f430
   6:     0x7ff9b31ff02e - rustc_driver_impl[39e757d53880b308]::describe_lints
   7:     0x7ff9c59babef - std::panicking::rust_panic_with_hook::h2561aa68cd5abc10
   8:     0x7ff9c59ba94e - <std::panicking::begin_panic_handler::StrPanicPayload as core::panic::BoxMeUp>::get::hcac4dd4b6ea7d20c
   9:     0x7ff9c59b78a9 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::he4dbe7f5385cd1e2
  10:     0x7ff9c59ba660 - rust_begin_unwind
  11:     0x7ff9c5a29165 - core::panicking::panic_fmt::he37b281957295da9
  12:     0x7ff9b50c6e0e - <&[rustc_ast[9d9e172e794f524b]::ast::Attribute] as rustc_serialize[a924d7a59c3ae2b4]::serialize::Decodable<rustc_query_impl[31ecb434a2a03a44]::on_disk_cache::CacheDecoder>>::decode
  13:     0x7ff9b156faa4 - <&rustc_index[16c4f1290dbf3aed]::vec::IndexVec<rustc_middle[298b6ec0f969779]::mir::Promoted, rustc_middle[298b6ec0f969779]::mir::Body> as rustc_serialize[a924d7a59c3ae2b4]::serialize::Decodable<rustc_query_impl[31ecb434a2a03a44]::on_disk_cache::CacheDecoder>>::decode
  14:     0x7ff9b2c815cf - <&[rustc_type_ir[e4a94c75124a4b3a]::Variance] as rustc_serialize[a924d7a59c3ae2b4]::serialize::Decodable<rustc_query_impl[31ecb434a2a03a44]::on_disk_cache::CacheDecoder>>::decode
  15:     0x7ff9b2b55957 - <rustc_query_impl[31ecb434a2a03a44]::Queries as rustc_middle[298b6ec0f969779]::ty::query::QueryEngine>::as_any
  16:     0x7ff9b25d58b0 - <rustc_mir_transform[d4e2a26eaceac4de]::dataflow_const_prop::DataflowConstProp as rustc_middle[298b6ec0f969779]::mir::MirPass>::is_enabled  
  17:     0x7ff9b5052c11 - <&[rustc_ast[9d9e172e794f524b]::ast::Attribute] as rustc_serialize[a924d7a59c3ae2b4]::serialize::Decodable<rustc_query_impl[31ecb434a2a03a44]::on_disk_cache::CacheDecoder>>::decode
  18:     0x7ff9b4ffa110 - <rustc_span[ad67946381e6ba24]::hygiene::ExpnId as rustc_serialize[a924d7a59c3ae2b4]::serialize::Encodable<rustc_query_impl[31ecb434a2a03a44]::on_disk_cache::CacheEncoder>>::encode
  19:     0x7ff9b1570335 - <&rustc_index[16c4f1290dbf3aed]::vec::IndexVec<rustc_middle[298b6ec0f969779]::mir::Promoted, rustc_middle[298b6ec0f969779]::mir::Body> as rustc_serialize[a924d7a59c3ae2b4]::serialize::Decodable<rustc_query_impl[31ecb434a2a03a44]::on_disk_cache::CacheDecoder>>::decode
  20:     0x7ff9b2c82b8f - <&[rustc_type_ir[e4a94c75124a4b3a]::Variance] as rustc_serialize[a924d7a59c3ae2b4]::serialize::Decodable<rustc_query_impl[31ecb434a2a03a44]::on_disk_cache::CacheDecoder>>::decode
  21:     0x7ff9b2b55b37 - <rustc_query_impl[31ecb434a2a03a44]::Queries as rustc_middle[298b6ec0f969779]::ty::query::QueryEngine>::as_any
  22:     0x7ff9b25d6831 - <rustc_mir_transform[d4e2a26eaceac4de]::dataflow_const_prop::DataflowConstProp as rustc_middle[298b6ec0f969779]::mir::MirPass>::is_enabled
  23:     0x7ff9b504d5e2 - <&[rustc_ast[9d9e172e794f524b]::ast::Attribute] as rustc_serialize[a924d7a59c3ae2b4]::serialize::Decodable<rustc_query_impl[31ecb434a2a03a44]::on_disk_cache::CacheDecoder>>::decode
  24:     0x7ff9b4ff598a - <rustc_span[ad67946381e6ba24]::hygiene::ExpnId as rustc_serialize[a924d7a59c3ae2b4]::serialize::Encodable<rustc_query_impl[31ecb434a2a03a44]::on_disk_cache::CacheEncoder>>::encode
  25:     0x7ff9b14ee20d - <&rustc_index[16c4f1290dbf3aed]::vec::IndexVec<rustc_middle[298b6ec0f969779]::mir::Promoted, rustc_middle[298b6ec0f969779]::mir::Body> as rustc_serialize[a924d7a59c3ae2b4]::serialize::Decodable<rustc_query_impl[31ecb434a2a03a44]::on_disk_cache::CacheDecoder>>::decode
  26:     0x7ff9b2c217b1 - <&[rustc_type_ir[e4a94c75124a4b3a]::Variance] as rustc_serialize[a924d7a59c3ae2b4]::serialize::Decodable<rustc_query_impl[31ecb434a2a03a44]::on_disk_cache::CacheDecoder>>::decode
  27:     0x7ff9b2b562ad - <rustc_query_impl[31ecb434a2a03a44]::Queries as rustc_middle[298b6ec0f969779]::ty::query::QueryEngine>::as_any
  28:     0x7ff9b28b55a2 - <rustc_middle[298b6ec0f969779]::ty::adjustment::AutoBorrowMutability as rustc_mir_build[692303aa0e857e66]::thir::cx::expr::ToBorrowKind>::to_borrow_kind
  29:     0x7ff9b14cb603 - <&rustc_data_structures[41be42171f3b2d17]::unord::UnordSet<rustc_span[ad67946381e6ba24]::def_id::LocalDefId> as rustc_serialize[a924d7a59c3ae2b4]::serialize::Decodable<rustc_query_impl[31ecb434a2a03a44]::on_disk_cache::CacheDecoder>>::decode
  30:     0x7ff9b14eba5c - <&rustc_index[16c4f1290dbf3aed]::vec::IndexVec<rustc_middle[298b6ec0f969779]::mir::Promoted, rustc_middle[298b6ec0f969779]::mir::Body> as rustc_serialize[a924d7a59c3ae2b4]::serialize::Decodable<rustc_query_impl[31ecb434a2a03a44]::on_disk_cache::CacheDecoder>>::decode
  31:     0x7ff9b2c1f4c7 - <&[rustc_type_ir[e4a94c75124a4b3a]::Variance] as rustc_serialize[a924d7a59c3ae2b4]::serialize::Decodable<rustc_query_impl[31ecb434a2a03a44]::on_disk_cache::CacheDecoder>>::decode
  32:     0x7ff9b2b59d20 - <rustc_query_impl[31ecb434a2a03a44]::Queries as rustc_middle[298b6ec0f969779]::ty::query::QueryEngine>::as_any
  33:     0x7ff9b1986e7a - once_cell[b880d516fb8c3c0e]::imp::initialize_or_wait
  34:     0x7ff9b0a8e52d - rustc_interface[2aa0624475714241]::callbacks::setup_callbacks
  35:     0x7ff9b0a7acf5 - rustc_interface[2aa0624475714241]::passes::analysis
  36:     0x7ff9b15803c1 - <&rustc_index[16c4f1290dbf3aed]::vec::IndexVec<rustc_middle[298b6ec0f969779]::mir::Promoted, rustc_middle[298b6ec0f969779]::mir::Body> as rustc_serialize[a924d7a59c3ae2b4]::serialize::Decodable<rustc_query_impl[31ecb434a2a03a44]::on_disk_cache::CacheDecoder>>::decode
  37:     0x7ff9b14cc13c - <&rustc_data_structures[41be42171f3b2d17]::unord::UnordSet<rustc_span[ad67946381e6ba24]::def_id::LocalDefId> as rustc_serialize[a924d7a59c3ae2b4]::serialize::Decodable<rustc_query_impl[31ecb434a2a03a44]::on_disk_cache::CacheDecoder>>::decode
  38:     0x7ff9b156afc5 - <&rustc_index[16c4f1290dbf3aed]::vec::IndexVec<rustc_middle[298b6ec0f969779]::mir::Promoted, rustc_middle[298b6ec0f969779]::mir::Body> as rustc_serialize[a924d7a59c3ae2b4]::serialize::Decodable<rustc_query_impl[31ecb434a2a03a44]::on_disk_cache::CacheDecoder>>::decode
  39:     0x7ff9b1613de8 - <&rustc_index[16c4f1290dbf3aed]::vec::IndexVec<rustc_middle[298b6ec0f969779]::mir::Promoted, rustc_middle[298b6ec0f969779]::mir::Body> as rustc_serialize[a924d7a59c3ae2b4]::serialize::Decodable<rustc_query_impl[31ecb434a2a03a44]::on_disk_cache::CacheDecoder>>::decode
  40:     0x7ff9b1477225 - <rustc_query_impl[31ecb434a2a03a44]::Queries as rustc_middle[298b6ec0f969779]::ty::query::QueryEngine>::try_mark_green
  41:     0x7ff9b0a27f31 - rustc_driver_impl[39e757d53880b308]::main
  42:     0x7ff9b0a35873 - <rustc_middle[298b6ec0f969779]::ty::SymbolName as core[bcfc2130cb4e43e7]::fmt::Display>::fmt
  43:     0x7ff9b0a3372d - rustc_driver_impl[39e757d53880b308]::args::arg_expand_all
  44:     0x7ff9b0a289d0 - rustc_driver_impl[39e757d53880b308]::main
  45:     0x7ff9b0a34d43 - rustc_driver_impl[39e757d53880b308]::args::arg_expand_all
  46:     0x7ff9c59ccaac - std::sys::windows::thread::Thread::new::h911abb68de5d2aa0
  47:     0x7ffa93ee7614 - BaseThreadInitThunk
  48:     0x7ffa952626b1 - RtlUserThreadStart

error: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: rustc 1.70.0 (90c541806 2023-05-31) running on x86_64-pc-windows-msvc

note: compiler flags: --crate-type bin -C embed-bitcode=no -C debuginfo=2 -C incremental=[REDACTED]

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
#0 [mir_built] building MIR for `components::annotated_number::<impl at src\components\annotated_number.rs:48:28: 48:38>::children`
#1 [mir_const] preparing `components::annotated_number::<impl at src\components\annotated_number.rs:48:28: 48:38>::children` for borrow checking
#2 [mir_promoted] processing MIR for `components::annotated_number::<impl at src\components\annotated_number.rs:48:28: 48:38>::children`
#3 [mir_borrowck] borrow-checking `components::annotated_number::<impl at src\components\annotated_number.rs:48:28: 48:38>::children`
#4 [analysis] running analysis passes on this crate
end of query stack

[Oliboy50]

I frequently hit this issue (using Rust 1.70.0), I wonder if this can be because of the use of the procedural macros from mockall crate (because we see Mock... traits in the errors message), and it started to happen when we moved our code to a shared lib within our cargo workspace

note: rustc 1.70.0 (90c541806 2023-05-31) running on x86_64-apple-darwin

note: compiler flags: --crate-type lib -C embed-bitcode=no -C split-debuginfo=unpacked -C debuginfo=2 -C incremental=[REDACTED]

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
#0 [mir_built] building MIR for `domain::port::event_pusher::__mock_MockEventPusher_EventPusher::__push::<impl at shared/src/domain/port/event_pusher.rs:14:1: 14:21>::with`
#1 [mir_const] preparing `domain::port::event_pusher::__mock_MockEventPusher_EventPusher::__push::<impl at shared/src/domain/port/event_pusher.rs:14:1: 14:21>::with` for borrow checking
#2 [mir_promoted] processing MIR for `domain::port::event_pusher::__mock_MockEventPusher_EventPusher::__push::<impl at shared/src/domain/port/event_pusher.rs:14:1: 14:21>::with`
#3 [mir_borrowck] borrow-checking `domain::port::event_pusher::__mock_MockEventPusher_EventPusher::__push::<impl at shared/src/domain/port/event_pusher.rs:14:1: 14:21>::with`
#4 [analysis] running analysis passes on this crate
end of query stack
error: internal compiler error: re-entrant incremental verify failure, suppressing message

error: could not compile `shared` (lib) due to 2 previous errors

[hrxi]

I'm hitting this with rustc 1.70.0 on Linux using cargo check.

error: internal compiler error: encountered incremental compilation error with mir_built(2b0ee002860f4e6d-86dbf79d6866cd33)
  |
  = help: This is a known issue with the compiler. Run `cargo clean -p nimiq_lib` or `cargo clean` to allow your project to compile
  = note: Please follow the instructions below to create a bug report with the provided information
  = note: See <https://github.com/rust-lang/rust/issues/84970> for more information

thread 'rustc' panicked at 'Found unstable fingerprints for mir_built(2b0ee002860f4e6d-86dbf79d6866cd33): Steal { value: RwLock(RefCell { value: Some(Body { basic_blocks: BasicBlocks { basic_blocks: [BasicBlockData { statements: [StorageLive(_3), StorageLive(_4), _4 = move _1, FakeRead(ForLet(None), _4), StorageLive(_5), StorageLive(_6), StorageLive(_7), StorageLive(_8), _8 = move _2], terminator: Some(Terminator { source_info: SourceInfo { span: lib/src/config/config.rs:523:17: 523:24 (#750), scope: scope[1] }, kind: _7 = <VALUE as Into<config::config::Credentials>>::into(move _8) -> [return: bb1, unwind: bb10] }), is_cleanup: false }, BasicBlockData { statements: [StorageDead(_8), _6 = std::option::Option::<config::config::Credentials>::Some(move _7)], terminator: Some(Terminator { source_info: SourceInfo { span: lib/src/config/config.rs:523:23: 523:24 (#750), scope: scope[1] }, kind: drop(_7) -> [return: bb2, unwind: bb9] }), is_cleanup: false }, BasicBlockData { statements: [StorageDead(_7), _5 = std::option::Option::<std::option::Option<config::config::Credentials>>::Some(move _6)], terminator: Some(Terminator { source_info: SourceInfo { span: lib/src/config/config.rs:523:23: 523:24 (#750), scope: scope[1] }, kind: drop(_6) -> [return: bb3, unwind: bb8] }), is_cleanup: false }, BasicBlockData { statements: [StorageDead(_6)], terminator: Some(Terminator { source_info: SourceInfo { span: lib/src/config/config.rs:523:17: 523:24 (#3465), scope: scope[1] }, kind: drop(((*_4).5: std::option::Option<std::option::Option<config::config::Credentials>>)) -> [return: bb4, unwind: bb5] }), is_cleanup: false }, BasicBlockData { statements: [((*_4).5: std::option::Option<std::option::Option<config::config::Credentials>>) = move _5], terminator: Some(Terminator { source_info: SourceInfo { span: lib/src/config/config.rs:523:23: 523:24 (#750), scope: scope[1] }, kind: drop(_5) -> [return: bb6, unwind: bb11] }), is_cleanup: false }, BasicBlockData { statements: [((*_4).5: std::option::Option<std::option::Option<config::config::Credentials>>) = move _5], terminator: Some(Terminator { source_info: SourceInfo { span: lib/src/config/config.rs:523:17: 523:24 (#3465), scope: scope[1] }, kind: goto -> bb8 }), is_cleanup: true }, BasicBlockData { statements: [StorageDead(_5), _3 = &mut (*_4), StorageDead(_4), _0 = &mut (*_3), StorageDead(_3)], terminator: Some(Terminator { source_info: SourceInfo { span: lib/src/config/config.rs:523:23: 523:24 (#750), scope: scope[0] }, kind: drop(_2) -> [return: bb7, unwind: bb12] }), is_cleanup: false }, BasicBlockData { statements: [], terminator: Some(Terminator { source_info: SourceInfo { span: lib/src/config/config.rs:523:24: 523:24 (#750), scope: scope[0] }, kind: return }), is_cleanup: false }, BasicBlockData { statements: [], terminator: Some(Terminator { source_info: SourceInfo { span: lib/src/config/config.rs:523:23: 523:24 (#750), scope: scope[1] }, kind: drop(_5) -> [return: bb11, unwind terminate] }), is_cleanup: true }, BasicBlockData { statements: [], terminator: Some(Terminator { source_info: SourceInfo { span: lib/src/config/config.rs:523:23: 523:24 (#750), scope: scope[1] }, kind: drop(_6) -> [return: bb11, unwind terminate] }), is_cleanup: true }, BasicBlockData { statements: [], terminator: Some(Terminator { source_info: SourceInfo { span: lib/src/config/config.rs:523:23: 523:24 (#750), scope: scope[1] }, kind: drop(_8) -> [return: bb11, unwind terminate] }), is_cleanup: true }, BasicBlockData { statements: [], terminator: Some(Terminator { source_info: SourceInfo { span: lib/src/config/config.rs:523:23: 523:24 (#750), scope: scope[0] }, kind: drop(_2) -> [return: bb12, unwind terminate] }), is_cleanup: true }, BasicBlockData { statements: [], terminator: Some(Terminator { source_info: SourceInfo { span: lib/src/config/config.rs:523:17: 523:24 (#750), scope: scope[0] }, kind: resume }), is_cleanup: true }], cache: Cache { predecessors: OnceCell(Uninit), switch_sources: OnceCell(Uninit), is_cyclic: OnceCell(Uninit), postorder: OnceCell(Uninit) } }, phase: Built, pass_count: 0, source: MirSource { instance: Item(WithOptConstParam { did: DefId(0:1033 ~ nimiq_lib[f8a8]::config::config::{impl#91}::credentials), const_param_did: None }), promoted: None }, source_scopes: [SourceScopeData { span: lib/src/config/config.rs:523:17: 523:24 (#750), parent_scope: None, inlined: None, inlined_parent_scope: None, local_data: Set(SourceScopeLocalData { lint_root: HirId(DefId(0:1033 ~ nimiq_lib[f8a8]::config::config::{impl#91}::credentials).0), safety: Safe }) }, SourceScopeData { span: lib/src/config/config.rs:523:17: 523:24 (#750), parent_scope: Some(scope[0]), inlined: None, inlined_parent_scope: None, local_data: Set(SourceScopeLocalData { lint_root: HirId(DefId(0:1033 ~ nimiq_lib[f8a8]::config::config::{impl#91}::credentials).0), safety: Safe }) }], generator: None, local_decls: [LocalDecl { mutability: Mut, local_info: Set(Boring), internal: false, ty: &mut config::config::RpcServerConfigBuilder, user_ty: None, source_info: SourceInfo { span: lib/src/config/config.rs:523:17: 523:24 (#750), scope: scope[0] } }, LocalDecl { mutability: Not, local_info: Set(User(ImplicitSelf(MutRef))), internal: false, ty: &mut config::config::RpcServerConfigBuilder, user_ty: None, source_info: SourceInfo { span: lib/src/config/config.rs:523:17: 523:24 (#750), scope: scope[0] } }, LocalDecl { mutability: Not, local_info: Set(User(Var(VarBindingForm { binding_mode: BindByValue(Not), opt_ty_info: None, opt_match_place: Some((None, lib/src/config/config.rs:523:17: 523:24 (#750))), pat_span: lib/src/config/config.rs:523:17: 523:24 (#750) }))), internal: false, ty: VALUE, user_ty: None, source_info: SourceInfo { span: lib/src/config/config.rs:523:17: 523:24 (#750), scope: scope[0] } }, LocalDecl { mutability: Mut, local_info: Set(Boring), internal: false, ty: &mut config::config::RpcServerConfigBuilder, user_ty: None, source_info: SourceInfo { span: lib/src/config/config.rs:523:17: 523:24 (#750), scope: scope[0] } }, LocalDecl { mutability: Mut, local_info: Set(User(Var(VarBindingForm { binding_mode: BindByValue(Mut), opt_ty_info: None, opt_match_place: Some((None, lib/src/config/config.rs:523:17: 523:24 (#750))), pat_span: lib/src/config/config.rs:523:17: 523:24 (#750) }))), internal: false, ty: &mut config::config::RpcServerConfigBuilder, user_ty: None, source_info: SourceInfo { span: lib/src/config/config.rs:523:17: 523:24 (#750), scope: scope[0] } }, LocalDecl { mutability: Mut, local_info: Set(Boring), internal: false, ty: std::option::Option<std::option::Option<config::config::Credentials>>, user_ty: None, source_info: SourceInfo { span: lib/src/config/config.rs:523:17: 523:24 (#750), scope: scope[0] } }, LocalDecl { mutability: Mut, local_info: Set(AggregateTemp), internal: false, ty: std::option::Option<config::config::Credentials>, user_ty: None, source_info: SourceInfo { span: lib/src/config/config.rs:523:17: 523:24 (#750), scope: scope[0] } }, LocalDecl { mutability: Mut, local_info: Set(AggregateTemp), internal: false, ty: config::config::Credentials, user_ty: None, source_info: SourceInfo { span: lib/src/config/config.rs:523:17: 523:24 (#750), scope: scope[0] } }, LocalDecl { mutability: Mut, local_info: Set(Boring), internal: false, ty: VALUE, user_ty: None, source_info: SourceInfo { span: lib/src/config/config.rs:523:17: 523:24 (#750), scope: scope[0] } }], user_type_annotations: [], arg_count: 2, spread_arg: None, var_debug_info: [VarDebugInfo { name: "self", source_info: SourceInfo { span: lib/src/config/config.rs:523:17: 523:24 (#750), scope: scope[0] }, value: _1, argument_index: Some(1) }, VarDebugInfo { name: "value", source_info: SourceInfo { span: lib/src/config/config.rs:523:17: 523:24 (#750), scope: scope[0] }, value: _2, argument_index: Some(2) }, VarDebugInfo { name: "new", source_info: SourceInfo { span: lib/src/config/config.rs:523:17: 523:24 (#750), scope: scope[1] }, value: _4, argument_index: None }], span: lib/src/config/config.rs:523:17: 523:24 (#750), required_consts: [], is_polymorphic: true, injection_phase: None, tainted_by_errors: None }) }) }', /rustc/90c541806f23a127002de5b4038be731ba1458ca/compiler/rustc_query_system/src/query/plumbing.rs:715:9
stack backtrace:
   0:     0x7faecbe53cca - std::backtrace_rs::backtrace::libunwind::trace::h9a6b80bbf328ba5d
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
   1:     0x7faecbe53cca - std::backtrace_rs::backtrace::trace_unsynchronized::hd162ec543a11886b
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x7faecbe53cca - std::sys_common::backtrace::_print_fmt::h78a5099be12f51a6
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:65:5
   3:     0x7faecbe53cca - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::ha1c5390454d74f71
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:44:22
   4:     0x7faecbeb80cf - core::fmt::write::h9ffde816c577717b
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/fmt/mod.rs:1254:17
   5:     0x7faecbe46805 - std::io::Write::write_fmt::h88186074961638e4
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/io/mod.rs:1698:15
   6:     0x7faecbe53a95 - std::sys_common::backtrace::_print::h184198273ed08d59
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:47:5
   7:     0x7faecbe53a95 - std::sys_common::backtrace::print::h1b4d8e7add699453
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:34:9
   8:     0x7faecbe5673e - std::panicking::default_hook::{{closure}}::h393bcea75423915a
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:269:22
   9:     0x7faecbe564e5 - std::panicking::default_hook::h48c64f31d8b3fd03
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:288:9
  10:     0x7faecf112995 - <rustc_driver_impl[fe6df70259db788d]::DEFAULT_HOOK::{closure#0}::{closure#0} as core[da82827a87f140f9]::ops::function::FnOnce<(&core[da82827a87f140f9]::panic::panic_info::PanicInfo,)>>::call_once::{shim:vtable#0}
  11:     0x7faecbe56f34 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::hb9b860f5a1175bda
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/alloc/src/boxed.rs:1987:9
  12:     0x7faecbe56f34 - std::panicking::rust_panic_with_hook::hafdc493a79370062
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:695:13
  13:     0x7faecbe56ca9 - std::panicking::begin_panic_handler::{{closure}}::h0a64bc82e36bedc7
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:582:13
  14:     0x7faecbe54136 - std::sys_common::backtrace::__rust_end_short_backtrace::hc203444fb7416a16
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:150:18
  15:     0x7faecbe56a02 - rust_begin_unwind
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:578:5
  16:     0x7faecbeb4443 - core::panicking::panic_fmt::h0f6ef0178afce4f2
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/panicking.rs:67:14
  17:     0x7faecf9e72de - rustc_query_system[5e34f8384464b8a5]::query::plumbing::incremental_verify_ich_failed::<rustc_middle[a487dff4633a86d0]::ty::context::TyCtxt>
  18:     0x7faece1c8527 - <std[b70fedfd8b77e9]::thread::local::LocalKey<core[da82827a87f140f9]::cell::Cell<*const ()>>>::with::<rustc_middle[a487dff4633a86d0]::ty::context::tls::enter_context<rustc_query_system[5e34f8384464b8a5]::query::plumbing::execute_job_incr<rustc_query_impl[3e42b93bfeb445bd]::queries::mir_built, rustc_query_impl[3e42b93bfeb445bd]::plumbing::QueryCtxt>::{closure#1}, core[da82827a87f140f9]::option::Option<(rustc_middle[a487dff4633a86d0]::query::erase::Erased<[u8; 8usize]>, rustc_query_system[5e34f8384464b8a5]::dep_graph::graph::DepNodeIndex)>>::{closure#0}, core[da82827a87f140f9]::option::Option<(rustc_middle[a487dff4633a86d0]::query::erase::Erased<[u8; 8usize]>, rustc_query_system[5e34f8384464b8a5]::dep_graph::graph::DepNodeIndex)>>
  19:     0x7faece1c74cb - rustc_query_system[5e34f8384464b8a5]::query::plumbing::try_execute_query::<rustc_query_impl[3e42b93bfeb445bd]::queries::mir_built, rustc_query_impl[3e42b93bfeb445bd]::plumbing::QueryCtxt>
  20:     0x7faeceb11f55 - <rustc_query_impl[3e42b93bfeb445bd]::Queries as rustc_middle[a487dff4633a86d0]::ty::query::QueryEngine>::mir_built
  21:     0x7faecdd856c7 - rustc_mir_transform[9506650463ce67a9]::mir_const
  22:     0x7faecfa62c61 - <std[b70fedfd8b77e9]::thread::local::LocalKey<core[da82827a87f140f9]::cell::Cell<*const ()>>>::with::<rustc_middle[a487dff4633a86d0]::ty::context::tls::enter_context<rustc_query_system[5e34f8384464b8a5]::query::plumbing::try_load_from_disk_and_cache_in_memory<rustc_query_impl[3e42b93bfeb445bd]::queries::mir_const, rustc_query_impl[3e42b93bfeb445bd]::plumbing::QueryCtxt>::{closure#1}, rustc_middle[a487dff4633a86d0]::query::erase::Erased<[u8; 8usize]>>::{closure#0}, rustc_middle[a487dff4633a86d0]::query::erase::Erased<[u8; 8usize]>>
  23:     0x7faecfa18141 - <rustc_middle[a487dff4633a86d0]::dep_graph::dep_node::DepKind as rustc_query_system[5e34f8384464b8a5]::dep_graph::DepKind>::with_deps::<rustc_query_system[5e34f8384464b8a5]::query::plumbing::try_load_from_disk_and_cache_in_memory<rustc_query_impl[3e42b93bfeb445bd]::queries::mir_const, rustc_query_impl[3e42b93bfeb445bd]::plumbing::QueryCtxt>::{closure#1}, rustc_middle[a487dff4633a86d0]::query::erase::Erased<[u8; 8usize]>>
  24:     0x7faece24af30 - <std[b70fedfd8b77e9]::thread::local::LocalKey<core[da82827a87f140f9]::cell::Cell<*const ()>>>::with::<rustc_middle[a487dff4633a86d0]::ty::context::tls::enter_context<rustc_query_system[5e34f8384464b8a5]::query::plumbing::execute_job_incr<rustc_query_impl[3e42b93bfeb445bd]::queries::mir_const, rustc_query_impl[3e42b93bfeb445bd]::plumbing::QueryCtxt>::{closure#1}, core[da82827a87f140f9]::option::Option<(rustc_middle[a487dff4633a86d0]::query::erase::Erased<[u8; 8usize]>, rustc_query_system[5e34f8384464b8a5]::dep_graph::graph::DepNodeIndex)>>::{closure#0}, core[da82827a87f140f9]::option::Option<(rustc_middle[a487dff4633a86d0]::query::erase::Erased<[u8; 8usize]>, rustc_query_system[5e34f8384464b8a5]::dep_graph::graph::DepNodeIndex)>>
  25:     0x7faece24a48b - rustc_query_system[5e34f8384464b8a5]::query::plumbing::try_execute_query::<rustc_query_impl[3e42b93bfeb445bd]::queries::mir_const, rustc_query_impl[3e42b93bfeb445bd]::plumbing::QueryCtxt>
  26:     0x7faecd7f990f - rustc_mir_transform[9506650463ce67a9]::mir_promoted
  27:     0x7faecfa5a8a1 - <std[b70fedfd8b77e9]::thread::local::LocalKey<core[da82827a87f140f9]::cell::Cell<*const ()>>>::with::<rustc_middle[a487dff4633a86d0]::ty::context::tls::enter_context<rustc_query_system[5e34f8384464b8a5]::query::plumbing::try_load_from_disk_and_cache_in_memory<rustc_query_impl[3e42b93bfeb445bd]::queries::mir_promoted, rustc_query_impl[3e42b93bfeb445bd]::plumbing::QueryCtxt>::{closure#1}, rustc_middle[a487dff4633a86d0]::query::erase::Erased<[u8; 16usize]>>::{closure#0}, rustc_middle[a487dff4633a86d0]::query::erase::Erased<[u8; 16usize]>>
  28:     0x7faecfa10d6d - <rustc_middle[a487dff4633a86d0]::dep_graph::dep_node::DepKind as rustc_query_system[5e34f8384464b8a5]::dep_graph::DepKind>::with_deps::<rustc_query_system[5e34f8384464b8a5]::query::plumbing::try_load_from_disk_and_cache_in_memory<rustc_query_impl[3e42b93bfeb445bd]::queries::mir_promoted, rustc_query_impl[3e42b93bfeb445bd]::plumbing::QueryCtxt>::{closure#1}, rustc_middle[a487dff4633a86d0]::query::erase::Erased<[u8; 16usize]>>
  29:     0x7faecdfa3a79 - <std[b70fedfd8b77e9]::thread::local::LocalKey<core[da82827a87f140f9]::cell::Cell<*const ()>>>::with::<rustc_middle[a487dff4633a86d0]::ty::context::tls::enter_context<rustc_query_system[5e34f8384464b8a5]::query::plumbing::execute_job_incr<rustc_query_impl[3e42b93bfeb445bd]::queries::mir_promoted, rustc_query_impl[3e42b93bfeb445bd]::plumbing::QueryCtxt>::{closure#1}, core[da82827a87f140f9]::option::Option<(rustc_middle[a487dff4633a86d0]::query::erase::Erased<[u8; 16usize]>, rustc_query_system[5e34f8384464b8a5]::dep_graph::graph::DepNodeIndex)>>::{closure#0}, core[da82827a87f140f9]::option::Option<(rustc_middle[a487dff4633a86d0]::query::erase::Erased<[u8; 16usize]>, rustc_query_system[5e34f8384464b8a5]::dep_graph::graph::DepNodeIndex)>>
  30:     0x7faecdfa26ee - rustc_query_system[5e34f8384464b8a5]::query::plumbing::try_execute_query::<rustc_query_impl[3e42b93bfeb445bd]::queries::mir_promoted, rustc_query_impl[3e42b93bfeb445bd]::plumbing::QueryCtxt>
  31:     0x7faecdfa1418 - rustc_borrowck[c07eeb7c7aba6c08]::mir_borrowck
  32:     0x7faecdfa0f6e - <rustc_borrowck[c07eeb7c7aba6c08]::provide::{closure#0} as core[da82827a87f140f9]::ops::function::FnOnce<(rustc_middle[a487dff4633a86d0]::ty::context::TyCtxt, rustc_span[2f805118d893a80f]::def_id::LocalDefId)>>::call_once
  33:     0x7faecdb9b31e - <std[b70fedfd8b77e9]::thread::local::LocalKey<core[da82827a87f140f9]::cell::Cell<*const ()>>>::with::<rustc_middle[a487dff4633a86d0]::ty::context::tls::enter_context<rustc_query_system[5e34f8384464b8a5]::query::plumbing::execute_job_incr<rustc_query_impl[3e42b93bfeb445bd]::queries::mir_borrowck, rustc_query_impl[3e42b93bfeb445bd]::plumbing::QueryCtxt>::{closure#2}, (rustc_middle[a487dff4633a86d0]::query::erase::Erased<[u8; 8usize]>, rustc_query_system[5e34f8384464b8a5]::dep_graph::graph::DepNodeIndex)>::{closure#0}, (rustc_middle[a487dff4633a86d0]::query::erase::Erased<[u8; 8usize]>, rustc_query_system[5e34f8384464b8a5]::dep_graph::graph::DepNodeIndex)>
  34:     0x7faecdb9a27b - rustc_query_system[5e34f8384464b8a5]::query::plumbing::try_execute_query::<rustc_query_impl[3e42b93bfeb445bd]::queries::mir_borrowck, rustc_query_impl[3e42b93bfeb445bd]::plumbing::QueryCtxt>
  35:     0x7faece5e4c10 - rustc_data_structures[fea82b52c2e47d17]::sync::par_for_each_in::<&[rustc_span[2f805118d893a80f]::def_id::LocalDefId], <rustc_middle[a487dff4633a86d0]::hir::map::Map>::par_body_owners<rustc_interface[7aa24cde61830128]::passes::analysis::{closure#2}::{closure#0}>::{closure#0}>
  36:     0x7faece5e484e - <rustc_session[99ba0ed9c533f203]::session::Session>::time::<(), rustc_interface[7aa24cde61830128]::passes::analysis::{closure#2}>
  37:     0x7faece5e40f9 - rustc_interface[7aa24cde61830128]::passes::analysis
  38:     0x7faece860633 - <std[b70fedfd8b77e9]::thread::local::LocalKey<core[da82827a87f140f9]::cell::Cell<*const ()>>>::with::<rustc_middle[a487dff4633a86d0]::ty::context::tls::enter_context<rustc_query_system[5e34f8384464b8a5]::query::plumbing::execute_job_incr<rustc_query_impl[3e42b93bfeb445bd]::queries::analysis, rustc_query_impl[3e42b93bfeb445bd]::plumbing::QueryCtxt>::{closure#2}, (rustc_middle[a487dff4633a86d0]::query::erase::Erased<[u8; 1usize]>, rustc_query_system[5e34f8384464b8a5]::dep_graph::graph::DepNodeIndex)>::{closure#0}, (rustc_middle[a487dff4633a86d0]::query::erase::Erased<[u8; 1usize]>, rustc_query_system[5e34f8384464b8a5]::dep_graph::graph::DepNodeIndex)>
  39:     0x7faece86010b - rustc_query_system[5e34f8384464b8a5]::query::plumbing::try_execute_query::<rustc_query_impl[3e42b93bfeb445bd]::queries::analysis, rustc_query_impl[3e42b93bfeb445bd]::plumbing::QueryCtxt>
  40:     0x7faece85fcd0 - <rustc_query_impl[3e42b93bfeb445bd]::Queries as rustc_middle[a487dff4633a86d0]::ty::query::QueryEngine>::analysis
  41:     0x7faece796726 - <std[b70fedfd8b77e9]::thread::local::LocalKey<core[da82827a87f140f9]::cell::Cell<*const ()>>>::with::<rustc_middle[a487dff4633a86d0]::ty::context::tls::enter_context<<rustc_middle[a487dff4633a86d0]::ty::context::GlobalCtxt>::enter<rustc_driver_impl[fe6df70259db788d]::run_compiler::{closure#1}::{closure#2}::{closure#4}, core[da82827a87f140f9]::result::Result<(), rustc_span[2f805118d893a80f]::ErrorGuaranteed>>::{closure#0}, core[da82827a87f140f9]::result::Result<(), rustc_span[2f805118d893a80f]::ErrorGuaranteed>>::{closure#0}, core[da82827a87f140f9]::result::Result<(), rustc_span[2f805118d893a80f]::ErrorGuaranteed>>
  42:     0x7faece3fd02e - <rustc_interface[7aa24cde61830128]::interface::Compiler>::enter::<rustc_driver_impl[fe6df70259db788d]::run_compiler::{closure#1}::{closure#2}, core[da82827a87f140f9]::result::Result<core[da82827a87f140f9]::option::Option<rustc_interface[7aa24cde61830128]::queries::Linker>, rustc_span[2f805118d893a80f]::ErrorGuaranteed>>
  43:     0x7faece3f8281 - rustc_span[2f805118d893a80f]::set_source_map::<core[da82827a87f140f9]::result::Result<(), rustc_span[2f805118d893a80f]::ErrorGuaranteed>, rustc_interface[7aa24cde61830128]::interface::run_compiler<core[da82827a87f140f9]::result::Result<(), rustc_span[2f805118d893a80f]::ErrorGuaranteed>, rustc_driver_impl[fe6df70259db788d]::run_compiler::{closure#1}>::{closure#0}::{closure#0}>
  44:     0x7faece3f782f - std[b70fedfd8b77e9]::sys_common::backtrace::__rust_begin_short_backtrace::<rustc_interface[7aa24cde61830128]::util::run_in_thread_pool_with_globals<rustc_interface[7aa24cde61830128]::interface::run_compiler<core[da82827a87f140f9]::result::Result<(), rustc_span[2f805118d893a80f]::ErrorGuaranteed>, rustc_driver_impl[fe6df70259db788d]::run_compiler::{closure#1}>::{closure#0}, core[da82827a87f140f9]::result::Result<(), rustc_span[2f805118d893a80f]::ErrorGuaranteed>>::{closure#0}::{closure#0}, core[da82827a87f140f9]::result::Result<(), rustc_span[2f805118d893a80f]::ErrorGuaranteed>>
  45:     0x7faece9db661 - <<std[b70fedfd8b77e9]::thread::Builder>::spawn_unchecked_<rustc_interface[7aa24cde61830128]::util::run_in_thread_pool_with_globals<rustc_interface[7aa24cde61830128]::interface::run_compiler<core[da82827a87f140f9]::result::Result<(), rustc_span[2f805118d893a80f]::ErrorGuaranteed>, rustc_driver_impl[fe6df70259db788d]::run_compiler::{closure#1}>::{closure#0}, core[da82827a87f140f9]::result::Result<(), rustc_span[2f805118d893a80f]::ErrorGuaranteed>>::{closure#0}::{closure#0}, core[da82827a87f140f9]::result::Result<(), rustc_span[2f805118d893a80f]::ErrorGuaranteed>>::{closure#1} as core[da82827a87f140f9]::ops::function::FnOnce<()>>::call_once::{shim:vtable#0}
  46:     0x7faecbe613b5 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::ha1f2224656a778fb
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/alloc/src/boxed.rs:1973:9
  47:     0x7faecbe613b5 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::haa29ed9703f354b7
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/alloc/src/boxed.rs:1973:9
  48:     0x7faecbe613b5 - std::sys::unix::thread::Thread::new::thread_start::h33b6dae3e3692197
                               at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys/unix/thread.rs:108:17
  49:     0x7faecbb509eb - <unknown>
  50:     0x7faecbbd4dfc - <unknown>
  51:                0x0 - <unknown>

error: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: rustc 1.70.0 (90c541806 2023-05-31) running on x86_64-unknown-linux-gnu

note: compiler flags: --crate-type lib -C embed-bitcode=no -C debuginfo=2 -C incremental=[REDACTED]

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
#0 [mir_built] building MIR for `config::config::<impl at lib/src/config/config.rs:523:17: 523:24>::credentials`
#1 [mir_const] preparing `config::config::<impl at lib/src/config/config.rs:523:17: 523:24>::credentials` for borrow checking
#2 [mir_promoted] processing MIR for `config::config::<impl at lib/src/config/config.rs:523:17: 523:24>::credentials`
#3 [mir_borrowck] borrow-checking `config::config::<impl at lib/src/config/config.rs:523:17: 523:24>::credentials`
#4 [analysis] running analysis passes on this crate
end of query stack
error: could not compile `nimiq-lib` (lib) due to previous error

[cnoam]

Another example: trying to compile the code in section 7.2.5.0.2 in zero2production (https://www.zero2prod.com/).

$ rustc --version
rustc 1.72.0 (5680fa1 2023-08-23)
$ cargo --version
cargo 1.72.0 (103a7ff2e 2023-08-15)
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.3 LTS
Release: 22.04
Codename: jammy

I kept the offending source code in cnoam/zero2production@2a117ea

HTH

[tengyifei]

Another example: #122116