Mutable references vs self-referential structs
RalfJung opened this issue · 31 comments
Turns out that stacked borrows and self-referential generators don't go well together. This code fails in Miri:
#![feature(generators, generator_trait)]
use std::{
ops::{Generator, GeneratorState},
pin::Pin,
};
fn firstn() -> impl Generator<Yield = u64, Return = ()> {
static move || {
let mut num = 0;
let num = &mut num;
yield *num;
*num += 1; //~ ERROR: borrow stack
yield *num;
*num += 1;
yield *num;
*num += 1;
}
}
fn main() {
let mut generator_iterator = firstn();
let mut pin = unsafe { Pin::new_unchecked(&mut generator_iterator) };
let mut sum = 0;
while let GeneratorState::Yielded(x) = pin.as_mut().resume() {
sum += x;
}
println!("{}", sum);
}The reason it fails is that each time through the loop, we call Pin::as_mut, which creates a fresh mutable reference to the generator. Since mutable references are unique, that invalidates the pointer that points from one field of the generator to another.
This is basically a particularly bad instance of #133.
In some sense there are two problems here:
- When a
Pin<&mut T>gets passed around, it gets retagged the same way as an&mut T, assuring uniqueness of this pointer to the entire memory range. This is probably not something we want. So maybe we need a "no retagging for private fields" kind of thing, similar to what we might need for protectors. This is the (relatively) easy part. - However, even just calling
Pin::as_mutcreates a mutable reference internally (before wrapping it inPinagain). So even if we fix the above, we'd still assert uniqueness here. In some sense it would be more correct if aPin<&mut T>would actually be represented as aNonNull<T>, as that's a more honest reflection of the aliasing situation: it's not a unique pointer, there can be pointers inside this data structure that alias. This is the hard part.
If we still had PinMut as a separate type, this would be fixable, but with Pin<impl Deref>, that's hard. And even then, map_unchecked_mut still creates a mutable reference and even passes it as a function argument, which is about as strongly asserting uniqueness as we can -- and which we don't want, aliasing pointers are allowed to exist.
If we ignore the fact that references are wrapped inside the Pin struct, self-referential generators do the equivalent of:
fn main() {
let mut local = 42;
let raw_ptr = &mut local as *mut i32; // create raw pointer
let safe_ref = &mut local; // create a reference, which is unique, and hence invalidates the raw pointer
println!("{}", unsafe { *raw_ptr }); // UB
}I explicitly wanted that code to be UB. While raw pointers aliasing with each other is allowed, having a single raw pointer and a mutable reference acts pretty much like having two mutable references -- after all, between the two of them, it's enough if one side does not allow aliasing, since "aliases-with" is a symmetric relation. If we replace raw_ptr by another reference above, the code is rejected by both the borrow checker and Stacked Borrows.
So to fix this problem, I see two general options:
- "Hide" references in struct fields from Stacked Borrows. Then the implementation of
Pinmethods would have to do lots oftransmuteto avoid ever creating a "bare" reference. Andmap_unchecked_muthas a problem. It should probably use raw pointers (and rust-lang/rfcs#2582 to get a field address). On the other hand, at least we have a plan. - Allow the example code above. We would have to somehow "lazily" activate mutable references. It's a bit like two-phase borrows but worse. I don't have a very concrete idea what this would look like, but I think Stacked Borrows would have to become Tree-shaped Borrows or so -- a stack just does not have enough structure.
@RalfJung: Could this be resolved by turning Pin<&'a mut T> into some abstraction around (&'a UnsafeCell<T>; PhantomData<&'a mut T>) that does not actually hold a reference, but only behaves like one and spawns actual references on demand?
This would likely require turning Pin into Compiler Magic(tm) that is hard to implement for user-defined pointer types, though, and I can imagine backwards-incompatible implications in other areas such as changing the safety contract for extracting mutable references from Pin...
EDIT: Ah, I see that you explored a NonNull-based variant of this strategy above.
You don't need UnsafeCell, *mut T would do it. And yes that's basically what I had in mind with option (1).
The way I understand how Pin works, the self referential references borrow their validity from the one in the pin. So under SB, they're retagged from the pin.
The most minimal change that would make pin play nicely with SB I think would be to somehow keep the interior references valid even while the unsafe map_unchecked_mut reference is used.
Could it be possible to not retag for pin's map_unchecked_mut and only pop tags when the reference is used to access that memory location? (This is super spitball, sorry)
The way I understand how Pin works, the self referential references borrow their validity from the one in the pin. So under SB, they're retagged from the pin.
Correct.
However, then the pin itself eventually gets retagged, and that kills the self-referential borrows. This currently happens all the time, but could be reduced to just "inside the Pin implementation" if we make retag respect privacy, or just generally not enter structs, or so.
Could it be possible to not retag for pin's map_unchecked_mut
Well it'd need a magic marker or so.
and only pop tags when the reference is used to access that memory location? (This is super spitball, sorry)
That's basically option (2) from above.
#194 indicates that this is a problem with self-referential structs in general, not just self-referential generators. That's not surprising, so I generalized the issue title.
@RalfJung: Related to your example of:
fn main() {
let mut local = 42;
let raw_ptr = &mut local as *mut i32; // create raw pointer
let safe_ref = &mut local; // create a reference, which is unique, and hence invalidates the raw pointer
println!("{}", unsafe { *raw_ptr }); // UB
}
While working on pin-project, I wanted to write this code:
use std::pin::Pin;
struct Foo {
// #[pin]
field1: u8,
field2: bool
}
struct FooProj<'a> {
__self_ptr: *mut Foo,
field1: Pin<&'a mut u8>,
field2: &'a mut bool
}
impl Foo {
fn project<'a>(self: Pin<&'a mut Self>) -> FooProj<'a> {
let this = unsafe { self.get_unchecked_mut() };
let __self_ptr: *mut Foo = this;
let field1 = unsafe { Pin::new_unchecked(&mut this.field1) };
let field2 = &mut this.field2;
FooProj {
__self_ptr,
field1,
field2
}
}
}
impl<'a> FooProj<'a> {
fn unproject(self) -> Pin<&'a mut Foo> {
unsafe {
let this: &mut Foo = &mut *self.__self_ptr;
Pin::new_unchecked(this)
}
}
}
fn main() {
let mut foo = Foo { field1: 25, field2: true };
let foo_pin: Pin<&mut Foo> = Pin::new(&mut foo);
let foo_proj: FooProj = foo_pin.project();
let foo_orig: Pin<&mut Foo> = foo_proj.unproject();
}The key thing here is the unproject method. Basically, I want to be able to the following:
- Convert an
&mut Selfto a*mut Self - Create mutable references to fields of
Self(&mut self.field, etc.) - Later, upgrade the
*mut Selfto a&mut Self, after all of the mutable fields references have gone out of scope.
However, Miri flags this as UB for (I believe) a similar reason as your example - creating a mutable reference to a field ends up transitively asserts that we have unique ownership of the base type. Therefore, any pre-existing raw pointers to the base type will be invalidated.
Allowing this kind of pattern would make pin-project much more useful. It would allow the following pattern:
impl MyType {
fn process(self: Pin<&mut Self>) {
let this = self.project();
// Use a pin-projected field - e.g. poll a wrapped future
...
// Construct a new instance of MyType - e.g. a new enum variant
let new_foo: MyType = ...;
// Overwrite ourself using Pin::set
let new_self: Pin<&mut Self> = this.unproject();
new_self.set(new_foo);
}
This is especially useful when working with enums - here's a real-world example in Hyper.
However, I can't see a way to avoid creating and later 'upgrading' a raw pointer when trying to safely abstract this pattern in pin-project.
With arbitrary_self_types, pin-project could take self: &mut Pin<&mut Self> to get a lifetime to tie the projection to, and let NLL take care of the projection lifetime then allow using self again. example.
It is the same kind of reasoning as to why this could be sound: in the self-referential case, it's the whole pointer existing that invalidates the part pointers, though their usage regions are disjoint. In the unproject case, it's the part pointers invalidating the whole pointer, even though they are statically required to be used in disjoint regions.
creating a mutable reference to a field ends up transitively asserts that we have unique ownership of the base type
I don't understand what you are trying to say here. The type of the referent doesn't really matter here. (It only matters for its size and where the UnsafeCell are.)
Therefore, any pre-existing raw pointers to the base type will be invalidated.
Only raw pointers that are not "parents" of the mutable reference get invalidated.
So, I think you can fix your code by deriving FooProj::field* from FooProj::__self_ptr.
As a passing thought:
Would the situation improve if there were a core lang type (name TBD) here called UnsafeWindow<T> with the fundamental property, an inverse of UnsafeCell<T>, that a &mut UnsafeWindow<T> is permitted simultaneously with &T to its field?
The property we need of some UnsafeAlias type is that two &mut UnsafeAlias<T> may alias. I don't see an immediate reason for why this would have to interact with & in interesting ways.
If two aliasing &mut UnsafeAlias<T> are truly required then UnsafeWindow<T> would indeed not help.
However, my understanding is that only aliasing &mut x and &mut x.subfield where the second is not derived from the first is required.
For that consider UnsafeWindow<UnsafeCell<T>>:
with a &[mut] UnsafeWindow<UnsafeCell<T>> one can create a &UnsafeCell<T> and then later a &mut T, simultaneously with the existence of (an address equivalent, but provenance distinct) &[mut] UnsafeWindow<UnsafeCell<T>>.
Or it could be I'm just imagining the consequences of this wrong. 🤷♂️
The property we need of some
UnsafeAliastype is that two&mut UnsafeAlias<T>may alias. I don't see an immediate reason for why this would have to interact with & in interesting ways.
Isn't it rather that &mut UnsafeAlias<T>/&UnsafeAlias<T> can alias with &mut T/&T? Shared references get involved in cases like
async {
let x = 5;
let (y, z) = (&x, &x);
yield_now().await;
dbg!((y, z));
}Assuming all fields in generators are wrapped in UnsafeAlias, then during the second poll you have one implied active &mut UnsafeAlias<i32> from the fields of the future, and two &i32 self-references.
However, my understanding is that only aliasing &mut x and &mut x.subfield where the second is not derived from the first is required.
This means we have two aliasing &mut x.subfield, effectively. I do not see the difference.
Isn't it rather that &mut UnsafeAlias/&UnsafeAlias can alias with &mut T?
That cannot work, &mut T has no way of knowing that there exists an &mut UnsafeAlias<T>.
This is similar for UnsafeCell: when an &T and an &UnsafeCell<T> alias, performing mutation through the &UnsafeCell<T> is UB! All aliases must use UnsafeCell to effectively opt-out of immutability of shared references. Likewise, all aliases must use UnsafeAlias to opt-out of uniqueness of mutable references.
But yes, the &mut UnsafeAlias<T> may also alias with &T -- however when it does, performing mutation is UB.
Likewise, all aliases must use
UnsafeAliasto opt-out of uniqueness of mutable references.
I don't see how the generator transform could do this.
fn bar(z: &mut i32) {
*z = 5;
}
async fn foo(x: &mut i32) {
yield_once().await;
bar(x);
}
async {
let mut y = 6;
foo(&mut y).await;
}On the second poll of the future there exists an implicit &mut of the field containing y, simultaneously on the stack frame of bar there exists an &mut i32 which is being written to, non-temporally derived from y. The generator transform can change the type of the field containing y to be something like UnsafeAlias<i32>, so that the implicit &mut of it is an &mut UnsafeAlias<i32>, but it cannot change the type of bar so that it takes an &mut UnsafeAlias<i32>.
I don't see how the generator transform could do this.
Hm, good point. But there's also another difference between this and what I imagined: the &mut UnsafeAlias<i32> is not, and must not, be used, right?
So maybe things should work like this: when an &T and an &UnsafeCell<T> alias, performing mutation through the &UnsafeCell is UB, but both of these references existing is fine, and even performing a read through either of them is fine. So likewise, when an &mut T and an &mut UnsafeAlias<T> alias, this in itself is fine, but performing any kind of access via the UnsafeAlias is UB (due to violating uniqueness of the &mut T). That should work for generators, right? Basically, &mut UnsafeAlias<T> opts out of "eagerly" asserting anything about the location, and doing any kind of access is treated similar to a raw pointer access.
rust-lang/miri#1952 could help with this by disabling uniqueness of mutable references for !Unpin types -- however, there is some risk that too many types are !Unpin, thus disabling uniqueness in too many cases.
I wonder what people think about this.
I think it would be very interesting to have a version of miri that disables uniqueness for !Unpin types. We have tried to make the Tokio types always be !Unpin when they are intrusively shared, but it would definitely not surprise me if it's not done perfectly. Much of Tokio's codebase is not testable with miri because of this issue.
One particular concern I have is regarding what happens when you have an Unpin field of a !Unpin type.
One particular concern I have is regarding what happens when you have an Unpin field of a !Unpin type.
The moment you create a reference to that field, it would require uniqueness.
have a version of miri that disables uniqueness for !Unpin types
As in, make this configurable via a flag? That's possible, but then the question remains what the default should be.
Whether it's a flag or not is not that important to me.
The moment you create a reference to that field, it would require uniqueness.
Yes, that was the impression I got as well. In Tokio I have been doing this because it's unclear to me exactly what is required:
However, we haven't been doing the same for all structs containing a Pointers. Tokio's MSRV is too old for the addr_of! macro, so it's pretty difficult to do it everywhere. It's also difficult if types are not Copy.
Hm, I don't quite understand what that Pointers struct is about, but indeed without addr_of! some code cannot be written properly. Would it be an option to have a build.rs set a cfg flag such that on recent enough Rust, addr_of! is used, but older Rust fall back to &expr as *const _? (Somewhat like the maybe-uninit crate.)
But anyway that seems unrelated to self-referential generators, so we should probably move the discussion elsewhere (e.g. Zulip).
I believe I'm running into a variation of this here: mitsuhiko/deser#37 — While I don't have a generator, I have the situation where I am trying to stash away a value temporarily on an iterator like type for one iteration and the returning the value invalidates all pointers.
While what I'm doing is probably not a very common issue, it seems like you can run into this in the absence of generators, futures and pin as well.
Yes, it's definitely a thing that happens outside of the case that's mentioned at the start.
This issue here is specific about self-referential data structures though, not sure how those would come up in an iterator? So on first sight that does not really sound like the same problem.
@RalfJung it's not a traditional iterator, it's a struct that is iterated like this:
while let Some(value) = thing.next()? { ... }And it works by stashing data onto self for next() to return a borrow to. Since I create self referential data within the "iterator" to make data survive until the next iteration in the form of some state machine, I am getting some pointers invalidated presumably because fn next has &mut self. That's at least my interpretation of what is happening.
Cross-linking recent zulip discussion about "UnsafeMutCell": https://rust-lang.zulipchat.com/#narrow/stream/213817-t-lang/topic/aliased.20mutability
Visiting for backlog bonanza. Retitling to clarify that this is not SB specific
As an observer, a thought occurred regarding the UnsafeWindow proposal. Would it be possible to prevent this type from ever being referenced? As in, it should be treated effectively like an UnsafeCell, but creating direct references to an UnsafeWindow should be disallowed.
Based on a minute's worth of thought it seems like if UnsafeWindow were allowed to be turned into a reference, we run into the same set of problems every other proposal has, where we end up having to make exemptions for certain referencing patterns wholesale where we could instead restrict it to certain regions through api enforcement.
It feels like an approach where UnsafeWindow would instead somehow provide non aliasing mutable references to the inner value may allow for easier restrictions on what is allowed in general while still allowing the self referential behaviour.
Thinking in terms of mutability, Id say UnsafeWindow would behave like UnsafeCell, as in an immutable UnsafeWindow would still allow for multiple mutable references to the inner value to exist.
In summary, UnsafeWindow would be ownership only, and dereferencing an unsafe window should result in, say, a new reference that is implicitly assumed to be non aliasing. Usage in ways that allow the reference to exist outside of the variable's scope would be UB, but I can' figure out if there is a way to statically disallow such a situation, aince if we attach ifetimes to the created references it would imply they are tied to the UnsafeWindow and we may end up looping around to the aliasing mutable reference problem again.
Welp, that RFC sort of is exactly what I envisioned.