proto: discuss serialisation of user.rootlesscontainers

[cyphar]

For some reason, Go has decided that all xattrs must be valid UTF-8 strings (when embedded in tar files). This is obviously incorrect for Linux xattrs, but I imagine that this restriction is based on the restrictions of SCHILY.xattr.*. In the case of umoci this makes implementing user.rootlesscontainers support harder than necessary.

However, if we defined that the protobuf must be serialised as base64 (or as a protobuf JSON) then this issue would be resolved. The downside is it might break other users.

/cc @AkihiroSuda -- wdyt?

[AkihiroSuda]

Thanks for notifying this, do you know any pointer for the Go maintainers decision on this?
This decision seems really wrong.. 😢

For some reason, Go has decided that all xattrs must be valid UTF-8 strings (when embedded in tar files).

s/must/should/?

https://golang.org/pkg/archive/tar/#Header: The key and value should be non-empty UTF-8 strings.
https://blog.golang.org/strings: It's important to state right up front that a string holds arbitrary bytes. It is not required to hold Unicode text, UTF-8 text, or any other predefined format.

So, IIUC, tar.Header structure definition itself can accept non-UTF8 byte slice.
Not sure whether the actual tar pkg implementation accepts it though.

when embedded in tar files

Is there any case where we need to put user.rootlesscontainers xattr to tar?

IIUC Differ (in containerd lingo) should convert the xattr to the tarhdr, and Applier should convert the tarhdr to the xattr. So the xattr does not need to be stored in the tar.

The downside is it might break other users.

Is there any existing implementation other than my forked version of PRoot?

[cyphar]

https://golang.org/pkg/archive/tar/#Header: The key and value should be non-empty UTF-8 strings.
https://blog.golang.org/strings: It's important to state right up front that a string holds arbitrary bytes. It is not required to hold Unicode text, UTF-8 text, or any other predefined format.

Huh, I guess I was mistaken about Go strings (though I have read that blog post before, I must've misread it). It was my impression that string(some_byte_slice) could break things if it isn't UTF-8. But that isn't true I guess (I also completely forgot that string iteration is over byte and not rune -- which added to the confusion).

However, I now have a different problem -- for some reason my wip user.rootlesscontainers code isn't pumping out valid xattr data (I thought the issue was my string casts, but since it cannot be the problem must lie somewhere else). I will keep this open until I figure out why user.rootlesscontainers isn't working with my umoci branch. If it turns out that it's just me needing some more coffee, I'll close this one.

Is there any existing implementation other than my forked version of PRoot?

You are the "other user" I was talking about. 😸

[cyphar]

Dammit. It turns out it does work, it's just that the tool I was using to display the xattr value (https://github.com/xattr/xattr) doesn't display the value correctly. If I use getfattr it turns out umoci is setting the values correctly.

Closing. 🤦‍♂️