Helper library of useful de/serialize_with functions
Closed this issue · 32 comments
For example the ones from #550 and serde-rs/serde-rs.github.io#22. These don't need to be in serde itself but if there are ones that people ask for over and over, we can stick those in a helper crate and provide a better out-of-the-box experience compared to "implement your own serialize_with" or "paste this code into your project."
Add a way for deserializing numbers from strings containing numbers (which is a feature that serde 0.9 had implicitly)
I suggest providing helper functions that would work for anything implementing TryFrom or FromStr traits.
Deserializing map and set types while treating duplicate entries as error, and treating duplicate entries as first-one-wins (rather than the current default of last-one-wins). #916
Deserializing map and set types while treating duplicate entries as error, and treating duplicate entries as first-one-wins (rather than the current default of last-one-wins). #916
How can you treat duplicate entries as both errors and first-one-wins? 🤔
Two functions.
It'd be nice if we could have the same level of configuration available for structs, where instead of erroring on duplicates, we could customise them as first-one-wins or last-one-wins.
Perhaps it might be nice to offer merging them together as well if they're of a map-like or list-like type.
Have you ever wanted that in a project? Can you explain the use case in more detail?
Serialize using Display and deserialize using FromStr, as in serde-rs/json#329.
I'm not sure if it would make sense to use Display for serialization, since Display is "for user-facing output" and so may drop data or not be interchangeable with FromStr.
std::string::ToString might be more appropriate for serialization, but I don't know if this is how it's intended to be used either.
#994 -- represent a field as nested JSON string.
{
"sms":"{\"Source\":4477665544,\"Destination\":1231231}",
"uuid":"69e123f4-4ced-4f8f-9853-df20ebc3937b"
}#1042 -- a helper for the "double option" pattern for distinguishing null vs absent value.
Stringification of a field (type is String, deserializes from numbers and strings, or even more?).
Example implementation: https://play.rust-lang.org/?gist=f0e00482fa5bb52c7c9e3db213d271db&version=stable
Note that there now is https://github.com/vityafx/serde-aux which has a few of these functions.
Oh, I was going to start a library of my own along these lines, but it seems someone beat me to it. Maybe we can build on that...
Yeah, I was thinking of doing one myself if I got around to it, but I'd rather see there be one library. It's a pain to remember too many library names (and versions if you aren't using cargo-edit)
@spease Probably it will officially come under serde control/ownership once it matures, but we could just submit PRs to this serde-aux project for now. (serde-helpers or serde-util is a better name in my mind.)
Here's another misc function. The use case for this is to get enum variant names where I needed to do #[serde(rename_all)] to match an API's case, and serialize them with a pipe (|) in between them.
use serde::ser::{self, Serialize, Serializer, SerializeStructVariant, SerializeTupleVariant, Impossible};
use std;
// Many thanks to dtolnay
pub fn variant_name<T: Serialize>(t: &T) -> &'static str {
#[derive(Debug)]
struct NotEnum;
type Result<T> = std::result::Result<T, NotEnum>;
impl std::error::Error for NotEnum {
fn description(&self) -> &str { "not struct" }
}
impl std::fmt::Display for NotEnum {
fn fmt(&self, _f: &mut std::fmt::Formatter) -> std::fmt::Result { unimplemented!() }
}
impl ser::Error for NotEnum {
fn custom<T: std::fmt::Display>(_msg: T) -> Self { NotEnum }
}
struct VariantName;
impl Serializer for VariantName {
type Ok = &'static str;
type Error = NotEnum;
type SerializeSeq = Impossible<Self::Ok, Self::Error>;
type SerializeTuple = Impossible<Self::Ok, Self::Error>;
type SerializeTupleStruct = Impossible<Self::Ok, Self::Error>;
type SerializeTupleVariant = Enum;
type SerializeMap = Impossible<Self::Ok, Self::Error>;
type SerializeStruct = Impossible<Self::Ok, Self::Error>;
type SerializeStructVariant = Enum;
fn serialize_bool(self, _v: bool) -> Result<Self::Ok> { Err(NotEnum) }
fn serialize_i8(self, _v: i8) -> Result<Self::Ok> { Err(NotEnum) }
fn serialize_i16(self, _v: i16) -> Result<Self::Ok> { Err(NotEnum) }
fn serialize_i32(self, _v: i32) -> Result<Self::Ok> { Err(NotEnum) }
fn serialize_i64(self, _v: i64) -> Result<Self::Ok> { Err(NotEnum) }
fn serialize_u8(self, _v: u8) -> Result<Self::Ok> { Err(NotEnum) }
fn serialize_u16(self, _v: u16) -> Result<Self::Ok> { Err(NotEnum) }
fn serialize_u32(self, _v: u32) -> Result<Self::Ok> { Err(NotEnum) }
fn serialize_u64(self, _v: u64) -> Result<Self::Ok> { Err(NotEnum) }
fn serialize_f32(self, _v: f32) -> Result<Self::Ok> { Err(NotEnum) }
fn serialize_f64(self, _v: f64) -> Result<Self::Ok> { Err(NotEnum) }
fn serialize_char(self, _v: char) -> Result<Self::Ok> { Err(NotEnum) }
fn serialize_str(self, _v: &str) -> Result<Self::Ok> { Err(NotEnum) }
fn serialize_bytes(self, _v: &[u8]) -> Result<Self::Ok> { Err(NotEnum) }
fn serialize_none(self) -> Result<Self::Ok> { Err(NotEnum) }
fn serialize_some<T: ?Sized + Serialize>(self, _value: &T) -> Result<Self::Ok> { Err(NotEnum) }
fn serialize_unit(self) -> Result<Self::Ok> { Err(NotEnum) }
fn serialize_unit_struct(self, _name: &'static str) -> Result<Self::Ok> { Err(NotEnum) }
fn serialize_unit_variant(self, _name: &'static str, _variant_index: u32, variant: &'static str) -> Result<Self::Ok> { Ok(variant) }
fn serialize_newtype_struct<T: ?Sized + Serialize>(self, _name: &'static str, _value: &T) -> Result<Self::Ok> { Err(NotEnum) }
fn serialize_newtype_variant<T: ?Sized + Serialize>(self, _name: &'static str, _variant_index: u32, variant: &'static str, _value: &T) -> Result<Self::Ok> { Ok(variant) }
fn serialize_seq(self, _len: Option<usize>) -> Result<Self::SerializeSeq> { Err(NotEnum) }
fn serialize_tuple(self, _len: usize) -> Result<Self::SerializeTuple> { Err(NotEnum) }
fn serialize_tuple_struct(self, _name: &'static str, _len: usize) -> Result<Self::SerializeTupleStruct> { Err(NotEnum) }
fn serialize_tuple_variant(self, _name: &'static str, _variant_index: u32, variant: &'static str, _len: usize) -> Result<Self::SerializeTupleVariant> { Ok(Enum(variant)) }
fn serialize_map(self, _len: Option<usize>) -> Result<Self::SerializeMap> { Err(NotEnum) }
fn serialize_struct(self, _name: &'static str, _len: usize) -> Result<Self::SerializeStruct> { Err(NotEnum) }
fn serialize_struct_variant(self, _name: &'static str, _variant_index: u32, variant: &'static str, _len: usize) -> Result<Self::SerializeStructVariant> { Ok(Enum(variant)) }
}
struct Enum(&'static str);
impl SerializeStructVariant for Enum {
type Ok = &'static str;
type Error = NotEnum;
fn serialize_field<T: ?Sized + Serialize>(&mut self, _key: &'static str, _value: &T) -> Result<()> { Ok(()) }
fn end(self) -> Result<Self::Ok> {
Ok(self.0)
}
}
impl SerializeTupleVariant for Enum {
type Ok = &'static str;
type Error = NotEnum;
fn serialize_field<T: ?Sized + Serialize>(&mut self, _value: &T) -> Result<()> { Ok(()) }
fn end(self) -> Result<Self::Ok> {
Ok(self.0)
}
}
t.serialize(VariantName).unwrap()
}
One thing that I'd suggest now that people are actually providing crates for this is to put the functions in a workspace similar to that for the unic crate: while one master serde-aux crate contains everything, smaller serde-aux-* crates contain self-contained bits of functionality.
Ideally, each sub-crate would be a piece that someone would use in one use case, so that people don't just import one giant crate from which most of the stuff in there is unused.
I'm not sure if there's any obvious logical grouping though...
Yeah, and it risks creating a fragmented ecosystem. It already feels like I end up adding the same ~10 crates every time I start a new rust crate. I don't want to have to look up which aux crate something belongs to, or have to search multiple crates for a helper function.
@spease Right. Let's stick to a serde-aux / serde-util crate, for these reasons, I think.
Came here to note that I find myself dealing with #908 for multiple classes, and that should probably go in serde-aux. Looks like it's already been noted.
Ran into multiple cases of #908 today. How does something like this seems as a solution?
#[proc_macro_derive(DeserializeFromStr)]
pub fn deserialize_from_str(input: TokenStream) -> TokenStream {
let ast: DeriveInput = syn::parse(input).unwrap();
let name = ast.ident;
let gen = quote! {
impl<'de> Deserialize<'de> for #name {
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
where D: Deserializer<'de>
{
let s = String::deserialize(deserializer)?;
FromStr::from_str(&s).map_err(de::Error::custom)
}
}
};
gen.into()
}
I am ready to close this issue because external Serde helper libraries now exist! Please continue to file use cases against those libraries, or make your own helper libraries as you see fit.
@dtolnay Do you know if all the above functions are in the serde-aux library? It would be nice to have an official helper library for serde, or at least some amount of official involvement...
I want to advertise another helper crate, which was not mentioned in the thread: serde_with
With regards to the helper functions from this issue, it is more complete than serde-aux. Some things, like the base64 helpers are not included, because there already exist other crates for that on crates.io.
Neither serde_with nor serde-aux is a superset of the other right now. Part of the reason is that serde-aux is only MIT licensed, while serde_with has a dual Apache2/MIT license, which prohibits code transfer from serde-aux to serde_with.
I am the maintainer of serde_with.
@alexreg
@jonasbb Sounds good. What do you think about trying to combine the two into some sort of serde-utils crate (or keeping the same name even), run by you and maybe with collaborator privileges for Serde Developers? I'd be willing to contribute and help integrate some serde-aux stuff too.
I am open to extending the crate or changing/migrating the helpers to fit with other crates. This thread is the wrong place to discuss this issue though. You are welcome to leave a comment on the serde_with repository, where we can discuss this in detail. There already is an issue for collaboration with serde-aux.