Nestru

Mix.install([:nestru], force: true, consolidate_protocols: false)

About

🔗 Full documentation is on hexdocs.pm

🔗 JSON parsing example is in elixir-decode-validate-json-with-nestru-domo repo.

Description

A library to serialize between maps and nested structs.

Turns a map into a nested struct according to hints given to the library. And vice versa turns any nested struct into a map.

It works with maps/structs of any shape and level of nesting. Highly configurable by implementing Nestru.Decoder and Nestru.Encoder protocols for structs.

Useful for translating map keys to struct's fields named differently. Or to specify default values missing in the map and required by struct.

The library's primary purpose is to serialize a map coming from a JSON payload or an Erlang term; at the same time, the map can be of any origin.

The input map can have atom or binary keys. The library takes the binary key first and then the same-named atom key if the binary key is missing while decoding the map. The library generates maps with atom keys during the struct encode operation.

Tour

Let's say we have an Order with a total field which is an instance of a Total struct. And we want to serialize between an instance of Order and a map.

Firstly, let's derive Nestru.Encoder and Nestru.Decoder protocols and give a hint that the field :total should hold a value of Total struct like the following:

defmodule Order do
  @derive [Nestru.Encoder, {Nestru.Decoder, hint: %{total: Total}}]
  defstruct [:id, :total]
end

defmodule Total do
  @derive [Nestru.Encoder, Nestru.Decoder]
  defstruct [:sum]
end

{:module, Total, <<70, 79, 82, 49, 0, 0, 8, ...>>, %Total{sum: nil}}

Secondly, we can encode the Order into the map like that:

model = %Order{id: "A548", total: %Total{sum: 500}}
{:ok, map} = Nestru.encode(model)

{:ok, %{id: "A548", total: %{sum: 500}}}

And decode the map back into the Order like the following:

map = %{
  "id" => "A548",
  "total" => %{"sum" => 500}
}

{:ok, model} = Nestru.decode(map, Order)

{:ok, %Order{id: "A548", total: %Total{sum: 500}}}

As you can see the data markup is in place, the Total struct is nested within the Order struct.

A list of structs in a field

Let's add the :items field to Order1 struct to hold a list of LineItems and give a hint to Nestru on how to decode that field:

defmodule Order1 do
  @derive {Nestru.Decoder, hint: %{total: Total, items: [LineItem]}}

  defstruct [:id, :items, :total]
end

defmodule LineItem do
  @derive Nestru.Decoder
  defstruct [:amount]
end

{:module, LineItem, <<70, 79, 82, 49, 0, 0, 8, ...>>, %LineItem{amount: nil}}

Let's decode:

map = %{
  "id" => "A548",
  "items" => [%{"amount" => 150}, %{"amount" => 350}],
  "total" => %{"sum" => 500}
}

{:ok, model} = Nestru.decode(map, Order1)

{:ok,
 %Order1{
   id: "A548",
   items: [%LineItem{amount: 150}, %LineItem{amount: 350}],
   total: %Total{sum: 500}
 }}

Voilà, we have field values as nested structs 🎉

For the case when the list contains several structs of different types, please, see the Serializing type-dependent fields section below.

Date Time and URI

Let's say we have an Order2 struct with some URI and DateTime fields in it. These attributes are structs in Elixir, at the same time they usually kept as binary representations in a map.

Nestru supports conversion between binaries and structs, all we need to do is to implement the Nestry.Encoder and Nestru.Decoder protocols for these structs like the following:

# DateTime
defimpl Nestru.Encoder, for: DateTime do
  def gather_fields_from_struct(struct, _context) do
    {:ok, DateTime.to_string(struct)}
  end
end

defimpl Nestru.Decoder, for: DateTime do
  def decode_fields_hint(_empty_struct, _context, value) do
    case DateTime.from_iso8601(value) do
      {:ok, date_time, _offset} -> {:ok, date_time}
      error -> error
    end
  end
end

# URI
defimpl Nestru.Encoder, for: URI do
  def gather_fields_from_struct(struct, _context) do
    {:ok, URI.to_string(struct)}
  end
end

defimpl Nestru.Decoder, for: URI do
  def decode_fields_hint(_empty_struct, _context, value) do
    URI.new(value)
  end
end

{:module, Nestru.Decoder.URI, <<70, 79, 82, 49, 0, 0, 8, ...>>, {:decode_fields_hint, 3}}

Order2 is defined like this:

defmodule Order2 do
  @derive [Nestru.Encoder, {Nestru.Decoder, hint: %{date: DateTime, website: URI}}]
  defstruct [:id, :date, :website]
end

{:module, Order2, <<70, 79, 82, 49, 0, 0, 8, ...>>, %Order2{id: nil, date: nil, website: nil}}

We can encode it to a map with binary fields like the following:

order = %Order2{id: "B445", date: ~U[2024-03-15 22:42:03Z], website: URI.parse("https://www.example.com/?book=branch")}

{:ok, map} = Nestru.encode(order)

{:ok, %{id: "B445", date: "2024-03-15 22:42:03Z", website: "https://www.example.com/?book=branch"}}

And decode it back:

Nestru.decode(map, Order2)

{:ok,
 %Order2{
   id: "B445",
   date: ~U[2024-03-15 22:42:03Z],
   website: %URI{
     scheme: "https",
     userinfo: nil,
     host: "www.example.com",
     port: 443,
     path: "/",
     query: "book=branch",
     fragment: nil
   }
 }}

Error handling and path to the failed part of the map

Every implemented function of Nestru protocols can return {error, message} tuple in case of failure. When Nestru receives the error tuple, it stops conversion and bypasses the error to the caller.

defmodule Location do
  @derive {Nestru.Decoder, hint: %{street: Street}}
  defstruct [:street]
end

defmodule Street do
  @derive {Nestru.Decoder, hint: %{house: House}}
  defstruct [:house]
end

defmodule House do
  defstruct [:number]

  defimpl Nestru.Decoder do
    def decode_fields_hint(_empty_struct, _context, value) do
      if Nestru.has_key?(value, :number) do
        {:ok, %{}}
      else
        {:error, "Can't continue without house number."}
      end
    end
  end
end

So when we decode the following map missing the number value, we will get the error back:

map = %{
  "street" => %{
    "house" => %{
      "name" => "Party house"
    }
  }
}

{:error, error} = Nestru.decode(map, Location)

{:error,
 %{
   get_in_keys: [#Function<8.67001686/3 in Access.key!/1>, #Function<8.67001686/3 in Access.key!/1>],
   message: "Can't continue without house number.",
   path: ["street", "house"]
 }}

Nestru wraps the error message into a map and adds path and get_in_keys fields to it. The path values point to the failed part of the map which can be returned like the following:

get_in(map, error.get_in_keys)

%{"name" => "Party house"}

Maps with different key names

In some cases, the map's keys have slightly different names compared to the target's struct field names. Fields that should be decoded into the struct can be gathered by adopting Nestru.PreDecoder protocol like the following:

defmodule Quote do
  @derive [
    {Nestru.PreDecoder, translate: %{"cost_value" => :cost}},
    Nestru.Decoder
  ]

  defstruct [:cost]
end

When we decode the map, Nestru will put the value of the "cost_value" key for the :cost key into the map and then complete the decoding:

map = %{
  "cost_value" => 1280
}

Nestru.decode(map, Quote)

{:ok, %Quote{cost: 1280}}

For more sophisticated key mapping you can implement the gather_fields_for_decoding/3 function of Nestru.PreDecoder explicitly.

Serializing type-dependent fields

To convert a struct with a field that can have the value of multiple struct types into the map and back, the type of the field's value should be persisted. It's possible to do that like the following:

defmodule BookCollection do
  defstruct [:name, :items]

  defimpl Nestru.Encoder do
    def gather_fields_from_struct(struct, _context) do
      items_kinds =
        Enum.map(struct.items, fn %module{} ->
          module
          |> Module.split()
          |> Enum.join(".")
        end)

      {:ok, %{name: struct.name, items: struct.items, items_kinds: items_kinds}}
    end
  end

  defimpl Nestru.Decoder do
    def decode_fields_hint(_empty_struct, _context, value) do
      items_kinds =
        Enum.map(value.items_kinds, fn module_string ->
          module_string
          |> String.split(".")
          |> Module.safe_concat()
        end)

      {:ok, %{items: &Nestru.decode_from_list(&1, items_kinds)}}
    end
  end
end

defmodule BookCollection.Book do
  @derive [Nestru.Encoder, Nestru.Decoder]
  defstruct [:title]
end

defmodule BookCollection.Magazine do
  @derive [Nestru.Encoder, Nestru.Decoder]
  defstruct [:issue]
end

Let's convert the nested struct into a map. The returned map gets extra items_kinds field with types information:

alias BookCollection.{Book, Magazine}

collection = %BookCollection{
  name: "Duke of Norfolk's archive",
  items: [
    %Book{title: "The Spell in the Chasm"},
    %Magazine{issue: "Strange Hunt"}
  ]
}

{:ok, map} = Nestru.encode(collection)

{:ok,
 %{
   items: [%{title: "The Spell in the Chasm"}, %{issue: "Strange Hunt"}],
   items_kinds: ["BookCollection.Book", "BookCollection.Magazine"],
   name: "Duke of Norfolk's archive"
 }}

And restoring of the original nested struct is as simple as that:

{:ok, collection} = Nestru.decode(map, BookCollection)

{:ok,
 %BookCollection{
   items: [
     %BookCollection.Book{title: "The Spell in the Chasm"},
     %BookCollection.Magazine{issue: "Strange Hunt"}
   ],
   name: "Duke of Norfolk's archive"
 }}

Use with other libraries

Jason

JSON maps decoded with Jason library are supported with both binary and atoms keys.

ex_json_schema

ex_json_schema library can be used before decoding the input map with the JSON schema. To make sure that the structure of the input map is correct.

ExJSONPath

ExJsonPath library allows querying maps (JSON objects) and lists (JSON arrays), using JSONPath expressions. The queries can be useful in Nestru.PreDecoder.gather_fields_for_decoding/3 function to assemble fields for decoding from a map having a very different shape from the target struct.

Domo

You can use the Domo library to validate the t() types of the nested struct values after decoding with Nestru.

Domo can validate a nested struct in one pass, ensuring that the struct's field values match its t() type and associated preconditions.

Changelog

1.0.1

Rename list functions decode_from_list_of_maps to decode_from_list and encode_to_list_of_maps to encode_to_list

1.0.0

Convert structs to/from binaries for better serialization of DateTime and URI to/from strings
Breaking changes in function names:
- Nestru.PreDecoder.gather_fields_from_map/3 has been renamed to gather_fields_for_decoding/3
- Nestru.Decoder.from_map_hint/1 has been renamed to Nestru.Decoder.decode_fields_hint/1
- Nestru.decode_from_map/3 has been renamed to Nestru.decode/3
- Nestru.encode_to_map/2 has been renamed to Nestru.encode/2
The Nestru.Decoder.decode_fields_hint/3 can now return a struct as a hint as {:ok, %struct{}}. In this case Nestru.decode/3 will return the struct as the decoded value.

0.3.3

Fix the regress - make the decoding of an empty list return an empty list

0.3.2

Return error from decode_from_list_of_maps(!)/2/3 for non-list values

0.3.1

Add :only and :except options for deriving of Nestru.Encoder protocol
Add explicit :translate option for deriving of Nestru.PreDecoder protocol
Add explicit :hint option for deriving of Nestru.Decoder protocol

0.3.0

Rename Nestru.PreDecoder.gather_fields_map/3 to gather_fields_for_decoding/3.
Rename Nestru.Encoder.encode/1 to Nestru.Encoder.gather_fields_from_struct/2
Make encode(!)/2 work only with structs and add encode_to_list_of_maps(!)/2 for lists.
Add context parameter to encode_to_* functions.

0.2.1

Fix decode(!)/2/3 to return the error for not a map value.

0.2.0

Fix to ensure the module is loaded before checking if it's a struct
Add decode and encode verbs to function names
Support [Module] hint in the map returned from decode_fields_hint to decode the list of structs
Support %{one_key: :other_key} mapping configuration for the PreDecoder protocol in @derive attribute.

0.1.1

Add has_key?/2 and get/3 map functions that look up keys both in a binary or an atom form.

0.1.0

Initial release.

License

This project is licensed under the MIT license.

IvanRublev/Nestru