How to handle dynamically named fields?

[ronlawrence3]

Thank you for this library. It has made a complex mapping I have much simpler to do!

I have a situation where my destination json is required to come out like this ( I have simplified the structure a little for brevity):

[
  {
    "key": "some string",
    "field1": {...deeper structure},
    "field2": {...deeper structure}
  },
 ... other instances of this with differing fields...
]

Where "field1" and "field2" are dynamic strings for the keys and "deeper structure" is defined as another Resource definition (lets call it "sub-resource"). This comes from another well defined json input and I have the resource for this working great as is.

So my goal here was to dynamically add the "sub-resource" fields to the resource instance as its mapping from the source json to the destination.

In creating the Resource for this output json I did something like:

class MyThing(odin.Resource):
    key = odin.StringField(null=True)
    # nothing here where I used to have defined fields but now need dynamic ones...
    

and in the mapping function I tried to use getmeta(thing) and add a field to it with add_field, then set the resource's __dict__[field1_name] = <the sub-resource instance>. This seemed to work on field1, but when I tried to create another instance of the resource it would die with something like odin.exceptions.MappingExecutionError: attribute name must be string, not 'NoneType' applying rule FieldMapping(from_field=('key',), action='blah', to_field=('blah', '...'), to_list=False, bind=False, skip_if_none=False)

example of the code in the mapping method:

                string = field_name_from_other_json
                component.__dict__[string] = cd
                meta: ResourceOptions = getmeta(component)
                meta.add_field(field=odin.DictAs(SubResourceClass, null=True, name=string))

Is there a better way to accomplish what I'm after, or would I be better off just doing this transformation using another approach?

[timsavage]

Hi Ron,

Thanks for the feedback.

Dynamic fields are doable, although you would have to generate the target resource and mapping first before execution. I don't think adding fields during a mapping operation would work.

I will see if I can find an example of doing this.

[timsavage]

One option could be to use a DictOf field. The DictOf field lets you define a Dict with a string key and a resource as the value.

Typically a dynamic field would still be setup before performing any mapping operations.

[ronlawrence3]

Thanks Tim, I'll attempt that. In my above example what would that look like?

class MyThing(odin.Resource):
    key = odin.StringField(null=True)
    ??? = odin.DictOf(???)

or would I do it another way programatically prior to mapping to this resource?

[ronlawrence3]

Below is what I came up with that works... suggestions for improvement are welcome:

import odin
from odin.codecs import dict_codec


class TestInputSub(odin.Resource):
    input1 = odin.StringField()
    input2 = odin.StringField()


class TestInput(odin.Resource):
    toplevel = odin.ListOf(TestInputSub)


class TestOutputSubSub(odin.Resource):
    name = odin.StringField()


class TestOutputSub(odin.Resource):
    field1 = odin.StringField()
    # dynamic here... defined in dynamic class in mapping


class TestOutput(odin.Resource):
    things = odin.ListOf(TestOutputSub)


class TestMapping(odin.Mapping):
    from_obj = TestInput
    to_obj = TestOutput

    @odin.map_field(to_field='things', from_field='toplevel', to_list=True)
    def toplevel_to_things(self, toplevel):
        result = []
        for input_sub in toplevel:
            cls_dict = dict(dyn1=odin.ObjectAs(TestOutputSubSub),
                            __module__='')
            DynResource = type('DynResource', (TestOutputSub, ), cls_dict)

            subsub = TestOutputSubSub(name=input_sub.input2)
            fields = {'field1': 'something', 'dyn1': subsub}
            sub = dict_codec.load(fields, DynResource)

            result.append(sub)
        return result


# run a test:
resp = dict_codec.load({'toplevel': [{
    'input1': 'out1',
    'input2': 'out2'
}]}, TestInput)
obj = TestMapping.apply(resp)
print(
    dict_codec.dump(obj,
                    include_type_field=False,
                    include_virtual_fields=False))
# {'things': [{'dyn1': {'name': 'out2'}, 'field1': 'something'}]}

[timsavage]

My only concern would be creating an awful lot of DynResource types, this could start using a lot of memory as every DynResource instance will also have its own type (and ResourceObject instance). If your toplevel list is small that might be acceptable, however, if that list is large it could get out of hand (internally Odin shares ResourceObject between all instances of a particular Resource type, saving memory and initialisation time)

Looking out your output this would give you the same result:

class TestOutputSub(odin.Resource):
    field1 = odin.StringField()
    dyn1 = odin.TypedDictField(odin.DictAs(TestOutputSubSub))

That would generate the same output as you have defined, and allow arbitrary key names within dyn1.

(I just realised that TypedDictField is missing from the docs!)

[ronlawrence3]

I understand, but I think I need to generate a new resource class with field names matching a value from the source object when I'm mapping, not when I define the resource class. In this contrived example, the "dyn1" could be "dyn2" or some other name from the source data depending on the input data values. if there is a better way to add fields to a resource, I'd be happy with that too. (sorry my example code was not a good one for this nuance)