Support for multiple data files for a single template

Question

Support for multiple data files for a single template

oharboe opened this issue 3 years ago · 11 comments

@David-Cowan Sometimes it's useful to pull multiple data files together.

Is there any interest in adding such a feature?

Maybe something like:

.. datatemplate:yaml:: file1.yaml file2.yaml
   {{data[0].foo}}
   {{data[1].bar}}

Answer 1 · 2021-06-04T17:58:31.000Z

Nice idea, I can see how that would be useful.

I worry a bit that someone might want to combine a JSON file and a YAML file and use them both with a template, so we would need to think about whether we want to support that and if so how the user would express the formats for the different inputs.

I considered suggesting that we leave the data argument alone and add a data_source template argument that is a map from filename to data structure. After thinking about it more, I think it does make sense to just change data for this case. I don't know whether a sequence (as you propose) or map would work better. The sequence involves less typing, I guess, and how many files is someone actually likely to try to combine?

Answer 2 · 2021-06-04T18:07:46.000Z

In my specific case it is a desire to join together two (or maybe more, but not very many) yaml files to "join" in description text to more functional descriptions of a thing. Functional descriptions of a thing can be generated from code, descriptions are best hand edited. And thus in separate files. But that barely scratches the surface of the uses I would put the feature to. I envisioned this as simply a virtual appending of the two (or more) files together prior to parsing, and would expect either unpredictable behaviour or errors (hopefully useful ones) if there were two identically defined (but not populated) collections in the separate files I can see the use of joining json to yaml, but that is necessarily more complex.

…

On Fri, 4 Jun 2021 at 10:58, Doug Hellmann ***@***.***> wrote: Nice idea, I can see how that would be useful. I worry a bit that someone might want to combine a JSON file and a YAML file and use them both with a template, so we would need to think about whether we want to support that and if so how the user would express the formats for the different inputs. I considered suggesting that we leave the data argument alone and add a data_source template argument that is a map from filename to data structure. After thinking about it more, I think it does make sense to just change data for this case. I don't know whether a sequence (as you propose) or map would work better. The sequence involves less typing, I guess, and how many files is someone actually likely to try to combine? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#80 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQJ2VVLPYOV4R5ALO2JTOLDTREH5RANCNFSM46CDWJHQ> .

-- *CONFIDENTIALITY NOTICE* -- This email is intended only for the person(s) named in the message header. Unless otherwise indicated, it contains information that is confidential, privileged and/or exempt from disclosure without written approval by Ascenium Inc.

Answer 3 · 2021-06-04T19:26:02.000Z

OK, let's not worry too much about the multiple format angle, for now.

It would help to have some small but specific examples of how you might want to join 2 files together, to work out a way to express that. For example, if the top level structure of the file is a map, then joining them with .update() would make sense and we would just need to set a precedence order to explain what happens when keys collide. If the 2 files are both lists, .extend() would work. But if one file is a map and one is a list, then the approach @oharboe suggested where the template refers explicitly to each file in some way would be safer. The template author knows the content structure, after all.

Answer 4 · 2021-06-04T20:07:06.000Z

Lets spell out a use case Company manufacturers cars: Kanban system has the following information that can be exported to yaml for use in documentation about inventory. (Not real data but a real match for the use case) partnumbers-inventory.yaml --- wip: 1WB0002: quantity 1001 2DR0013: quantity 401 1DX0077: quantity 14 Separate manually maintained descriptions yaml file pn-description.yaml --- partnumber_descriptions: 1WB0002: description: "Rear Window" 2DR0013: description: "Rear Door" 1DX0077: description: "Hatchback Door" Being able to use both yaml files in the same datatemplate would clearly result in a more human readable table Sphinx example .. datatemplate:yaml:: ../some-dir/partnumbers-inventory.yaml, ../pn-description.yaml (used Oyvind sample syntax, but any you mentioned would work) .. list-table:: What's in stock :header-rows: 1 * - part-num - quantity - description {% for item in data['wip'] | sort %} * - {{item}} - {{item.quantity}} - {{data['partnumber_descriptions'].description}} Can't run this example to test it, but I think it makes a use case clear. What we mostly need to get flat is the syntax for included files, and you have a point that mixing yaml, json, and such things as xml (why does anyone use xml?) might be desirable. And would require different syntax than Oyvind put forth. .csv is clearly not mixable with other key/pair formats.

…

-david

On Fri, 4 Jun 2021 at 12:26, Doug Hellmann ***@***.***> wrote: OK, let's not worry too much about the multiple format angle, for now. It would help to have some small but specific examples of how you might want to join 2 files together, to work out a way to express that. For example, if the top level structure of the file is a map, then joining them with .update() would make sense and we would just need to set a precedence order to explain what happens when keys collide. If the 2 files are both lists, .extend() would work. But if one file is a map and one is a list, then the approach @oharboe <https://github.com/oharboe> suggested where the template refers explicitly to each file in some way would be safer. The template author knows the content structure, after all. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#80 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQJ2VVOZ2TH6MOKSS5455WDTRESFXANCNFSM46CDWJHQ> .

-- *CONFIDENTIALITY NOTICE* -- This email is intended only for the person(s) named in the message header. Unless otherwise indicated, it contains information that is confidential, privileged and/or exempt from disclosure without written approval by Ascenium Inc.

Answer 5 · 2021-06-05T15:59:16.000Z

Maybe instead of extending the directive syntax, we should expose the loader functions. Then a template author could use the set statement to load data into a variable and use it however they want.

https://jinja.palletsprojects.com/en/3.0.x/templates/#assignments

Answer 6 · 2021-06-05T19:57:31.000Z

Maybe instead of extending the directive syntax, we should expose the loader functions. Then a template author could use the set statement to load data into a variable and use it however they want.

https://jinja.palletsprojects.com/en/3.0.x/templates/#assignments

Makes sense. Example syntax?

Answer 7 · 2021-06-05T20:00:49.000Z

So my syntax would become something like this?? .. datatemplate:yaml:: ../some-dir/partnumbers-inventory.yaml .. list-table:: What's in stock :header-rows: 1 {% set description = loader(../pn-description.yaml,"partnumber_descriptions" ) %} * - part-num - quantity - description {% for item in data['wip'] | sort %} * - {{item}} - {{item.quantity}} - {{description[item]}}

…

On Sat, 5 Jun 2021 at 08:59, Doug Hellmann ***@***.***> wrote: Maybe instead of extending the directive syntax, we should expose the loader functions. Then a template author could use the set statement to load data into a variable and use it however they want. https://jinja.palletsprojects.com/en/3.0.x/templates/#assignments — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#80 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQJ2VVIUME34EIRCP4AN5ZLTRJCWBANCNFSM46CDWJHQ> .

-- *CONFIDENTIALITY NOTICE* -- This email is intended only for the person(s) named in the message header. Unless otherwise indicated, it contains information that is confidential, privileged and/or exempt from disclosure without written approval by Ascenium Inc.

Answer 8 · 2021-06-05T20:02:13.000Z

or more likely .. datatemplate:yaml:: ../some-dir/partnumbers-inventory.yaml .. list-table:: What's in stock :header-rows: 1 {% set description = loader(../pn-description.yaml:yaml,"partnumber_descriptions" ) %} * - part-num - quantity - description {% for item in data['wip'] | sort %} * - {{item}} - {{item.quantity}} - {{description[item]}}

…

On Sat, 5 Jun 2021 at 13:00, David Cowan ***@***.***> wrote: So my syntax would become something like this?? .. datatemplate:yaml:: ../some-dir/partnumbers-inventory.yaml .. list-table:: What's in stock :header-rows: 1 {% set description = loader(../pn-description.yaml,"partnumber_descriptions" ) %} * - part-num - quantity - description {% for item in data['wip'] | sort %} * - {{item}} - {{item.quantity}} - {{description[item]}} On Sat, 5 Jun 2021 at 08:59, Doug Hellmann ***@***.***> wrote: > Maybe instead of extending the directive syntax, we should expose the > loader functions. Then a template author could use the set statement to > load data into a variable and use it however they want. > > https://jinja.palletsprojects.com/en/3.0.x/templates/#assignments > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#80 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AQJ2VVIUME34EIRCP4AN5ZLTRJCWBANCNFSM46CDWJHQ> > . >

-- *CONFIDENTIALITY NOTICE* -- This email is intended only for the person(s) named in the message header. Unless otherwise indicated, it contains information that is confidential, privileged and/or exempt from disclosure without written approval by Ascenium Inc.

Answer 9 · 2021-06-05T20:45:41.000Z

See what you think of #81 as an approach.

Answer 10 · 2021-06-07T03:05:36.000Z

This appears to work for me. Thank You Essentially creates another "data" object with the name I give it when I load into it. I tried driving a list-table both ways and it both worked. Loaded test.yaml to test and then did a list-table based on a list in test, supplemented with a field from data. And a list-table based on a list in data supplemented with a field from the test (loaded data). Lots of potential use here. Not sure if there are any limits on how big a dataset I can "load", my testing has been non-exhaustive so far.

…

-david

On Sat, 5 Jun 2021 at 13:45, Doug Hellmann ***@***.***> wrote: See what you think of #81 <#81> as an approach. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#80 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQJ2VVI4WGLBJKKRDB3B7BTTRKEIDANCNFSM46CDWJHQ> .

-- *CONFIDENTIALITY NOTICE* -- This email is intended only for the person(s) named in the message header. Unless otherwise indicated, it contains information that is confidential, privileged and/or exempt from disclosure without written approval by Ascenium Inc.

Answer 11 · 2021-06-07T12:51:47.000Z

The limit would likely be RAM, since all of the data is loaded into memory immediately. That's the case for all of the formats currently supported.