Request: Examples of using yaml-set, especially for insertion and structured values

[AndydeCleyre]

Is your feature request related to a problem? Please describe.
I don't know if it's possible to use yaml-set to insert elements that don't match any existing elements, I don't know if it's possible to use yaml-set to insert hashes, etc.

Describe the solution you'd like
A few examples of yaml-set operations could be added to the readme.

Describe alternatives you've considered
I've been guessing the syntax, but either those things are not possible or I'm just bad at guessing.

Additional context
For example, with a small file default.yml:

matches:
  - trigger: :lem
    replace: Ł

Let's say I want to insert a hash as another element of the matches list, that looks like:

trigger: btw
replace: by the way

I've tried things like:

$ yaml-set -g 'matches[trigger=btw]' -a '{"trigger": "btw", "replace": "by the way"}'

and

$ data='trigger: btw
replace: by the way'
$ yaml-set -g 'matches[trigger=btw]' -a "$data" default.yml

But I think nothing changes because no matches are found. If I add an element for it to match, though, the element is replaced by a string instead of a hash.

[AndydeCleyre]

Really a straight comparison to jq would be immensely helpful. For example, using jq's tutorial:

$ curl 'https://api.github.com/repos/stedolan/jq/commits?per_page=5' -o r.json
$ jq '.[0]' < r.json
$ yaml-get -p '[0]' r.json

etc.

[wwkimball]

I'll start by saying I accept your request for more documentation. I consider the README file to be search fodder; it mentions the most notable capabilities of the project. More depth and exploration are meant for the project Wiki (linked in the README and found at https://github.com/wwkimball/yamlpath/wiki). Otherwise, the already very long README file would become untenable. I will spend more time discussing the specific options for each of the reference commands, including yaml-set, at the Wiki.

However, I've never used jq. If you'd like to write up a comparison, I'd be happy to share it on the project Wiki.

Now, onto the underlying technical capability you are really asking for. Merging two YAML/compatible structures is outside the scope of yaml-set. It may not be obvious, but indeed, you are asking for a merge of two documents. The reference yaml-set tool supports only scalar values. An entirely different -- far more complex -- tool is necessary to merge two complex data structures together.

That said, yaml-set is more capable than might be obvious and with multiple commands, you can force it to build up an original data structure. To wit, yaml-set can add missing parent keys and elements when instructed to add a value at a non-existent YAML Path. This is no substitute for a true YAML merging tool, but it works in a pinch for trivial needs.

For example, take your sample target file, default.yml:

---
matches:
  - trigger: :lem
    replace: Ł

In this case, adding a new element to the matches array requires knowing the index of one more element than the array has (and remember that the first element is at index zero (0)). Like I said already, this is no substitute for a true YAML merging tool. To add the new match using yaml-set, run these commands:
yaml-set --change /matches/[1]/trigger --value btw default.yml
yaml-set --change /matches/[trigger=btw]/replace --value 'by the way' default.yml

The changes default.yml to the following:

---
matches:
  - trigger: :lem
    replace: Ł

  - trigger: btw
    replace: by the way

As a bonus example, consider that this command:
yaml-set --change /something/else/with/a/deeply/nested/structure/[5] --value 'yaml-set will build all missing parents' default.yml

produces this YAML:

---
matches:
  - trigger: :lem
    replace: Ł

  - trigger: btw
    replace: by the way
something:
  else:
    with:
      a:
        deeply:
          nested:
            structure:
              - ''
              - ''
              - ''
              - ''
              - ''
              - yaml-set will build all missing parents

All considered, this is a workaround. You present what is becoming a frequently-requested-feature. A while back, someone else even attempted to add the capability you are looking for via a Pull Request; it fizzled out because what you are looking for is extremely complex, despite how simple it may seem. In reality, it is not simple to inject complex structures into a YAML document; we aren't just appending some text into another text document. I'll explain.

YAML supports "anchors" and "aliases". Merging logic is required because both documents must be carefully merged together in order to detect and coalesce any new versus pre-existing anchors. We must protect existing anchors from attempts to override them in a new child lest the document become corrupt or change unexpectedly. Did the user mean to redefine the existing anchor or is this a typo and a new anchor was intended? This cannot be safely guessed.

In addition, the target data structure must be considered; it's all too simple to mistakenly attempt to add a scalar list element to a parent that previously only had maps (hashes), or a map to a list (array) of scalars. Both of these are invalid and would corrupt the document, so care must be taken to enforce that each new element is valid at the given path.

Further, consider the case of an obvious merge. If you attempt to set a complex data structure anywhere other than the leaf of a target data structure, you are obviously attempting to merge two complex data structures together. How should conflicts be handled? Are lists supposed to contain only unique elements? What about when it's a list of maps; how is uniqueness determined? What happens when one map has a different structure than the target already has? All of these questions must ultimately be answered by the user. A formal document merging tool must provide a means of obtaining these answers from the user. Such a tool is vastly beyond the scope of yaml-set.

One day, I may take up the challenge of creating such a tool. It would be no small feat.

[AndydeCleyre]

Thanks very much for the explanation!

[wwkimball]

@AndydeCleyre (and everyone else looking for a means of merging YAML/Compatible documents/data together): By popular demand, I have finally written a new yaml-merge tool. It replaces yaml-set for adding complex data.

Using your original requirement to demonstrate:

1. Create a YAML document (56left.yaml) with content:

---
matches:
  - trigger: :lem
    replace: Ł

2. Employing the - pseudo-file as input (to read from STDIN), add the new /matches entry from the command-line via yaml-merge like so: echo "{trigger: btw, replace: by the way}" | yaml-merge --mergeat=/matches 56left.yaml -

The result is:

---
matches:
  - trigger: :lem
    replace: Ł
  - trigger: btw
    replace: by the way

In addition to passing the new data via STDIN, you could also merge complete or fragmented YAML documents together to achieve the same result.

This requires yamlpath at least version 2.4.0. More documentation for the new yaml-merge command is available at yaml-merge --help and the Wiki (which I'm actively adding to) at: https://github.com/wwkimball/yamlpath/wiki/yaml-merge

[wwkimball]

I am closing this issue as complete. I have exhaustively documented the new yaml-merge tool which full addresses the original use-case.

Thank you!