Clarification needed on collection events

[thallgren]

It's a bit unclear to me how to use the collection events.

If a new value is added, an add event is sent.
If a value is deleted, remove event is sent.
What event should be send if a value is replaced?

I'm guessing that for a collection ["a", "b"], an add event with payload {"value": "x", "idx": 1} implies that "x" was inserted at position 1 to get the result ["a", "x", "b"]. Or does it imply that a replace with the result ["a", "x"] took place? If my guess is correct, what event do I send to denote that the value at index 1 was replaced?

Please add some text that clarifies this.

Side note:
I would have expected an add to mean "new value was inserted at index (and all values from index was shifted right)" and a set event meaning "value at index was replaced".

[jirenius]

I will start by clarify here (and then clarifying in the specification).

First to confirm your assumptions:

• add event inserts the value at the index position, and shifts all previous values with the same index or higher to the "right".
• remove event removes the value at the index position, and shifts all previous values with a higher index to the "left".
• set event doesn't exist. There is no set event in the specs.

These two events are enough to describe any mutation between two states of a collection.

A replace can be seen as a remove followed by an add at the same index.
A move can be seen as a remove followed by an add of the same value at a different index.

Why don't we have replace and move events?

Are replace and move redundant? Not entirely. Sometimes you wish not just to know the new state, but also how the data transformed into the new state.

Maybe the front-end UI wish to do different types of animations depending on the transformation. Maybe to animate a replaced value in a lice by cross-fading from the old to the new, while a removed value should have the height of the graphical component collapsing.

But, to introduce these two new events in the spec requires consideration:

• Services can choose to ignore sending them. No problem.
• Resgate implementation is fairly easy, and only needs to be done once. No big problem.
• ResClient implementation is fairly easy. But this would need to be done for each client library for each language. No huge problem.
• Client UI. Every graphical component needs to be able to handle these events as well, even if they just fall back to doing an remove/add combo (or just "rerenders" as in the case of React). This is more problematic.

The task for this is in Q2 2020 on the roadmap (https://resgate.io/docs/about/roadmap/#fn:res-replace-move-events). But it is still not decided if this should be done as a new resource type (Eg. ExtendedCollection), or by extending the existing Collection type.

Hopes that clarifies! :)

/Samuel

[thallgren]

Thanks for that elaborate explanation. I have some thoughts around this.

1. I would consider replace far more important than move. Simply because it's very common to replace and fairly uncommon to move. Also, a move that affects a major part of the collection is probably often interchangeable with a reset.

2. Using remove followed by an add to simulate a replace and expect clients to act on both events can be very resource consuming on large collections. There's much to gain by using a replace. I'm glad to hear that it's in the roadmap.

3. My personal preference would be if replace made it into the existing Collection type and the, IMO, more esoteric move ended up in an ExtendedCollection.

[jirenius]

Thanks or the input. Always appreciated and welcome! A few thoughts:

1. The move event is useful for any type of scoreboard; eg. top traded stocks, leader boards, top load monitoring for services. Whenever the internal ordering may change, and you want to visualize how the item moved from one position to another.

   The proposed move event would only describe the move of a single value within a collection, and would not be able to affect "major parts". Only like this (moving 2): 1,2,3,4 -> 1,3,4,2

2. True that remove+add may cause overhead on the client if it first renders the remove, then the add. This is another reason why the two new events are considered.

3. I can't tell if move or replace would be most commonly used. To be honest, I am not entirely sure what sort of use case would typically benefit from replace.

   What sort of data would you use it for? Understanding the use helps in the decision on designing the replace event.

[thallgren]

The service I'm working on will maintain data that describes things like targets (nodes identified with certificate name and fully qualified domain name) which in turn can have configurations and facts in the form of maps of key/value where value can be maps, arrays, etc. nested to any depth. So, in essence, I want to manage nested data structures of arbitrary complexity.

The current service loads data from files on disk and has listeners that checks when those files are modified. I've written logic to compare two trees and generate all resgate events needed to propagate the resulting delta. This logic, when comparing arrays, creates replace events whenever it finds values that differ at the same index. It's not much different from comparing maps and create model changes when values differ at the same key. It's not unlikely that the changes stem from adds, removals, or even reordering but I haven't written any logic to analyze that.

I wrote a ModelList javascript class (similar to CollectionList but it uses a componentFactory that acts on key/value instead of item/index). I came to realize that visualizing moves, adds, and removes is really nice on a web-page. I would surely spend more time analyzing array deltas if that was important for the stuff I'm writing.

[jirenius]

Thanks for the explanation!

Then I understand why you want replace. Most algorithms for generating diffs for lists/collections uses replace in addition to add and remove. Resgate instead uses the (LCS algorithm](https://en.wikipedia.org/wiki/Longest_common_subsequence_problem), with a modification that prioritized the remove events before adds (to prevent momentary duplicates).

You can find the implementation in resourceSubscription.go.

The specification has been clarified in commit f29d7bd, so I close this issue.