Resilient indexing and aggregation services

[anomit]

Is your feature request related to a problem?
The present architecture of the indexer and aggregator services that generate important datapoints for the Uniswap v2 use case does not deal well with broken chains and skipped snapshots that can arise because of

• network errors
• cache errors
• RPC timeouts or rate limits
  ...among many other factors that can go wrong.

Its attempt to find a common end epoch across 100+ projects to just get started on building aggregates along with a lot of convoluted logic on trying to sync up other higher order aggregates with the same causes the entire process to come to a halt in case of above issues creating gaps or duplicate entries in snapshot chains. The end result is that even though snapshots continue to proceed fine, there are no usable data points generated or updated.

Describe the solution you'd like

Move the indexing and aggregation ahead as long as majority of the snapshotted projects' snapshot chains progress fine. A couple of projects with broken chains should not cause an entire cessation in data point generation and updation.

Describe alternatives you've considered

Not applicable

Additional context

This scope is only for the phase where these nodes participate in an offchain consensus system and maintain a local state of the DAG chain and other indexes and aggregates on top of it. In the next phase of release, the protocol state of DAG chains, indexes, aggregates will be available on its own protocol chain and breakages in snapshot chains will not be a matter of concern.

Apart from the logical issue itself of halted indexing and aggregation, there are violations of accepted Pythonic design patterns, coding styles and standards which need to be addressed.