stellar/go

services/horizon/ingest: express parsing routines for Offers as public, reusable functions

Opened this issue · 0 comments

sreuland commented

What problem does your feature solve?

New application development that needs to run ingestion pipeline will tend to implement(repeat) many similar parsing routines to generate derived 'Offer' models that Horizon ingestion does internally.

Horizon has many routines that are package private, internal to ingestion which parse and derive 'Offer' from stellar tx meta, so, they can't be re-used externally as-is.

What would you like to see?

Follows same design proposal for reusable transformer function for a common model

A few of these private ingestion parsing routines in Horizon exposed as pure(no side effects) functions and package public so they can be re-used by external apps to derive the same 'Offer' data models quickly in their ingestion pipelines.

The presence of public functions for parsing 'Offer' model can further encourage external adoption of proposed streaming ingestion pipeline of which operators within a stream invoke these functions directly to include offers data in their stream processing.

Create new pure function and model for 'Offer' in the shared ingest package. Horizon internal ingestion should not be changed or refactored, later this may be revisited.

  • ingest.HasOffer(op xdr.Operation) bool

  • for all operations that qualified as offer in HasOffer(), pull up the 'details' parsing routines of these operations from transactionOperationWrapper.Details() and fold into -> ingest.DeriveOffer(tx xdr.Transaction, op xdr.Operation) []OfferEvent:

    • managebuy
    • managesell
    • createsell

  • Offer data is modeled as a derived event:

    type OfferEvent struct {    
      Price:                         "8.0000000"
      Amount:                    "89.9342000"
      PriceAssetRatioD:    "1"
      PriceAssetRatioN:    "8"
      BuyingAsset:            "native"
      SellingAsset:             "XCZ:GC2BTZPHJEVOGPW6MJUVUPANWLRGK3JMD26ZMP64ZRMYXBRSW5CIKEW5"
      Timestamp:               "1723153694"
 }

  • Unit test coverage on new functions added to ingest package

Visualization of where this transform function fits in the larger CDP design for data pipeline:

Image

What alternatives are there?

we don't publicize the routines, external apps use horizon code as reference, and will clone from this to their codebase as they choose to re-implement.