[Epic] Solver Slippage V2
Opened this issue · 7 comments
Based on Specification Documentation
We layout a top-down approach plan for this repo. Development will take place in a "Feature Branch" and this is an open PR containing an aggregation of all changes.
Phase 1: Acquiring Internalized Transfers
Note that a Proof of Concept has already been implemented in python here and can be used for reference when reviewing (cc @gentrexha)
- Setup Initial DB Migration (#178)
- Basic Tenderly Web3 Action For Settlement Events (#181)
- Add Models & Parsing Logic to WebHook (#182)
- Parse Trades and Transfers from Settlement Transaction Receipts. #193
- Use Action to Insert SettlementEvents to DB (#185)
- Fetch winning settlement from orderbook API by txHash (#220)
- Simulate Full Call Data from Winning Settlement (#227)
- Parse Simulation Results (#228)
- Parse transfer events from simulated transaction (#229)
- Aggregate transfers with accumulator as "imbalance" (#233)
- Token Imbalance Arithmetic (#234)
- Project Milestone Compute Internalized Imbalance as
Full - Actual(#231) - Insert
InternalizedImbalancesinto DB (#242) - Tie together all functionality as a single workflow to call from within the web-hook (#261)
- Introduce Data dumping ground for unconfirmed transactions (#271)
- Adjust pipeline to account for finalized and unconfirmed transactions (#272)
- Extend Tx Receipt so that Settlement Log index is block-wise not transaction-wise. (#275)
- Mark Processed (#276)
- Deploy to Staging DB (announcement)
- Bug - #281
- Bug - #279
- MAIN BRANCH CONCLUDING PHASE 1 (#184)
Phase 2: Dune Sync
- Obtain a Production DB and Deploy Phase 1 to production (slack)
- Configure auto deployments.
- Inject data into dune community sources. This is part of a different project -- namely dune-sync (cowprotocol/dune-sync#37)
- This project will need to be able to to determine if there is any missing data before sync.
- If something is missing, it should be able to trigger the pipeline "en masse" (for a collection of
txHashes).
Phase 3: Refactor Slippage Query
- Replace this portion of the slippage query with internalized transfer data. (#259)
Additional TODOs & Cleanup
- #226
- #230
- #232
- #235
- #236
- #237
- #246
- #245
- Manual mass application of Phase 1 (i.e. without web-hook trigger) and ensure idempotency. (#244)
- #274
Follow Up Tasks
- Migrate primary functionality of Phase 1 into an AWS lambda and deploy. This will reduce attack surface by not exposing DB credentials to a third party service. Will also give us a gateway to more involved pipeline projects. Prototype exists here
- ... probably more stuff.
Some very helpful Proof of Concept Code can be found
- A Typed Python Implementation of the full pipeline excluding DB writes.
- Components of the Grand Scheme including Tenderly Action that writes to DB & invokes AWS lambda.
I think one open design decision is what data we should store in our database (future data warehouse). I believe @olgafetisova was in favor of ingesting the raw transaction receipts/simulation results and then run the slippage computation based on another script on our database:
- Tenderly Simulation Result --> DB --> Script to compute slippage --> DB (or unmaterialized view)
- Tenderly Simulation Result --> Script to compute slippage --> DB
The latter has the advantage IMO that it would make the script easily reproducible by third parties (all they need is tenderly access, no need to access our DB). On the other hand, having one script ingesting the simulation result and another script computing the slippage based off that splits responsibilities and may allow us more easily replace the tenderly dependency (e.g by our own node simulation) later on. Also, if there are other use cases that would use those simulation results, we wouldn't have to run the simulation multiple times.
Thanks Felix, these are all very good points to consider and the avenue I have been going is definitely the second. I want to point out that you keep mentioning "Script to compute slippage" as if that is what we are doing here... however this project is aimed at computing "internalized token imbalance". If that is not your understanding, then this should be aligned (maybe you want to read the introduction section of the specification). We can easily change gears if necessary, but the goal here currently is to replace the approximation being made in the existing slippage query. Long term, we would transition this into a fully internal slippage computation.
raw transaction receipts/simulation results
I am very much in favour of saving simulations and storing their IDs on our side, but each simulation is a VERY LARGE JSON blob that I don't think we should be keeping on our side (at least not without reducing it by pruning unnecessary parts).
replace tenderly dependency
The project has been written in a way (with an interface TxSimulator -- cf #227) that should already make it easy enough to swap out our TenderlySimulator
you keep mentioning "Script to compute slippage" as if that is what we are doing here...
I was judging from the title of this EPIC ("Solver Slippage v2"), that the final deliverable of this EPIC was a script to compute more accurate solver slippage. On the way there the goal is to pioneer and build the foundation for our data warehouse infra.
Foreseeable Issues & Complications
- This "uninternalizedCallData" which this project requires may not be accessible upon solver-driver colocation. This would essentially kill this project (in ~2-3 months)
- Going back in time to recover historical data will likely result in rate limiting.
Workflow Failure Points
- Each settlement requires a passing simulation to yield "correct" results. When simulation fails? Try
simBlock + 1? Then what? We could treat it as if there were no internalized transfers and "let it slide" - What happens when the array of solutions has length zero?
- what to do when the actual settlement solver does not agree with the winning solver from the competition?
final deliverable of this EPIC was a script to compute more accurate solver slippage
Yes this is correct, but does not mention computing it from our own database. Rather an adaptation of our existing query.
On the way there the goal is to pioneer and build the foundation for our data warehouse infra.
In fact this is already happening. Its just that we are not just gonna start pumping a bunch of bloat into a database right from the start. It will be very small and compact representation of what we need to "compute accurate slippage"
As agreed earlier with Ben, he is going to give us a presentation with the choices that have been already made and the actual state of the project, additionally, he is going to host a discussion session on the open questions so that all the reviewers of the project are aligned on the next steps.