vacp2p/research

Roadmap(DST)

kaiserd opened this issue · 3 comments

kaiserd commented

(WIP)

Finalize Wakurtosis & Separate Engineering/Analysis

(All tasks listed in this section have to be finished until 31/07/2023)

Analysis Area (mainly)

  • Finalize Techreport for Waku
    • deliver first version on 16/06/2023
    • describes Wakurtosis
    • covers vacp2p/wakurtosis#108
    • compares simulation results to theoretical analysis
  • Vac research log post based on this Techreport

Engineering Area (mainly)

  • finalize discv5 support in Wakurtosis
    • evaluation in Techreport will cover discv5
  • move Wakurtosis github repo to Vac org
  • separate analysis module out into a separate github repo in Vac org
  • make data available via Prometheus (or single file interface for "end-of-simulation-summary" data)
    • currently analysis scrips scrape data from various sources
    • consolidating most in Prometheus is a first step towards separating analysis and engineering
  • running & fixing simulations (involved process for now); make data available for analysis
  • setup Nomos test environments using Wakurtosis

Next steps: Org

  • hire Test engineer
  • hire Senior Test Analyst

Next steps: Engineering

(starting after 31/07/2023)

  • develop new simulation framework (using Kubernetes?)

    • main goal
      • scale to 10k gossipsub-only nodes, bandwidth evaluation only
      • scale to 10k Waku relay-only nodes, bandwidth evaluation only

  • prepare Wakurtosis for use in CI, liaise with test analyst (to be hired)

  • goal: API for regression test engineer

  • extend simulation framework

    • export further data (beyond bandwidth)
    • support other nodes (e.g. Nomos, Codex)
    • support more features of nodes (e.g. further Waku protocols)

  • bundle simulation data as data-set provided to the DST analysis area

    • data-sets can be used by other interested parties, too
    • this completes separating Engineering and Analysis, and defines are concise interface between the areas

  • add QoS parameters

    • packet loss
    • ...

Next steps: Analysis

(starting after 31/07/2023)

  • #194

  • Topology analysis (incl. topologies established by discv5)

  • theoretical analyses (to be specified)

    • (mainly work on analyses not depended on Engineering until engineering can provide data sets)

Next steps: Simulations

(starting after 31/07/2023)

Next steps: Regression Testing

(starting after 31/07/2023)

fryorcraken commented

Few things that seem to be missing from this roadmap (or is it tracked somewhere else?).

  • Testing Waku Store at scale (initially defined here: waku-org/pm#12 (comment))
  • Testing Waku Filter and Light push (needed for Status Mobile)
jm-clius commented

For Waku Store, some test requirements are:

  • For a single Store instance in isolation - functional testing for non-regression:
    • use RFC 13 as guideline to write test scenarios covering all specified Store behaviour. This involves time-based queries, content queries and combinations of the two.
    • expand existing library of unit tests to ensure full coverage of RFC-specified Store behaviour
    • perhaps in addition: extract query use cases from real users (specifically Status) to ensure we have tests covering typical query structures, etc. and to reinforce back to users how to use Store in specified manner
  • Store nodes in the Waku Network. Assume a Waku Network where all nodes act as both relayers and store nodes. Assume that all nodes store n days of messages and are servicing m queries per minute (these parameters can be decided upon based on real data from Status Communities)
    • What is the impact on message throughput/latency for different network sizes if Store is enabled?
    • What is the impact on node memory and CPU usage if Store is enabled?

cc @kaiserd @fryorcraken

  • 👍1
kaiserd commented

Related not yet completed milestones have moved to the Vac DST roadmap