Saving and Visualizing

[davisv7]

I have a few ideas for how to generate and store the graphs.

To keep track of change over time, we could:

• have a separate git repo for the graph data and have it as a submodule to this one, or
• we could use a graph database.

For generating the graph, make a request to https://1ml.com/visual/networkdata or scrape the full network and push it to the repo/save in db. A program hosted on a free https://pythonanywhere.com server can make the request and push it where it needs to be.

If we use python notebooks, this would be a good visualizer: https://mode.com/blog/python-data-visualization-with-plotly/

[nullcount]

https://1ml.com/visual/networkdata is just for visualization. It contains less information (like fees) than a regular lncli describegraph. The downside is that lncli only works when run from a lightning node that is connected to the network.

However, using snapshot.py --remote-write with cron, I have essentially created an API at https://example.com/graph.json using my own server. The data at that url is updated nightly.

Good tip with plotly. I'm open to a notebook integration once we have a problem that would benefit from it.

[nullcount]

I can write 2 scripts:

snapshot + recover

snapshot.sh

get new snapshot
run it thru jq -> sort, pretty, strip last_updated
save diff/patch from last snapshot

recover.sh

Snap = patch of snapshot desired
Starting at genesis, apply patches chronologically until Snap applied

[nullcount]

We are now saving snapshots into a tar archive. 80-90% space reduction.