Overlays and multicharts UX: follow ups to #420 (and friends)

Question

Overlays and multicharts UX: follow ups to #420 (and friends)

Opened this issue 2 years ago · 0 comments

The UI follow up to #414, it was originally a PR #420 but ended up being
a sequence of smaller (factored) PRs which landed finishing up with #455
📚

Hopefully this is the final core-UX feature you always wanted 🏄🏼

mult-instrument overlayed real-time and historical data feeds with
simultaneous interaction and "current symbol" selectable order mode
control

To run and test the UI:

piker chart btcusdt.binance ethusdt.binance xmrusdt.binance

This should (currently) spawn a chart with all real-time quote feeds and history overlayed on both the fast and slow chart and boot using the method=loglin_ref_to_curve' overlay method by default.

First draft feature set included:

The first MVP for overlay charts lands in #455 and contained a plethora of precursor PRs including:

full multi-fqsn support in our data feed layer with multiple symbols per brokerd feed (#414)
new samplerd daemon introduced to conduct singleton sampler step event synchronization across multiple flow-actor processors (#440)
mega rework to the pre-graphics ShmArray -> QPainterPath rendering pipeline by adding a formalized IncrementalFormatter layer which allows incremental update (and cache) of source data to pre-graphics formats suitable for minimal conversion to a painter path (#447)
internally adjusted Axis sticky label APIs (moved to lower level PlotItem in #448)
multi-symbol input to the real-time graphics system(s) supporting first draft of multicharts (#449)
epoch-time index alignment of multiple time series and a graphics-layer driven "data viz" sub-sys (#451)
drastically simplified L1 labels to enable tight (horizontal) stacking of y-axis co-domains (#452)
initial multichart latency tuning (#453) through reworks of the display loop's:
- maxmin() y-ranging and better support for only draw last uppx pixel column style
- only update from last framed (by type) tick
- distribution of throttle rate uniformly over number of input fqsns
- better Viz API pipeline usage to speed up interaction handlers
- pre-graphics 1-d data caching and use in only draw last uppx pixel column style updates
#455 algorithm which allows sanely viewing multiple time series with co-domains normalized via a returns transform:
- includes support for incremental yrange update from display loop via inputs to ChartView.interact_graphics_loop()
- removal of all Qt.Signal stuff from view box handlers instead opting to call our .interact_graphics_cycle() manually from mouse event callbacks
- better leverage of new Viz update methods for faster pipeline updates from incremental quote driven updates in the .ui._display loop.
- support for multiple overlay methods as finalized in this commit:
```
method: Literal[
    'loglin_ref_to_curve',
    'loglin_ref_to_first',
    'mxmn',

] = 'loglin_ref_to_curve',
```

Init basic funtionality / usage:

basic initial usage is to start a chart with multiple input symbols:
piker chart btcusdt.binance ethusdt.binance xmrusdt.binance
overlay method control for how to align overlays to one another.
L1 labels can be simplified
- we decided to make them overlay on their respective y-axis
how should last datum graphics be offset
- currently all last datums are time step aligned.

History view

curves need to be epoch time aligned instead shm index aligned
since obviously it will result in non-time aligned data sets as it
is now using the 'index' basis..
fix pg.LinearRegion item to show all background overlay curves transparently.

Graphics loop

use single Feed.open_multi_steam() for the update task and have it look up appropriate graphics update apis per symbol-quote
refactor a bunch of the charting/graphics apis to mostly to remove
the global-ish ChartPlotWidget instance and API calls, instead
moving all this state into Flow / Flume items and possibly add
a Viz type to replace the Flow and instead make the Flow
a table of associated src data -> procesor pipelines?
- more to write up as individual subtasks here

Search UI/UX:

make search results and caching work with multi-symbol selections by the GodWidget
all selected symbols should be shown in fqsn form in the window title bar

Follow up ToDo task list after the last PR (#455) lands above:

These include outstanding todo features which are still in discussion / research.

Charting graphics layer: flows mgmt, backendy controls, viewlists, schema:

(Edit-able) list of rt-feeds on chart: the ViewList:

more or less very similar to the legacy idea of loading and saving
watchlists but instead is a more sophisticated abstraction: a list
of trade-able asset pairs that maps to a display of rt-data-flows
shown overlayed in a single chart view
obviusly a declarative syntax much like you'd expect for
a legacy watchlist but with support for additional metadata:
[{'btcusdt.binance: {'pin_to': 'qqq.nasdaq.ib'}}, 'qqq.nasdaq.ib']
or something or other.. (we need to figure out the generic
data-structure syntax form(s)
a way to edit multiple lists within the chart UI using (probably)
vi-like controls similar to those in the ranger file manager
except for moving entries between multiple stacked view lists in the
search pane.

bugs^

current cachelist is not in same order as that passed on CLI?

common axes for common dst-asset:

how to support same-dst but cross-broker markets, eg. xbtusdt.kraken and btcusdt.binance
- should we use a single axis?
- how to determine this automatically?
  - Do a in the same range calc ?
  - have some kinda of adhoc asset name mapping?

OHLC chart overlays:

for ohlc main chart overlays how should we do coloring?
- currently we pull from a sequence of colors in .ui._display but
  we should likely make this configurable, possibly through the UI,
  definitely through a config file.
how to handle overlaid (levered) derivatives where the %returns
are substantially larger due to the lever profile?
- it might make sense in these cases to instead use a "leverage
  and delta adjusted" scalar for the opt contract so that you can see
  a "somewhat linear" translation of the deriv price vs. the underlying?
  - the issue is that the call option is clearly causing the bitcoin
    move to be mega squashed in the y-range.
L1 labels were simplified (we decided to make them stack on
respective y-axis):
- only show (humanized) size on bid/ask not price?
in terms of last datum bars, could they be placed in some kind
of no-exactly-aligned (reverse) order of the y-axis from
left-to-right?
des-synced last bar issue on 1Min chart seemingly due to startup
races where there are N > 2 overlay curves from diff providers?
- is this something we can fix in the .data.feed shm allocation
  logic?

FSP (subcharts)

should fsps like vlm which require a subchart be overlayed or only
the currently selected shown by default?
- we can make this configurable and maybe offer a sidepane drop down
  showing all combos and/or a toggle set of symbols from the overlay
  set?
- can we offer a pair selector to make it possible to show the fsp
  subchart for any market (in the viewlist) not currently selected
  on the OHLC chart(s)?
  - this would probably help perf issues when using binance 😂

=> bugs^

we still have some strange issue on our (vlm) FSP charts where
when multiple overlays are being backfilled the chart will get all
mucked up with strange lines from offscreen connecting to every step?
(TODO: PUT PIC HERE from @guilledk!)

Widget UIs: panes and controls, search..:

UI for "pair selection"

UI for overlay-curve config:

support config (meta) data to choose the pin-method (to
.ui.view_mode.overlay_viewlists()) as well as parameters for each
mode if supported (eg. 'loglin_ref_to_curve' can take a target curve
to pin to)
unify this with the current "feed config" controls: eg. throttle
rate, shm buffer sizing, host discovery, what daemon to use as
source.. etc?

Search UI/UX:

what viewlist(s) should be loaded by default?
- obvs make it configurable
- how lazily should we load the rt feed for each?
a new result should be added to existing viewlist?
- obvs make this configurable through toml file.
- how would this play with the idea of an alt-search for
  derivatives?
how to open/start a new view list?
how to search through existing viewlists
- if you pick a result from one do you load the whole thing?

=> bugs^:

current results order is not the same as CLI input order..

better distinction of pairs on axes?

draw little line from each last label to curve?
some kind of label system to easily know which curve is which feed?
- last label should include fqsn?
- each y-axis should have fqsn?

Order mode

order mode client multi-market API support:

allow passing a fqsns: list[str] to open_order_mode() and
having this setup the client loop and the ems tasks to use
feed.open_feed() with the same fqsns passed through?
- what should be the handle delivered back? a table of clients
  or should we add some kind of client-selectable API already that
  strat code can (eventually) use for easy arbing?

selection oriented UX, show each mkt's EMS state by axis:

pp line arrow-marker should sit on the specific y-axis
- depending on "selected symbol" only show that pp?
how/when to show all pps?
- show active order summary markers (for those not in view) on each sub-yaxis
- maybe only pinned arrow at ppu level always?
support submitting orders only for current selection:
- eg. alert submitted should show on only that axis's y-domain
- cc cancelling should only cancel current instrument's orders?
- tie this in with the viewlist / search result pane, show
  active orders indicator of some sort?