- graph suggestions - select one or more columns and get suggestions for what graphs are possible
- columns sheet shows graphs, with info about each individual column
- contextual command palette - command palette that shows suggestion depending on state of the sheet. E.g. searching "hide" on an active column can help hide that column
- live search
where was i
- download button causing CPU usage to skyrocket
- add more buttons (regex search 2 buttons - enter and escape)
add warning for vimium users
Lighthouse audit revealed several optimizations
design of column dropdown
- Select column
- Histogram of this column
- Histogram of all selected columns
bug
-
apostrophe in "display_name" of demo datasets throws off sql syntax
-
explore
- row numbers (duckdb row_numbers())
- make histogram cover entire row in freq table (was having trouble getting the css right, the color of the td element was taking precedence over color of tr) slightly bigger challenges
- versioning javascript files
- if html is cached for a long time, how will html/js updates happen on client side?
- wrapped_col_indices implementation seems brittle, need a straightforward way of initializing browser state from the server, same with key_cols implementation
- col should be wrapped even if js is not loaded
- use $persist and $scroll provided by alpinejs
- way to embed table in html
- write some blogs with embedded table
make tables immutable? and cache the f out of them pre-release checklist
-
[FH] display large numbers nicely
-
[FM] tooltip on sort sign?
-
[FM] button to open new table with filters
-
[FM] add buttons for toggling width of column with shortcut tooltips
-
[FM] column hover - buttons and [shortcut], and "help"
-
[FM] multi-line cols should remain multi-line after an operation
-
[F] on zoom, table window size shouldn't change
-
[F] all "help" should link to command palette
-
[F] links to previous table to have shorter names with tooltip
-
[F] chrome loading issue? (going back after regex operation keeps the search bar open but out of focus)
-
[F] mouse should hide itself
-
[F] indicator of multi-line col
-
[F] load table into pandas
-
refactor - write tests
-
refactor performOp functions (some take arguments when they should be using self.)
-
refactor, communicate with backend using actions rather than operations (e.g. openCell instead of facetOperation)
-
[BM] think about how to implement caching
-
[B] show useful info like col-types
-
[B] download as parquet
-
[B] col dtypes
todo
-
refactor - move js logic from Alpine.bind into Alpine.data
-
BUG: filtering on NA doesn't work
-
BUG: freq of freq - "num_rows" duplicate name
-
low priority: can hidden cols be key cols? shouldn't matter
-
check the column names using https://github.com/duckdb/duckdb/blob/master/examples/python/duckdb-python.py#L76
-
links to previous tables
- list of previous tables should not change on sort
- browser history can still change
-
changing order of cols
- are these preserved when sharing table?
-
insert values
-
column summary info
-
left,right hover-on-scroll div
-
get as far as possible can with server-side
display message if server responds with 400 code on POST request example of message - "pivot column needs to have less than X values"
check for SQL injection
need to think of a better way in which JS code can construct operations
use color to indicate the kind of table
operations todo
- filter
- pivot
- insert
- distinct counts
- multiple col shift-F
- show keypress hints (for available ops)
- on pressing and holding just Shift, it shows which aggregation functions are applied to which columns
make this run in browser with pyiodide and wasm
rename columns
- if shift-F is done twice, both columns are called
count_star()
frontend-interactivity
- how would column shifting work?
column types
- if we make all of them string, how will we plot floats
- NaN values
- operations on columns
interface for excluding columns
RestrictedPython package https://stackoverflow.com/questions/63160370/how-can-i-accept-and-run-users-code-securely-on-my-web-app
there's an easy way to check the column names in duckdb, and their types https://github.com/duckdb/duckdb/blob/master/examples/python/duckdb-python.py#L76
duckdb-wasm (READ thoroughly) https://duckdb.org/2021/10/29/duckdb-wasm.html
use this for fully-client side vow?
This is a complex issue
- Some table attributes are used for re-creating the table e.g. view, query_params, dbtype
- Other table attributes inform the look of the table e.g. wrapped_col_indices, desc
- the
source
attribute is used to re-create the lineage- there's still no explicit way to find which operation was used to transform older table into current one. This info is implicitly stored in desc
There are also in-memory tables created at the start of the application. Should they be persisted? No Its ok if the master and about table cannot be re-created from storage Only need to make sure that the uid for these tables is mapped to their in-memory objects for lookup WHAT ABOUT COLUMN TABLES????
For now, doing the following: Persist all table attributes, including visual ones Don't persist in-memory tables (i.e. tables where dbtype = 'memory')
It may lead to breaking changes in the future
- when attribute name changes e.g.
desc
->description
- or when attribute value changes e.g.
gta[fil]
->filter
may need to migrate older database to new one, or start from scratch
may have to change the way I'm persisting these Instead of storing UID -> query, source I should store UID -> operation, source That way I'll only have to worry about keeping operations backward compatible, and making sure that the source table implements that operation