frictionlessdata/data-quality-dashboard

Reload data when database changes

Closed this issue · 3 comments

pwalsh commented

Description

Notice that the database for the app is loaded when the app initialises, and saved in memory:

(unless caching is disabled)

This is good, and makes it really nice to work with the database from a github repo. However, it means that the app does not know if the database changes, unless the app itself is restarted.

There are several possible solutions. A simple one is to make a single request to check if the database changed. A possible implementation is explained below. Alternatives are possible if a simpler solution can be found.

Tasks

  • Add a last_modified field to instance.json of the database. The instructions for creating a database should explain to add a timestamp here when a new database is committed (can automate this perhaps, but not necessary now)
  • Always check instance.json and if the timestamp has changed, always load the instance.json and the full database files, effectively flushing and rebuilding cache
georgiana-b commented

I am a bit confused by this if clause: https://github.com/okfn/data-quality-dashboard/blob/master/app/backend/middlewares.js#L8. The !cacheData part means the block will execute when cacheData is undefined and when cacheData is false . This means data will be cached even when cacheData = false in the config. Am I missing something? If not, is this ok or should it get cached only when it's undefined?

pwalsh commented

@georgiana-b fix it so that we (re)load the instance data when it is currently empty, and if cacheData is falsy.

pwalsh commented

FIXED.