Key-value store backend?
ebiiii opened this issue · 0 comments
Currently the interface to the different databases backend is through the StorageManager. It was built assuming SQL databases (mySQL, postgresQL,...), offering functions that takes SQL queries as input and JDBC drivers are doing the connection to the databases.
But if we want to introduce NoSQL backends, like HBASE, or even memcached, the process of understanding the SQL query and performing it is done at the JDBC driver level (like Apache Phoenix project). Having such adapters would work out almost of the box, but when looking at the whole streaming process would introduce a huge and useless overhead.
Indeed, a lot of functions are just building SQL queries to be able to interact with the DB, but if we know that the DB is just a K-V store, and that the query will just need to be parsed in the next function call, why still using SQL internally?
SQL is very handy for using in the VS configuration files and should be kept here, but it should be parsed much sooner and replaced by an abstraction that can be efficiently passed to SQL and NoSQL backends.
This is just a suggestion and it is not clear for me yet at which level this should be implemented (JDBC, StorageManager, SlidingTimeWindows, VSConfig) ?