Starting rdb without a tickerplant

Question

Starting rdb without a tickerplant

Closed this issue 4 years ago · 5 comments

While it might not be a conventional approach, at the moment if you try to start an rdb process without supplying a valid tickerplant type (i.e. tickerplanttypes:` in the rdb settings), it results in the rdb getting stuck in a loop where it attempts to subscribe/connect to something that doesn't exist (and incorrectly says subscription successful!).
I think it should be possible to start the rdb with no tickerplant given, such that the subscribe step is skipped when tickerplanttypes is null. It shouldn't cause the process to essentially hang while it keeps trying to connect to a non-existent tickerplant.
In terms of EOD, in this case the rdb could save down all tables in the top namespace (except those in ignorelist), or you could have a more sophisticated system for adding tables to a writedown list (since there would be no .rdb.subtables).
This issue is probably also applicable to the wdb or similar processes that are set up to connect to a tickerplant.

Answer 1 · 2018-06-27T11:52:30.000Z

Hi Andrea,

It would be useful to have a better sense of your use case for the RDB in this context.

I suppose the concern is whether or not you need at any point to be able to connect and subscribe to the tickerplant, or if you'd just like the option to operate within the RDB without ever connecting (and with the RDB not hanging because of that) to deal with non-streaming data for example.

At the moment, we're looking to implement a change across TorQ such that processes can operate without necessarily connecting/subscribing to the processes that they would normally do so to, which may suffice for your needs.

Thanks,
Joseph

Answer 2 · 2018-06-28T05:47:45.000Z

Hi Joseph, Thanks for getting back to me so quickly! In the setup I am using there is no tickerplant at all as the RDB is getting data through other means. So the change you are implementing sounds like it would be sufficient - I’d just like to be able to start the RDB without ever connecting or subscribing to a tickerplant. Please let me know if you have any further questions, I’m happy to help! Thanks, Andrea

…

On 27 Jun 2018, at 9:52 pm, JedwardsEdwards ***@***.***> wrote: Hi Andrea, It would be useful to have a better sense of your use case for the RDB in this context. I suppose the concern is whether or not you need at any point to be able to connect and subscribe to the tickerplant, or if you'd just like the option to operate within the RDB without ever connecting (and with the RDB not hanging because of that) to deal with non-streaming data for example. At the moment, we're looking to implement a change across TorQ such that processes can operate without necessarily connecting/subscribing to the processes that they would normally do so to, which may suffice for your needs. Thanks, Joseph — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#100 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/Amk4guNIlFcaqSPnq5z2krK2PsxY0Qalks5uA3IBgaJpZM4U5Ar3>.

Answer 3 · 2018-06-28T08:55:41.000Z

Hi Andrea,

We've gone over this issue again our end, and have reached a decision for now to leave the RDB startup code as is for the time being. I think it comes down to the functionality of the process, and that it in using the RDB as intended as a real-time database for streaming data, it doesn't really make sense for the RDB to start up without a connection to its data source, the tickerplant.

While this process could be expanded to serve a number of purposes, I think it is clearer and cleaner to set up a similar, but distinct process to handle the loading of data through other means. Putting something like this together would be fairly straighforward, using parts of the RDB code as a template, and this is what I suggest to fit your needs. You'd just have to set this up as a vanilla TorQ process, without any connection requirements, load up your schema, and then start loading your data.

We will be fixing a bug in the near future such that if the tickerplant connection isn't well defined the RDB process will exit rather than sitting in the loop.

Just let us know if you have any other questions about setting up a process like this,

Thanks,
Joseph

Answer 4 · 2018-06-28T23:28:06.000Z

Hi Joseph, No problem, I understand. As I said it’s not a conventional use of an RDB but I definitely agree that the bug where it gets stuck in a loop should be addressed. I already have a solution for my use case but thank you for the suggestions! Kind regards, Andrea

…

On 28 Jun 2018, at 6:55 pm, JedwardsEdwards ***@***.***> wrote: Hi Andrea, We've gone over this issue again our end, and have reached a decision for now to leave the RDB startup code as is for the time being. I think it comes down to the functionality of the process, and that it in using the RDB as intended as a real-time database for streaming data, it doesn't really make sense for the RDB to start up without a connection to its data source, the tickerplant. While this process could be expanded to serve a number of purposes, I think it is clearer and cleaner to set up a similar, but distinct process to handle the loading of data through other means. Putting something like this together would be fairly straighforward, using parts of the RDB code as a template, and this is what I suggest to fit your needs. You'd just have to set this up as a vanilla TorQ process, without any connection requirements, load up your schema, and then start loading your data. We will be fixing a bug in the near future such that if the tickerplant connection isn't well defined the RDB process will exit rather than sitting in the loop. Just let us know if you have any other questions about setting up a process like this, Thanks, Joseph — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#100 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/Amk4gssulCp9ab5u5tMmely0WiwSI5wEks5uBJoPgaJpZM4U5Ar3>.

Answer 5 · 2019-11-12T10:48:33.000Z

We have recently had the same issue @JedwardsEdwards

We are now escaping the subscribe function if (::)~.rdb.subscribeto or .ctp.subscribeto etc.
For our purposes, we wanted to initialise the connection to the tickerplant, but as each of our rdb's are essentially sandboxes, the data required varies and therefore we initialise subscriptions from within the process after launch for only the specific data streams required (possibly none from tickerplant, and subscribing to a derived stream being published from a chainedtp).