Keeping the CT server list up to date without restarting certstream-server

[mzpqnxow]

Hello, I was looking through the source to better understand how certstream-server worked and I had a question that I thought you might be able to easily answer

I see that certstream-server uses the metadata that Google maintains when it bootstraps, to get the list of CT server URLs:

certstream-server/lib/certstream/ct_watcher.ex

Lines 22 to 28 in 41c0547

	def start_and_link_watchers(name: supervisor_name) do
	Logger.info("Initializing CT Watchers...")
	# Fetch all CT lists
	ctl_log_info = "https://www.gstatic.com/ct/log_list/v3/all_logs_list.json"
	|> HTTPoison.get!([], @default_http_options)
	|> Map.get(:body)
	|> Jason.decode!

What are your thoughts on how to handle when a new CT server may be added to (or removed from) this list? Do you think this should be entirely up to the operator of certstream-server? Or do you think it would be appropriate to include some mechanism within certstream-server to intelligently detect and handle that case, maybe by polling every 24h?

I see that you use Supervisor as a high-level management mechanism/interface for the workers. Perhaps that offers some clean way to do this, either via a signal from an external process, or via a dedicated worker that could be part of certstream-server, whose job it is to simply monitor for changes in all_logs_list.json, and then either terminate a worker or add a new worker when a CT server has been removed or added to the all_logs_list.json

It's certainly not a major problem to just kick the server over every week, month, etc. via cron (or whatever) to ensure that there's always a fresh set of CT servers being polled. But it would be nice to not have to worry about that

Thanks again for your work on this project (as well as certstream-python) - it has significantly simplified a project of mine which had a very clunky solution in place for trying to track CT data

[hendursaga]

It's certainly not a major problem to just kick the server over every week, month, etc. via cron (or whatever) to ensure that there's always a fresh set of CT servers being polled.

It IS Elixir, is it not? "Let it crash!" 😄