Thread blocking

Hey,

I've been experimenting with the app and encountered the following issue:

Although the app is mostly async, there are several blocking calls:

hasura-metric-adapter/metrics/src/collectors/mod.rs

Line 27 in 0ab0c6e

    
           match termination_rx.recv_timeout(std::time::Duration::from_millis(cfg.collect_interval)) {

hasura-metric-adapter/metrics/src/logreader.rs

Line 33 in 0ab0c6e

match termination_rx.recv_timeout(Duration::from_millis(sleep_time)) {

hasura-metric-adapter/metrics/src/logreader.rs

Line 51 in 0ab0c6e

match watch_receiver.recv_timeout(Duration::from_millis(sleep_time)) {

When execution reaches any of these calls, it will block the thread.

Let's take a look at the first example (inside run_metadata_collector function):

This stage is reached after every iteration of the metadata collector.
The thread is being blocked for the duration of cfg.collect_interval, which is 15s be default.

run_metadata_collector is joined with the read_file async branch:

hasura-metric-adapter/metrics/src/main.rs

Lines 162 to 166 in 0ab0c6e

    
           let res = tokio::try_join!( 
        
               webserver(&config), 
        
               logreader::read_file(&config.log_file, &metric_obj, config.sleep_time, &terminate_rx), 
        
               collectors::run_metadata_collector(&config, &metric_obj, &terminate_rx) 
        
           );

Now the read_file is unable to continue, until the thread is unblocked by run_metadata_collector.
This results in logs being processed by read_file in short bursts every 15s and not as soon as they arrive.

The effect this has on the app's behavior when hasura generates lots of logs depends on where the logs are redirected to:

normal file - the metrics adapter can't keep up and reports the logs metrics with delay, giving incorrect representation of the system state.
named pipe - as soon as the pipe's buffer is filled, it blocks the writer process (hasura) until the reader process (metrics adapter) frees the buffer by reading more lines.

The named pipe case is the most troublesome, as it results in hasura requests being throttled. I think it is the cause of this issue: #31.

Reproduction case:

Clone the repo.
Start docker compose.
Go to the Data tab in the hasura console (http://localhost:8080/console/data/)
Create an empty table (e.g. table test with single column id).
Go to the API tab in the hasura console (http://localhost:8080/console/api/api-explorer).
Prepare a simple request to the table created above in the GraphiQL, e.g.:

query MyQuery {
  test {
    id
  }
}

Open the browser network tab and start constantly sending requests with Ctrl+Enter.
After a short while the requests will stop resolving in the network tab as the named pipe buffer is filled with unprocessed logs due to metric adapter's thread being blocked.
At this point the whole hasura console will stop working correctly and will get back to normal only when the metric adapter catches up with the accumulated logs.

@afitzek I've created a PR with the proposed fix, please take a look when you have time: #38

@dddenis Thank you very much for the very detailed explanation and the nice PR!

	let res = tokio::try_join!(
	webserver(&config),
	logreader::read_file(&config.log_file, &metric_obj, config.sleep_time, &terminate_rx),
	collectors::run_metadata_collector(&config, &metric_obj, &terminate_rx)
	);