ClickHouse/clickhouse-js

Watch mode query doesn't produce any events

Wain-PC opened this issue · 4 comments

Wain-PC commented

Describe the bug

I have a Live View that updates over time.

I'd like to receive updates for this live view using WATCH query. However, this query just hangs and does not produce any data.

Steps to reproduce

  1. Create a live view
  2. Create client with Live View support
  3. Run WATCH lv EVENTS query

Expected behaviour

Stream produces data events with live view updates.

Code example

CREATE LIVE VIEW lv WITH REFRESH 5 AS SELECT now()
// Create client with live view support
const client = createClient({
  host: 'http://localhost:8123',
  database: 'default',
  clickhouse_settings: {
    allow_experimental_live_view: 1,
  },
});

// Run query
const rows = await client.query({
    query: `WATCH lv EVENTS`,
    format: 'JSONEachRow',
  });

// Get stream and wait for data
  const stream = rows.stream();
  stream.on('data', (rows) => {
    // Data never arrives here
    console.log('Stream update received', rows);
  });

Error log

No errors

Configuration

Environment

  • Client version: 0.0.15
  • Language version: en-US
  • OS: Ubuntu Server 22.04

ClickHouse server

  • ClickHouse Server version: 2.4.5.9
  • ClickHouse Server non-default settings, if any: allow_experimental_live_view: 1
slvrtrn commented

That's interesting; we will check how it works with the HTTP protocol. Thanks for the report.

  • 👍1
slvrtrn commented

@Wain-PC, I wrote a test for this issue (see commit).

It turned out that the response compression was the culprit here. If you turn it off, then the test passes.

For example:

const client = await createTestClient({
  compression: {
    response: false, // WATCH won't work with response compression
  },
  clickhouse_settings: {
    allow_experimental_live_view: 1,
  },
})

Interestingly enough, it also does not work with CURL from the console:

➜  ~ curl -sS "http://localhost:8123/?enable_http_compression=1" \
    -H 'Accept-Encoding: gzip' -d 'WATCH watch_stream_test_4e85058ac42d41d28cbe51aa8eea3cb1 EVENTS
FORMAT JSONEachRow' | gunzip -

(hangs forever until Ctrl + C, while it works fine without compression)

So, as a (maybe temporary) workaround, I suggest disabling the response compression in the client for such queries.

I will come back with an update once it is clear whether it's possible to make WATCH + response compression work.

slvrtrn commented

@Wain-PC, I've got a confirmation: WATCH will not work with response compression due to how LIVE VIEW works in ClickHouse itself. Disabling response compression in the client (as mentioned in the previous message) is the only option here.

Wain-PC commented

Thank you, I've checked live view with compression disabled and it's working fine!

I think this issue is worth noting in the Known Limitations section.
I've created a PR there.

  • ❤️1