Socket closed

Question

Socket closed

thiagohora opened this issue 2 months ago · 9 comments

thiagohora commented 2 months ago

Describe the bug

This happens intermittently every now and then.

Expected behaviour

Error when processing batch of inserts, the list doesn't contain more than 1000 items.

Code example

var statement = connection.createStatement(INSERT);

for (Iterator<Item> iterator = items.iterator(); iterator.hasNext();) {
    var item = iterator.next();

    statement.bind("id", item.id())
            .bind("itemId", datasetId)
            .bind("input", item.input().toString())
            .bind("source", item.source().getValue());

    if (iterator.hasNext()) {
        statement.add();
    }
}

statement.fetchSize(items.size());

return statement.execute();
}

Error log

! java.net.ConnectException: Socket closed
! at com.clickhouse.client.http.ApacheHttpConnectionImpl.post(ApacheHttpConnectionImpl.java:276)
! at com.clickhouse.client.http.ClickHouseHttpClient.send(ClickHouseHttpClient.java:195)
! at com.clickhouse.client.AbstractClient.sendAsync(AbstractClient.java:161)
! at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:273)
! ... 4 common frames omitted
! Causing: com.clickhouse.client.ClickHouseException: Socket closed
! at com.clickhouse.client.ClickHouseException.of(ClickHouseException.java:149)
! at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:275)
! at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
! at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
! at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
! ... 1 common frames omitted
! Causing: reactor.core.Exceptions$ReactiveException: com.clickhouse.client.ClickHouseException: Socket closed
! at reactor.core.Exceptions.propagate(Exceptions.java:408)
! at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:101)
! at reactor.core.publisher.Mono.block(Mono.java:1728)

Configuration

Environment

Client version: 0.6.4
Language version: Java 21
OS: MAC

ClickHouse server

ClickHouse Server version: :23.8.15.35-alpine

Answer 1 · 2024-08-28T10:26:27.000Z

Any update?

Answer 2 · 2024-08-30T20:13:54.000Z

Good day, @thiagohora !
Is it a 0.6.2 version of the client (issue mentions 0.7.2)?
Is there any other logs from application related to the client?
Where is server running?
Is there anything on server side? Can you please provide a part of server log while the time of a failure?

Thanks!

Answer 3 · 2024-09-02T08:32:34.000Z

Sorry, 0.6.4.

There are not many application-related logs, to be honest. The server is running on docker: 23.8.15.35-alpine

Answer 4 · 2024-09-02T19:48:34.000Z

2024.09.02 19:41:24.079130 [ 1 ] {} <Warning> Context: Linux transparent hugepages are set to "always". Check /sys/kernel/mm/transparent_hugepage/enabled
2024.09.02 19:41:24.079200 [ 1 ] {} <Warning> Context: Delay accounting is not enabled, OSIOWaitMicroseconds will not be gathered. You can enable it using `echo 1 > /proc/sys/kernel/task_delayacct` or by using sysctl.
2024.09.02 19:41:24.079261 [ 1 ] {} <Warning> Application: Integrity check of the executable skipped because the reference checksum could not be read.
2024.09.02 19:41:24.083734 [ 1 ] {} <Error> Application: Disabling cgroup memory observer because of an error during initialization: Code: 107. DB::Exception: Cannot find cgroups v1 or v2 current memory file. (FILE_DOESNT_EXIST), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000bd62308
1. DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x0000000007744734
2. DB::CgroupsMemoryUsageObserver::CgroupsMemoryUsageObserver(std::chrono::duration<long long, std::ratio<1l, 1l>>) @ 0x000000000be5bb60
3. DB::Server::main(std::vector<String, std::allocator<String>> const&) @ 0x000000000beb925c
4. Poco::Util::Application::run() @ 0x0000000012d54ddc
5. DB::Server::run() @ 0x000000000beae190
6. Poco::Util::ServerApplication::run(int, char**) @ 0x0000000012d5e084
7. mainEntryClickHouseServer(int, char**) @ 0x000000000beaa7f0
8. main @ 0x0000000007735dd8
9. ? @ 0x0000000000020e10
 (version 24.3.8.13 (official build))
2024.09.02 19:41:24.132162 [ 1 ] {} <Warning> Application: Listen [::]:9009 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 24.3.8.13 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2024.09.02 19:41:24.193879 [ 1 ] {} <Warning> Access(local_directory): File /var/lib/clickhouse/access/users.list doesn't exist
2024.09.02 19:41:24.193891 [ 1 ] {} <Warning> Access(local_directory): Recovering lists in directory /var/lib/clickhouse/access/
2024.09.02 19:41:24.214186 [ 1 ] {} <Warning> Application: Listen [::]:8123 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 24.3.8.13 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2024.09.02 19:41:24.214241 [ 1 ] {} <Warning> Application: Listen [::]:9000 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 24.3.8.13 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2024.09.02 19:41:24.214292 [ 1 ] {} <Warning> Application: Listen [::]:9004 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 24.3.8.13 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2024.09.02 19:41:24.214371 [ 1 ] {} <Warning> Application: Listen [::]:9005 failed: Poco::Exception. Code: 1000, e.code() = 0, DNS error: EAI: Address family for hostname not supported (version 24.3.8.13 (official build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2024.09.02 19:42:30.437928 [ 48 ] {456460b8-2cca-49e6-acb7-b2250ce1057d} <Error> executeQuery: Code: 60. DB::Exception: Unknown table expression identifier 'default.DATABASECHANGELOGLOCK' in scope SELECT COUNT(*) FROM default.DATABASECHANGELOGLOCK. (UNKNOWN_TABLE) (version 24.3.8.13 (official build)) (from 172.17.0.1:57044) (in query: SELECT COUNT(*) FROM default.DATABASECHANGELOGLOCK FORMAT TabSeparatedWithNamesAndTypes), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000bd62308
1. DB::Exception::Exception<String const&, String>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String>::type>, String const&, String&&) @ 0x0000000007c357cc
2. DB::(anonymous namespace)::QueryAnalyzer::resolveQuery(std::shared_ptr<DB::IQueryTreeNode> const&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000000f5936e0
3. DB::QueryAnalysisPass::run(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::Context const>) @ 0x000000000f58ee6c
4. DB::QueryTreePassManager::run(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000000f58d6e4
5. DB::(anonymous namespace)::buildQueryTreeAndRunPasses(std::shared_ptr<DB::IAST> const&, DB::SelectQueryOptions const&, std::shared_ptr<DB::Context const> const&, std::shared_ptr<DB::IStorage> const&) (.llvm.5642323065489652412) @ 0x000000000f7bfd24
6. DB::InterpreterSelectQueryAnalyzer::InterpreterSelectQueryAnalyzer(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000000f7bdd3c
7. std::unique_ptr<DB::IInterpreter, std::default_delete<DB::IInterpreter>> std::__function::__policy_invoker<std::unique_ptr<DB::IInterpreter, std::default_delete<DB::IInterpreter>> (DB::InterpreterFactory::Arguments const&)>::__call_impl<std::__function::__default_alloc_func<DB::registerInterpreterSelectQueryAnalyzer(DB::InterpreterFactory&)::$_0, std::unique_ptr<DB::IInterpreter, std::default_delete<DB::IInterpreter>> (DB::InterpreterFactory::Arguments const&)>>(std::__function::__policy_storage const*, DB::InterpreterFactory::Arguments const&) (.llvm.5642323065489652412) @ 0x000000000f7c1808
8. DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x000000000f7627d0
9. DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000000fb2f124
10. DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000000fb33744
11. DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x0000000010953ff8
12. DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x0000000010957de8
13. DB::HTTPServerConnection::run() @ 0x00000000109c4284
14. Poco::Net::TCPServerConnection::start() @ 0x0000000012d47de4
15. Poco::Net::TCPServerDispatcher::run() @ 0x0000000012d48db0
16. Poco::PooledThread::run() @ 0x0000000012e68a94
17. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000012e66e4c
18. start_thread @ 0x0000000000007624
19. ? @ 0x00000000000d162c

2024.09.02 19:42:30.438036 [ 48 ] {456460b8-2cca-49e6-acb7-b2250ce1057d} <Error> DynamicQueryHandler: Code: 60. DB::Exception: Unknown table expression identifier 'default.DATABASECHANGELOGLOCK' in scope SELECT COUNT(*) FROM default.DATABASECHANGELOGLOCK. (UNKNOWN_TABLE), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000bd62308
1. DB::Exception::Exception<String const&, String>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<String>::type>, String const&, String&&) @ 0x0000000007c357cc
2. DB::(anonymous namespace)::QueryAnalyzer::resolveQuery(std::shared_ptr<DB::IQueryTreeNode> const&, DB::(anonymous namespace)::IdentifierResolveScope&) @ 0x000000000f5936e0
3. DB::QueryAnalysisPass::run(std::shared_ptr<DB::IQueryTreeNode>&, std::shared_ptr<DB::Context const>) @ 0x000000000f58ee6c
4. DB::QueryTreePassManager::run(std::shared_ptr<DB::IQueryTreeNode>) @ 0x000000000f58d6e4
5. DB::(anonymous namespace)::buildQueryTreeAndRunPasses(std::shared_ptr<DB::IAST> const&, DB::SelectQueryOptions const&, std::shared_ptr<DB::Context const> const&, std::shared_ptr<DB::IStorage> const&) (.llvm.5642323065489652412) @ 0x000000000f7bfd24
6. DB::InterpreterSelectQueryAnalyzer::InterpreterSelectQueryAnalyzer(std::shared_ptr<DB::IAST> const&, std::shared_ptr<DB::Context const> const&, DB::SelectQueryOptions const&, std::vector<String, std::allocator<String>> const&) @ 0x000000000f7bdd3c
7. std::unique_ptr<DB::IInterpreter, std::default_delete<DB::IInterpreter>> std::__function::__policy_invoker<std::unique_ptr<DB::IInterpreter, std::default_delete<DB::IInterpreter>> (DB::InterpreterFactory::Arguments const&)>::__call_impl<std::__function::__default_alloc_func<DB::registerInterpreterSelectQueryAnalyzer(DB::InterpreterFactory&)::$_0, std::unique_ptr<DB::IInterpreter, std::default_delete<DB::IInterpreter>> (DB::InterpreterFactory::Arguments const&)>>(std::__function::__policy_storage const*, DB::InterpreterFactory::Arguments const&) (.llvm.5642323065489652412) @ 0x000000000f7c1808
8. DB::InterpreterFactory::get(std::shared_ptr<DB::IAST>&, std::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) @ 0x000000000f7627d0
9. DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x000000000fb2f124
10. DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::shared_ptr<DB::Context>, std::function<void (DB::QueryResultDetails const&)>, DB::QueryFlags, std::optional<DB::FormatSettings> const&, std::function<void (DB::IOutputFormat&)>) @ 0x000000000fb33744
11. DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x0000000010953ff8
12. DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x0000000010957de8
13. DB::HTTPServerConnection::run() @ 0x00000000109c4284
14. Poco::Net::TCPServerConnection::start() @ 0x0000000012d47de4
15. Poco::Net::TCPServerDispatcher::run() @ 0x0000000012d48db0
16. Poco::PooledThread::run() @ 0x0000000012e68a94
17. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000012e66e4c
18. start_thread @ 0x0000000000007624
19. ? @ 0x00000000000d162c
 (version 24.3.8.13 (official build))
2024.09.02 19:42:34.208296 [ 739 ] {6afe2942-8f65-4684-bcdb-a0d3fdcd29f1} <Error> executeQuery: Code: 131. DB::Exception: String too long for type FixedString(36): while converting source column dataset_id to destination column dataset_id: while executing 'FUNCTION _CAST(dataset_id :: 0, FixedString(36) :: 11) -> _CAST(dataset_id, FixedString(36)) FixedString(36) : 10'. (TOO_LARGE_STRING_SIZE) (version 24.3.8.13 (official build)) (from 172.17.0.1:58138) (in query:  INSERT INTO dataset_items ( id, dataset_id, source, trace_id, span_id, input, expected_output, metadata, created_at, workspace_id, created_by, last_updated_by ) SELECT new.id, multiIf( LENGTH(CAST(old.dataset_id AS Nullable(String))) > 0 AND notEquals(old.dataset_id, new.dataset_id), leftPad('', 40, '*'), LENGTH(CAST(old.dataset_id AS Nullable(String))) > 0, old.dataset_id, new.dataset_id ) as dataset_id, new.source, new.trace_id, new.span_id, new.input, new.expected_output, new.metadata, multiIf( notEquals(old.created_at, toDateTime64('1970-01-01 00:00:00.000', 9)), old.created_at, new.created_at ) as created_at, multiIf( LENGTH(old.workspace_id) > 0 AND notEquals(old.workspace_id, new.workspace_id), CAST(leftPad('', 40, '*') AS FixedString(19)), LENGTH(old.workspace_id) > 0, old.workspace_id, new.workspace_id ) as workspace_id, if( LENGTH(old.created_by) > 0, old.created_by, new.created_by ) as created_by, new.last_updated_by FROM ( SELECT '0191b442-acb3-70a6-b5e5-67b7d0ea3da5' AS id, '0191b442-aca8-73c1-b49c-b0d9d38fe086' AS dataset_id, 'span' AS source, '0191b442-acb3-70a6-b5e5-67b7d0ea3da3' AS trace_id, '0191b442-acb3-70a6-b5e5-67b7d0ea3da4' AS span_id, '{"jklSBNC4oJ":"pbTAuMhh0r","vvYmheCrDx":"N2af7OUexl","Y3QAFOW0ca":"bFFOh6yY1b","p7xBvFyHkB":"HbCxvWT5v0","GzOSiB1R0N":"cS1Tl9XFg6"}' AS input, '{"jYpXxbY5e5":"u5njWT6MVf","amjmwouOnI":"bAznwxHXuA","m294IaIhjA":"sohOs6ZUQe","ROax1buF66":"tfQ9RRqgYG","uEFS22luhd":"6A8iRdsu4z"}' AS expected_output, '{"v8O6fcbpA8":"95Iek8OG5K","S8OXo3zY53":"UL1M1ru84y","re5w1DvKu7":"BEXHB0pCE5","50Lk5arhev":"Bq1VBHCUG0","HuuoZUU0W9":"2pY0MhSDuk"}' AS metadata, now64(9) AS created_at, 'e3577c3d-5cca-497b-b6be-b1b2f4cfa334' AS workspace_id, '0866bfa8-d0e6-4ea9-bfce-633c787b8394' AS created_by, '0866bfa8-d0e6-4ea9-bfce-633c787b8394' AS last_updated_by ) AS new LEFT JOIN ( SELECT * FROM dataset_items WHERE id = '0191b442-acb3-70a6-b5e5-67b7d0ea3da5' ORDER BY last_updated_at DESC LIMIT 1 BY id ) AS old ON old.id = new.id ; ), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000bd62308
1. DB::Exception::Exception<String>(int, FormatStringHelperImpl<std::type_identity<String>::type>, String&&) @ 0x0000000007739030
2. COW<DB::IColumn>::immutable_ptr<DB::IColumn> DB::FunctionToFixedString::executeForN<(DB::ConvertToFixedStringExceptionMode)0>(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, unsigned long) @ 0x000000000796a8c8
3. DB::(anonymous namespace)::ExecutableFunctionCast::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x0000000007803d30
4. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000e48436c
5. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000e4851fc
6. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000e486914
7. DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x000000000f0668f0
8. DB::ExpressionTransform::transform(DB::Chunk&) @ 0x0000000010c420cc
9. DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x000000000d4aa700
10. DB::ISimpleTransform::work() @ 0x00000000109f670c
11. DB::ExecutionThreadContext::executeTask() @ 0x0000000010a0ee0c
12. DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000010a05250
13. void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000010a067dc
14. ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000bdfad10
15. void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000000bdfdf40
16. void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000bdfcf04
17. start_thread @ 0x0000000000007624
18. ? @ 0x00000000000d162c

2024.09.02 19:42:34.208497 [ 739 ] {6afe2942-8f65-4684-bcdb-a0d3fdcd29f1} <Error> DynamicQueryHandler: Code: 131. DB::Exception: String too long for type FixedString(36): while converting source column dataset_id to destination column dataset_id: while executing 'FUNCTION _CAST(dataset_id :: 0, FixedString(36) :: 11) -> _CAST(dataset_id, FixedString(36)) FixedString(36) : 10'. (TOO_LARGE_STRING_SIZE), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000bd62308
1. DB::Exception::Exception<String>(int, FormatStringHelperImpl<std::type_identity<String>::type>, String&&) @ 0x0000000007739030
2. COW<DB::IColumn>::immutable_ptr<DB::IColumn> DB::FunctionToFixedString::executeForN<(DB::ConvertToFixedStringExceptionMode)0>(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, unsigned long) @ 0x000000000796a8c8
3. DB::(anonymous namespace)::ExecutableFunctionCast::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x0000000007803d30
4. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000e48436c
5. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000e4851fc
6. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000e486914
7. DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x000000000f0668f0
8. DB::ExpressionTransform::transform(DB::Chunk&) @ 0x0000000010c420cc
9. DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x000000000d4aa700
10. DB::ISimpleTransform::work() @ 0x00000000109f670c
11. DB::ExecutionThreadContext::executeTask() @ 0x0000000010a0ee0c
12. DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000010a05250
13. void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000010a067dc
14. ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000bdfad10
15. void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000000bdfdf40
16. void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000bdfcf04
17. start_thread @ 0x0000000000007624
18. ? @ 0x00000000000d162c
 (version 24.3.8.13 (official build))
2024.09.02 19:42:34.534957 [ 723 ] {410d85bc-9499-4af2-9181-6cd16ba92948} <Error> executeQuery: Code: 131. DB::Exception: String too long for type FixedString(36): while converting source column dataset_id to destination column dataset_id: while executing 'FUNCTION _CAST(dataset_id :: 0, FixedString(36) :: 11) -> _CAST(dataset_id, FixedString(36)) FixedString(36) : 10'. (TOO_LARGE_STRING_SIZE) (version 24.3.8.13 (official build)) (from 172.17.0.1:58258) (in query:  INSERT INTO dataset_items ( id, dataset_id, source, trace_id, span_id, input, expected_output, metadata, created_at, workspace_id, created_by, last_updated_by ) SELECT new.id, multiIf( LENGTH(CAST(old.dataset_id AS Nullable(String))) > 0 AND notEquals(old.dataset_id, new.dataset_id), leftPad('', 40, '*'), LENGTH(CAST(old.dataset_id AS Nullable(String))) > 0, old.dataset_id, new.dataset_id ) as dataset_id, new.source, new.trace_id, new.span_id, new.input, new.expected_output, new.metadata, multiIf( notEquals(old.created_at, toDateTime64('1970-01-01 00:00:00.000', 9)), old.created_at, new.created_at ) as created_at, multiIf( LENGTH(old.workspace_id) > 0 AND notEquals(old.workspace_id, new.workspace_id), CAST(leftPad('', 40, '*') AS FixedString(19)), LENGTH(old.workspace_id) > 0, old.workspace_id, new.workspace_id ) as workspace_id, if( LENGTH(old.created_by) > 0, old.created_by, new.created_by ) as created_by, new.last_updated_by FROM ( SELECT '0191b442-adfd-723e-9cb6-597d194669da' AS id, '0191b442-ae1b-756c-9936-17709c162a6c' AS dataset_id, 'manual' AS source, '' AS trace_id, '' AS span_id, '{"3gDu8uttcI":"Xa4MNhwlet","0SjHnR7G8z":"caB5XIrpOF","pSARxIqvdy":"eHVx8QwV2r","edI1WyyYU7":"wt64NJJP5G","cHKC9L2vLJ":"qIhNGliysR"}' AS input, '{"0gVO3uaRve":"HYndoszKQp","Og2RgZ2L9a":"Y3NniHAoB7","TF5pwE9gNP":"Qe7VLjFGT9","VwsflYuN2S":"o4wYaFrBmL","Z9NCOKQhzq":"rWwCQ4DdV1"}' AS expected_output, '{"iMLhNgX830":"W0FZkt1Zu4","ba4NEPRGAA":"92BTKIy24b","bD5GBiQL1o":"LZtwE5pbwR","Ald8n5Gy3g":"o8eQX0myZ5","Ok5SYakB5o":"u933QXk4tC"}' AS metadata, now64(9) AS created_at, 'e3577c3d-5cca-497b-b6be-b1b2f4cfa334' AS workspace_id, '0866bfa8-d0e6-4ea9-bfce-633c787b8394' AS created_by, '0866bfa8-d0e6-4ea9-bfce-633c787b8394' AS last_updated_by ) AS new LEFT JOIN ( SELECT * FROM dataset_items WHERE id = '0191b442-adfd-723e-9cb6-597d194669da' ORDER BY last_updated_at DESC LIMIT 1 BY id ) AS old ON old.id = new.id ; ), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000bd62308
1. DB::Exception::Exception<String>(int, FormatStringHelperImpl<std::type_identity<String>::type>, String&&) @ 0x0000000007739030
2. COW<DB::IColumn>::immutable_ptr<DB::IColumn> DB::FunctionToFixedString::executeForN<(DB::ConvertToFixedStringExceptionMode)0>(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, unsigned long) @ 0x000000000796a8c8
3. DB::(anonymous namespace)::ExecutableFunctionCast::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x0000000007803d30
4. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000e48436c
5. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000e4851fc
6. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000e486914
7. DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x000000000f0668f0
8. DB::ExpressionTransform::transform(DB::Chunk&) @ 0x0000000010c420cc
9. DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x000000000d4aa700
10. DB::ISimpleTransform::work() @ 0x00000000109f670c
11. DB::ExecutionThreadContext::executeTask() @ 0x0000000010a0ee0c
12. DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000010a05250
13. void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000010a067dc
14. ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000bdfad10
15. void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000000bdfdf40
16. void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000bdfcf04
17. start_thread @ 0x0000000000007624
18. ? @ 0x00000000000d162c

2024.09.02 19:42:34.535162 [ 723 ] {410d85bc-9499-4af2-9181-6cd16ba92948} <Error> DynamicQueryHandler: Code: 131. DB::Exception: String too long for type FixedString(36): while converting source column dataset_id to destination column dataset_id: while executing 'FUNCTION _CAST(dataset_id :: 0, FixedString(36) :: 11) -> _CAST(dataset_id, FixedString(36)) FixedString(36) : 10'. (TOO_LARGE_STRING_SIZE), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000bd62308
1. DB::Exception::Exception<String>(int, FormatStringHelperImpl<std::type_identity<String>::type>, String&&) @ 0x0000000007739030
2. COW<DB::IColumn>::immutable_ptr<DB::IColumn> DB::FunctionToFixedString::executeForN<(DB::ConvertToFixedStringExceptionMode)0>(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, unsigned long) @ 0x000000000796a8c8
3. DB::(anonymous namespace)::ExecutableFunctionCast::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x0000000007803d30
4. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000e48436c
5. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000e4851fc
6. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000000e486914
7. DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x000000000f0668f0
8. DB::ExpressionTransform::transform(DB::Chunk&) @ 0x0000000010c420cc
9. DB::ISimpleTransform::transform(DB::Chunk&, DB::Chunk&) @ 0x000000000d4aa700
10. DB::ISimpleTransform::work() @ 0x00000000109f670c
11. DB::ExecutionThreadContext::executeTask() @ 0x0000000010a0ee0c
12. DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000010a05250
13. void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000010a067dc
14. ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000bdfad10
15. void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000000bdfdf40
16. void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000bdfcf04
17. start_thread @ 0x0000000000007624
18. ? @ 0x00000000000d162c
 (version 24.3.8.13 (official build))

Answer 5 · 2024-09-02T19:58:47.000Z

 java.net.SocketException: Socket closed
! at java.base/sun.nio.ch.NioSocketImpl.endRead(NioSocketImpl.java:243)
! at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:323)
! at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:346)
! at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:796)
! at java.base/java.net.Socket$SocketInputStream.read(Socket.java:1099)
! at org.apache.hc.core5.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:149)
! at org.apache.hc.core5.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
! at org.apache.hc.core5.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:247)
! at org.apache.hc.core5.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:54)
! at org.apache.hc.core5.http.impl.io.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:299)
! at org.apache.hc.core5.http.impl.io.HttpRequestExecutor.execute(HttpRequestExecutor.java:175)
! at org.apache.hc.core5.http.impl.io.HttpRequestExecutor.execute(HttpRequestExecutor.java:218)
! at org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager$InternalConnectionEndpoint.execute(PoolingHttpClientConnectionManager.java:712)
! at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.execute(InternalExecRuntime.java:216)
! at org.apache.hc.client5.http.impl.classic.MainClientExec.execute(MainClientExec.java:116)
! at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
! at org.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:188)
! at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
! at org.apache.hc.client5.http.impl.classic.ProtocolExec.execute(ProtocolExec.java:192)
! at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
! at org.apache.hc.client5.http.impl.classic.HttpRequestRetryExec.execute(HttpRequestRetryExec.java:96)
! at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
! at org.apache.hc.client5.http.impl.classic.RedirectExec.execute(RedirectExec.java:115)
! at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
! at org.apache.hc.client5.http.impl.classic.InternalHttpClient.doExecute(InternalHttpClient.java:170)
! at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:123)
! at com.clickhouse.client.http.ApacheHttpConnectionImpl.post(ApacheHttpConnectionImpl.java:260)
! at com.clickhouse.client.http.ClickHouseHttpClient.send(ClickHouseHttpClient.java:195)
! at com.clickhouse.client.AbstractClient.sendAsync(AbstractClient.java:161)
! at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:273)
! at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run$$$capture(CompletableFuture.java:1768)
! at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java)
! at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
! at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
! at java.base/java.lang.Thread.run(Thread.java:1583)
ERROR [2024-09-02 19:42:37,419] com.clickhouse.client.http.ApacheHttpConnectionImpl: HTTP request failed
! java.net.SocketException: Socket closed
! at java.base/sun.nio.ch.NioSocketImpl.endRead(NioSocketImpl.java:243)
! at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:323)
! at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:346)
! at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:796)
! at java.base/java.net.Socket$SocketInputStream.read(Socket.java:1099)
! at org.apache.hc.core5.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:149)
! at org.apache.hc.core5.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
! at org.apache.hc.core5.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:247)
! at org.apache.hc.core5.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:54)
! at org.apache.hc.core5.http.impl.io.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:299)
! at org.apache.hc.core5.http.impl.io.HttpRequestExecutor.execute(HttpRequestExecutor.java:175)
! at org.apache.hc.core5.http.impl.io.HttpRequestExecutor.execute(HttpRequestExecutor.java:218)
! at org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager$InternalConnectionEndpoint.execute(PoolingHttpClientConnectionManager.java:712)
! at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.execute(InternalExecRuntime.java:216)
! at org.apache.hc.client5.http.impl.classic.MainClientExec.execute(MainClientExec.java:116)
! at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
! at org.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:188)
! at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
! at org.apache.hc.client5.http.impl.classic.ProtocolExec.execute(ProtocolExec.java:192)
! at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
! at org.apache.hc.client5.http.impl.classic.HttpRequestRetryExec.execute(HttpRequestRetryExec.java:96)
! at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
! at org.apache.hc.client5.http.impl.classic.RedirectExec.execute(RedirectExec.java:115)
! at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
! at org.apache.hc.client5.http.impl.classic.InternalHttpClient.doExecute(InternalHttpClient.java:170)
! at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:123)
! at com.clickhouse.client.http.ApacheHttpConnectionImpl.post(ApacheHttpConnectionImpl.java:260)
! at com.clickhouse.client.http.ClickHouseHttpClient.send(ClickHouseHttpClient.java:195)
! at com.clickhouse.client.AbstractClient.sendAsync(AbstractClient.java:161)
! at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:273)
! at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run$$$capture(CompletableFuture.java:1768)
! at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java)
! at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
! at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
! at java.base/java.lang.Thread.run(Thread.java:1583)
ERROR [2024-09-02 19:42:37,419] reactor.core.publisher.Operators: Operator called default onErrorDropped
! java.net.ConnectException: Socket closed
! at com.clickhouse.client.http.ApacheHttpConnectionImpl.post(ApacheHttpConnectionImpl.java:276)
! at com.clickhouse.client.http.ClickHouseHttpClient.send(ClickHouseHttpClient.java:195)
! at com.clickhouse.client.AbstractClient.sendAsync(AbstractClient.java:161)
! at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:273)
! at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run$$$capture(CompletableFuture.java:1768)
! at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java)
! at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
! at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
! at java.base/java.lang.Thread.run(Thread.java:1583)
! Causing: com.clickhouse.client.ClickHouseException: Socket closed
! at com.clickhouse.client.ClickHouseException.of(ClickHouseException.java:149)
! Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
! Assembly trace from producer [reactor.core.publisher.MonoCompletionStage] :
! 
! Error has been observed at the following site(s):
! *__Mono.fromFuture ⇢ at com.clickhouse.r2dbc.ClickHouseStatement.lambda$execute$0(ClickHouseStatement.java:149)
! Original Stack Trace:
! 	at com.clickhouse.client.ClickHouseException.of(ClickHouseException.java:149)
! 	at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:275)
! 	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run$$$capture(CompletableFuture.java:1768)
! 	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java)
! 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
! 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
! 	at java.base/java.lang.Thread.run(Thread.java:1583)

Answer 6 · 2024-09-03T15:11:22.000Z

We've seen similar behavior in our applications.

Our code to construct the client looks like this:

    public Client client() {
        return new Client.Builder()
                .addEndpoint(Protocol.HTTP, host, port, false)
                .setUsername(username)
                .setPassword(password)
                .compressServerResponse(true)
                .setDefaultDatabase(ClickHouseConstants.DB_NAME)
                .setSocketTimeout(30, ChronoUnit.SECONDS)
                .build();
    }

and when using the client, we construct a new client for each request:

    @Override
    public QueryResponse querySync(String sqlQuery, Map<String, Object> params, QuerySettings settings) {
        try (Client client = connection.client()) {
            return client.query(sqlQuery, params, settings).join();
        }
    }

I'd say we get socket timeouts roughly 1/3 requests with this setup (although previously had none when there were only inserts, not queries using the v2 api). Switching to a long-lived client, created at the start of the application, seems to have resolved the issue for the most part. Is that recommended practice? @thiagohora does your code regularly close the client too?

We have seen this issue using versions 0.6.4 and 0.6.5 of client-v2 when running clickhouse through Testcontainers (haven't deployed the new code to production yet because of this).

Answer 7 · 2024-09-03T17:46:22.000Z

@alxhill Thank you for the report!
Yes, it is a recommended practice to reuse client. There are few comments:

client instance is quite lightweight and doesn't spawn background threads.
internally client uses Apache HTTP client and this client has own connection pool (what can be disabled if needed)

Starting 0.6.4 uses caller thread to execute requests. It is done to:

reduce resources creation
let application fully control when to run request (Spring for example supports async bean methods)
reduce thread handling overhead (when application has many requests to DB it can be significant)

Answer 8 · 2024-09-03T17:55:17.000Z

Thank you @thiagohora !
Have you tried to reconfigure the server to remove warning:

?

Is there any related issue in the server log?
Would you please turn on debug on the clickhouse side?

Here is successful request looks like in log:

2024.09.01 14:43:10.687368 [ 296710 ] {} <Trace> HTTPHandler-factory: HTTP Request for HTTPHandler-factory. Method: POST, Address: 127.0.0.1:48794, User-Agent: Apache-HttpClient/5.3.1 (Java/21.0.2), Content Type: text/plain, Transfer Encoding: chunked, X-Forwarded-For: (none)
2024.09.01 14:43:10.687459 [ 296710 ] {} <Trace> DynamicQueryHandler: Request URI: /?compress=1
2024.09.01 14:43:10.687486 [ 296710 ] {} <Debug> HTTP-Session: 1e39e2e8-cf85-4e3f-8ccb-0febaa574102 Authenticating user 'default' from 127.0.0.1:48794
2024.09.01 14:43:10.687520 [ 296710 ] {} <Debug> HTTP-Session: 1e39e2e8-cf85-4e3f-8ccb-0febaa574102 Authenticated with global context as user 94309d50-4f52-5250-31bd-74fecac179db
2024.09.01 14:43:10.687552 [ 296710 ] {} <Debug> HTTP-Session: 1e39e2e8-cf85-4e3f-8ccb-0febaa574102 Creating session context with user_id: 94309d50-4f52-5250-31bd-74fecac179db
2024.09.01 14:43:10.687596 [ 296701 ] {} <Trace> HTTPHandler-factory: HTTP Request for HTTPHandler-factory. Method: POST, Address: 127.0.0.1:48796, User-Agent: Apache-HttpClient/5.3.1 (Java/21.0.2), Content Type: text/plain, Transfer Encoding: chunked, X-Forwarded-For: (none)
2024.09.01 14:43:10.687651 [ 296710 ] {} <Trace> HTTP-Session: 1e39e2e8-cf85-4e3f-8ccb-0febaa574102 Creating query context from session context, user_id: 94309d50-4f52-5250-31bd-74fecac179db, parent context user: default
2024.09.01 14:43:10.687685 [ 296701 ] {} <Trace> DynamicQueryHandler: Request URI: /?compress=1
2024.09.01 14:43:10.688288 [ 296701 ] {} <Debug> HTTP-Session: 27fb3354-2c3f-4e21-8ab0-b019f420ab08 Authenticating user 'default' from 127.0.0.1:48796
2024.09.01 14:43:10.688436 [ 296701 ] {} <Debug> HTTP-Session: 27fb3354-2c3f-4e21-8ab0-b019f420ab08 Authenticated with global context as user 94309d50-4f52-5250-31bd-74fecac179db
2024.09.01 14:43:10.688460 [ 296701 ] {} <Debug> HTTP-Session: 27fb3354-2c3f-4e21-8ab0-b019f420ab08 Creating session context with user_id: 94309d50-4f52-5250-31bd-74fecac179db
2024.09.01 14:43:10.687931 [ 296710 ] {085f0643-7ed9-45fc-bf2e-4b6571447a69} <Debug> executeQuery: (from 127.0.0.1:48794) SELECT generateUUIDv4() as id, toUInt32(number) as p1, number, toFloat32(number/100000) as p2, toFloat64(number/100000) as p3 FROM system.numbers LIMIT 100000 (stage: Complete)
2024.09.01 14:43:10.688691 [ 296688 ] {e7ee4c7c-326e-4795-aa1b-000e3554d993} <Debug> executeQuery: Read 100000 rows, 781.25 KiB in 0.05579 sec., 1792435.9204158452 rows/sec., 13.68 MiB/sec.
2024.09.01 14:43:10.688776 [ 296693 ] {28877523-e8f2-4cdd-80df-c130ad5bb755} <Debug> DynamicQueryHandler: Done processing query
2024.09.01 14:43:10.688824 [ 296693 ] {28877523-e8f2-4cdd-80df-c130ad5bb755} <Debug> MemoryTracker: Peak memory usage (for query): 9.87 MiB.
2024.09.01 14:43:10.688842 [ 296693 ] {} <Debug> HTTP-Session: a75b2146-2193-4f66-857b-7fcd08864b16 Logout, user_id: 94309d50-4f52-5250-31bd-74fecac179db

Anything between HTTP-Session: 1e39e2e8-cf85-4e3f-8ccb-0febaa574102 Authenticating user 'default' from 127.0.0.1:48794 and Logout, user_id: 94309d50-4f52-5250-31bd-74fecac179db is interesting. you may track by thread ID like [ 296693 ]

Here is doc how to find all logs https://clickhouse.com/docs/en/guides/troubleshooting#check-the-logs

Answer 9 · 2024-09-05T21:28:34.000Z

I cannot see, However, the inserts seemed very slow. Not sure if this can be related