what is the scaleWithCpu meaning?
zdyj3170101136 opened this issue · 4 comments
I find the scaleWithCpu define the relationship between batchsize and cpu cores.
If i have a 2 core machine, then the batchsize is 2000, every 2000 span start a goroutine and save the data.
If i have a 8 core machine, then the batchsize is 8000, every 8000 span start a goroutine and save the data.
I do not understand why it is helpful?
This is the wrong repo, but here is the answer anyway.
Basically, the bigger the batchSize the more data you can insert into ClickHouse. It does not scale infinitely, but it helps.
But the bigger batchSize means bigger RAM requirements, because those spans are kept in RAM. So scaleWithCpu assumes that the more CPU cores you have the more RAM you can dedicate to Uptrace. But why CPU cores? Because you can limit the number of cores with GOMAXPROCS and tweak Uptrace that way.
Assuming that per span cost 1 kb memory.
Thousands of span only cost 1 mb memory.
Even if you write with batchsize 50000, it only cost 50mb memory.
I think it is helpless.
Assuming it is 1 kb - yes. But span can be much more than that.
Our jaeger receive more than 10w span/s in production.
And the 1kb is the average storage cost before compress in clickhouse.
So i think 1kb is an appropriate value.