Reactive benchmark
Closed this issue · 14 comments
https://codesandbox.io/s/oby-bench-cellx-forked-mqlv4x?from-embed
If I select 5000 for usignal it says Maximum call stack size exceeded. Why is that?
If i select sinuous and put 50000 also doesn't break anything.
I clicked usignal and then run and it was OK, what am I looking at?
Change the number of layers to 5000 or above
VanillaJS, Knockout, MobX, and Preact also fail at 10000 here, while usignal doesn't ... it fails after.
I think what you are seeing is a regular call stack error that affect all libraries based on call stacks instead of push/pop? as many fails, I don't think this is a real-world concern, not more than any call stack issue we have in JS in general.
About this line though:
// P.computed(() => { //FIXME: Never executed??nothing tofix there, computedare computed only if accessed/their value is needed. If you don't need their values, these do nothing, and this is by design (both usignal since ever, preact recently too).
P.S. others will likely fail at array stack limit Math.pow(2, 32) although that's arguably a much higher limit than 10K ... maybe I cn use a "linked list" instead of a callstack ... interesting bench though.
OK, I did drop some recursion using a lazy stack but that gave me few extra calls in NodeJS. Curious to see bun having way less troubles there while NodeJS indeed struggles to even reach 5000 (I tested online with Firefox, btw).
Using push/pop gave me some extra room and so did the linked list but still the elephant in the room seems to be the try/finally eating even more stack ... however, I don't want to remove the try/finally as it's essential to keep the application state sane ad correct even if some callback fails to execute while computing, and same goes for effects.
Basically, while focusing and optimizing for the compute part on signal changes, the issue here is with recursive computed all flagged as must execute even if the only signal that really matters is the one at the root of the chain.
Anyway, some simplification has been made and now I can reach in NodeJS up to 4829 layers, if I add one more it breaks.
btw, updating to 0.7.8 seems to work fine in both Chrome, WebKit, and Firefox, up to 5000 ... do you experience the same?
@WebReflection No, I cannot run 5000 layers no matter what. But anyway if this is not an issue/blocker you can close this ticket.
even more bizarre ... this test here passes but if I change the top to:
testUsignal(report, 4830);it fails! however, if I run that twice it works without issues even with 5000!
testUsignal(report, 4829);
testUsignal(report, 5000);same code, NodeJS just decides to allocate more stack there ... what the hell 😂
I will keep an eye to that callstack benchmark and see how/if I can solve the issue, but it looks like as soon as I invoke 8000++ nested computed values results become pretty unpredictable.
Other libraries that don't fail clearly use a different resolution for the problem, like updating on signal changes right away, which might explain why both usignal and preact fail with big numbers.
I cannot run 5000 layers no matter what.
can you confirm you have the same issue also with other libraries I've mentioned? Thanks, if that's the case, I think I'd be OK in closing this. If only usignal fails in your machine, then I'd love to know what machine is it, which OS, and which version of which browser you are using.
even more bizarre ... this test here passes but if I change the top to:
testUsignal(report, 4830);it fails! however, if I run that twice it works without issues even with 5000!
testUsignal(report, 4829); testUsignal(report, 5000);same code, NodeJS just decides to allocate more stack there ... what the hell 😂
I will keep an eye to that callstack benchmark and see how/if I can solve the issue, but it looks like as soon as I invoke 8000++ nested computed values results become pretty unpredictable.
Other libraries that don't fail clearly use a different resolution for the problem, like updating on signal changes right away, which might explain why both usignal and preact fail with big numbers.
I cannot run 5000 layers no matter what.
can you confirm you have the same issue also with other libraries I've mentioned? Thanks, if that's the case, I think I'd be OK in closing this. If only usignal fails in your machine, then I'd love to know what machine is it, which OS, and which version of which browser you are using.
ooh, that's strange and funny lol...
All others fail beyond 5000 or 10000 but sinuous stays very strong even If I use 500000 Layers. https://github.com/luwes/sinuous/tree/main/src/observable
Processor Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz 2.30 GHz
Installed RAM 16.0 GB (15.8 GB usable)
System type 64-bit operating system, x64-based processor
Chrome Browser - 105.0.5195.127
Window 11
hah ... it uses the same runtime swapping concept through the scope but it disable tracking on computed updates ... I start thinking the benchmark uses a feature other libraries might miss, or it's not comparing apples to apples ... I'll give that tracking a chance though!
OK, I've pushed a new branch called tracking and indeed I can run this demo/test/benchmark without issues with 500000 or more changes ... however:
- it's not clear to me how conditional computed values can keep track of new subscribers when a signal changes and no-tracking is enforced ... as that new invoke might result into new subscribers due the different returned value while executing
- it's instead clear to me this benchmark is based on libraries that (imho, wrongly) compute values on every signal update, as opposite of computing values only when the computed is actually needed/invoked/accessed
- because of the previous points, the fact every library fails at some point due inevitable call stack issue, and the fact the benchmark is comparing apples to oranges (libraries that side-effects on computed VS libraries that don't do that) I think I'll close this issue as won't fix
That being said, I wonder if results, currently not tested in terms of correct results, would show shenanigans for Sinous if one computed is like this instead:
prop2: computed(function () {
return (Math.random() < .5 ? m.prop1() : m.prop2()) - m.prop3();
})I am speculating here in terms of results, but it'd be interesting to test correctness around that pattern that could be simplified, and make it observable (pun intended) with a module value:
prop2: computed(function () {
return (m.prop1() % 2 ? m.prop1() : m.prop2()) - m.prop3();
})Now this would be interesting to follow up, as I believe it's impossible from the pattern logic to know if the computed actually used both signals or not, which is something I've recently fixed in usignal 0.5.x+ due the fact indeed computed can use conditional signals internally.
Still speculating, I believe the Sinuous library there either will fall into the callstack issue like every other, or it will show different results from others.
I hope this makes sense, yet it was super interesting and (personally) entertaining thing to look at, which resulted in me improving the code, despite my inability to ecape call stacks limits, but then again, this demo/benchmark is more accademic than a real-world scenario, so I am happy with my current results.
Closing, but feel free to ask/comment more, if needed 👋
OK, after a night thinking about it, I want the call stack issue to go, specially after discovering solid has untrack, which I wanted to test in there but solid seems to be a copy and paste of code in the baseline (why is that?).
In short, computed can be both lazy and allow untrack which solves call stack recursion.
Actually ... never mind. I need to test/read better before changing my mind ... untrack, if it doesn't listen to anything, is like a regular callback, so I am not sure how that works in solid-js and surely if it doesn't get notified about changes it won't ever run again due it's memoized nature.
Nah, will closeit again and keep it as won't fix until I find something that actually both work and make sense.
Sorry for the noise.