Consider using another metric than time to check for hot spots
julienfouilhe opened this issue · 3 comments
This tool is great, I solved a few problems thanks to it.
So I thought that I would add it as a step of our CI to prevent regressions or adding new performance issues.
But our CI runs on much slower machines than mine so it catches a lot more things.
Isn't there a metric more constant that this tool could be based on to figure out how much typescript has to work and where things could be improved?
@julienfouilhe Thanks for the feedback! I'm not sure I understand your question though. It sounds like you're asking whether the tool could highlight the parts of compilation that are relatively slow, so that it would be independent of the speed of a particular machine (though probably not independent of machine architecture or node version) but, if so, I'm not sure how you could integrate such a check into your CI. Can you please elaborate?
In the meantime, some thoughts on perf testing in CI:
- If you want to use the trace analyzer, I'd suggest playing with the various threshold options to find values that work on your CI machines. Probably, you want a relatively high bar to prevent a lot of unnecessary investigations.
- Tracing was really meant to diagnose problems, rather than to detect them. If you just want to know if your code is getting slower (the sort of thing you might want to check in CI), I'd probably start with
--extendedDiagnostics. Then, once you know that things have slowed down, you can run tracing manually to find out why.
My two cents. :)
@amcasey Thank you for your answer!
Our main problem currently is not the global time of compilation, but in parts of our app working with typescript in vscode is extremely slow and I was hoping removing hot spots would solve those devX problems. The thing is I expect other developers to find some of the changes made to remove those hot spots strange and to revert them, hence why I wanted to include this tool in our CI.
What I would do before I used this tool was analyze the trace and investigate chunks that were "bigger" than the others.
When I created this issue I was thinking that maybe another metric could be made available in traces to know how much work Typescript has to do to resolve something. In my mind a metric like that would make this tool much more reliable and useful.
I understand that maybe this metric doesn't exist and would be impossible to maintain in the typescript codebase.
But thanks anyway, you're right I'll play with the threshold options to find something that works with our CI!
Oh, I see now. Yeah, I did actually think about that when I was prototyping this work. I experimented with using the number of types allocated as a proxy for time to make traces reproducible. I ended up abandoning the approach for reasons I don't fully recall (I assume they must have been quite good since, as you point out, the advantages are substantial). Off the top of my head, I would guess that there were too many false positives - sometimes a lot of types are generated very cheaply and investigating those areas doesn't make sense. It also doesn't generalize well to other phases, especially module resolution.
Unfortunately, I think the threshold options are your best option for now. Thanks again for reaching out!