Performance Issue with large API specs

Question

Performance Issue with large API specs

Opened this issue 5 years ago · 8 comments

Hi,
we're stuck parsing, validating and resolving the full API surface of our system using this new parser (running it via the JS output).

Our API spec is public here: https://github.com/commercetools/commercetools-api-reference

The main entry file is api.raml

One problem is a recursion in the Categories API, but the performance problem remains even after commenting that recursion out in types/category/CategoryReference.raml

Feel free to use that huge API as a smoketest but keep in mind it's not OSS licensed (unlicensed) so you can't redistribute, modify or use the domain IP otherwise).

Any idea what could be the issue?

PS: we're currently working on it using our own parser / validator implementation which can handle it:
https://github.com/vrapio/rest-modeling-framework

Answer 1 · 2019-11-12T10:18:51.000Z

Hi @nkuehn.

What kind of performance issue do you have - memory, execution time?
Did you use a different parser before?

Answer 2 · 2019-11-12T10:48:20.000Z

It's running at 70% CPU on one core (typical node.js situation). It's parsing and validating but I have the impression that resolving is stuck.

We do use our own EMF based parser for all our code generation and other needs, see the link in the P.S.. We did try with the previous RAML JS parser, too which had issues with the recursive situation but was able to load the spec at all after commenting out the recursive situation (slowly, but not getting completely stuck).

Answer 3 · 2019-11-14T08:21:08.000Z

@nkuehn I see. I'm unable to resolve the doc you provided as well. I've reported this issues to our dependency lib. Please see the issue referenced above.

Answer 4 · 2019-11-14T09:47:31.000Z

thank you so much!

Answer 5 · 2020-06-19T08:34:29.000Z

Hi @nkuehn.

According to the answer I got (aml-org/amf#567 (comment)), our dependency lib performance has increased dramatically since the time this issue was reported.

We are currently working on updating our dependencies to the latest version. I'll let you know when the new version is available so you could test again.

Answer 6 · 2020-06-19T09:06:54.000Z

Thanks for the update @postatum ! We've switched to our own parser implementation in the meanwhile because we use it intensively for code generation too and have the knowledge in house. Nevertheless I'm tracking progress here, the topic will remain relevant.

Answer 7 · 2020-06-22T05:12:15.000Z

Thanks for the update @postatum ! We've switched to our own parser implementation in the meanwhile because we use it intensively for code generation too and have the knowledge in house. Nevertheless I'm tracking progress here, the topic will remain relevant.

I see. Is your parser open-sourced?

Answer 8 · 2020-06-22T07:02:20.000Z

Yes, it's not as productized or marketed though.

https://github.com/vrapio/rest-modeling-framework (Eclipse EMF based RAML model)
https://github.com/commercetools/rmf-codegen (our code generator on top of it)

It's JVM-based, so usage in javascript projects is not as convenient but by building on EMF we don't have to spend a lot of time in the low-level nitty gritty of metamodeling etc. (I'm not personally the maintainer, just one of the internal users)