When graphqlgen large schema will get stuck

Question

When graphqlgen large schema will get stuck

Victorkangsh opened this issue 6 years ago · 31 comments

Description

In one of my projects, the datamodel has five hundred lines. My schema.graphql file imports the generated prisma.graphql schema file and has a hundred lines of Query and Mutation.

I get stuck when I run graphqlgen. Cpu and memory are slightly higher. I noticed a prompt for the find prettier, but I still didn't succeed after removing the prettier configuration.

My three computers have linux, mac, and windows systems, and they are not working properly. But the surprise is that my mac has been successful once, just once.

Steps to reproduce

When schema.graphql too large.

My other small project run graphqlgen perfectly.

Expected results

Graphqlgen work success.

Actual results

Stucked. And console has no message.

Versions

graphqlgen: 0.4.0
OS name and version: mac: 16G memory, mojava(darwin-x64). windows: win10 16G memory i7.
Node: node-v10.8.0

Answer 1 · 2018-11-23T05:11:49.000Z

Supplement：
When wait 1 hour later, console log err as follows：

graphqlgen
<--- Last few GCs --->

[14832:000002002A721AF0]  8948164 ms: Mark-sweep 1159.4 (1202.0) -> 1159.4 (1165.5) MB, 3679.4 / 0.0 ms  last resort GC in old space requested
[14832:000002002A721AF0]  8948951 ms: Mark-sweep 1159.4 (1165.5) -> 1159.4 (1165.5) MB, 786.8 / 0.0 ms  last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0000038B35425EE1 <JSObject>
    0: builtin exit frame: concat(this=000002C19A6AE511 <JSArray[40009211]>,000002C19A682201 <JSArray[18728217]>)

    1: flatten(aka flatten) [C:\Users\T47\AppData\Roaming\npm\node_modules\graphqlgen\dist\utils.js:~49] [pc=000002BC79E8D883](this=0000006577382311 <undefined>,a=000002C19A6AE511 <JSArray[40009211]>,b=000002C19A682201 <JSArray[18728217]>)
    2: arguments adaptor frame: 4->2
    ...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

And when I set --max_old_space_size setting it still not working.

Answer 2 · 2018-11-25T16:54:49.000Z

Same issue. Looks like it's in a loop of some kind. I am importing one type from Prisma via graphql-import.

MacOS 10.14.1, 2015 iMac with 16gb RAM and 2016 MBP with 8gb RAM

<--- Last few GCs --->

[1153:0x104001c00]   403737 ms: Mark-sweep 1219.6 (1227.3) -> 1219.6 (1226.3) MB, 736.6 / 0.0 ms  (average mu = 0.079, current mu = 0.000) allocation failure GC in old space requested
[1153:0x104001c00]   404472 ms: Mark-sweep 1219.6 (1226.3) -> 1219.6 (1225.3) MB, 735.3 / 0.0 ms  (average mu = 0.052, current mu = 0.000) last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x05326251d9f1 <JSObject>
    0: builtin exit frame: concat(this=0x0532c9e86071 <JSArray[15602824]>,0x0532e3b266a1 <JSArray[14814181]>,0x0532c9e86071 <JSArray[15602824]>)

    1: /* anonymous */(aka /* anonymous */) [0x53221beec69] [/Users/dennismacbookpro/WebstormProjects/wk-mono-repo/node_modules/graphqlgen/dist/generators/common.js:~142] [pc=0x19d5dd850c51](this=0x0532387825b1 <undefined>,0x05328ef92ef9 <String[15]: S...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x10005eeab node::Abort() [/usr/local/bin/node]
 2: 0x10005f68a node::FatalTryCatch::~FatalTryCatch() [/usr/local/bin/node]
 3: 0x100172027 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
 4: 0x100171fc8 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
 5: 0x100433768 v8::internal::Heap::UpdateSurvivalStatistics(int) [/usr/local/bin/node]
 6: 0x100435185 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [/usr/local/bin/node]
 7: 0x100432afd v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/usr/local/bin/node]
 8: 0x1004318f3 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/bin/node]
 9: 0x100431b8f v8::internal::Heap::CollectAllAvailableGarbage(v8::internal::GarbageCollectionReason) [/usr/local/bin/node]
10: 0x100439835 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/usr/local/bin/node]
11: 0x1004195b3 v8::internal::Factory::AllocateRawArray(int, v8::internal::PretenureFlag) [/usr/local/bin/node]
12: 0x100419469 v8::internal::Factory::NewFixedArrayWithFiller(v8::internal::Heap::RootListIndex, int, v8::internal::Object*, v8::internal::PretenureFlag) [/usr/local/bin/node]
13: 0x100424f5a v8::internal::Factory::NewJSArrayStorage(v8::internal::Handle<v8::internal::JSArray>, int, int, v8::internal::ArrayStorageAllocationMode) [/usr/local/bin/node]
14: 0x100424e2b v8::internal::Factory::NewJSArray(v8::internal::ElementsKind, int, int, v8::internal::ArrayStorageAllocationMode, v8::internal::PretenureFlag) [/usr/local/bin/node]
15: 0x1003ce3ee v8::internal::ElementsAccessor::Concat(v8::internal::Isolate*, v8::internal::Arguments*, unsigned int, unsigned int) [/usr/local/bin/node]
16: 0x1001d9220 v8::internal::(anonymous namespace)::Fast_ArrayConcat(v8::internal::Isolate*, v8::internal::BuiltinArguments*) [/usr/local/bin/node]
17: 0x1001d6d5a v8::internal::Builtin_Impl_ArrayConcat(v8::internal::BuiltinArguments, v8::internal::Isolate*) [/usr/local/bin/node]
18: 0x19d5dd7cfcdd 
19: 0x19d5dd850c51 
20: 0x19d5dd7876a6 
21: 0x19d5dd871a16 
[1]    1153 abort      graphqlgen

Answer 3 · 2018-11-28T19:25:15.000Z

@Victorkangsh Did you try downgrading to 0.3.0. Mine got stuck without any info tho' but works fine with 0.3.0

Answer 4 · 2018-11-28T21:06:49.000Z

@mentrie Thanks for the suggestion. But if I downgrade, then I'm hit with the #280 bug 😢

Answer 5 · 2018-11-28T21:40:36.000Z

@LawJolla Right. For the past 1hr, 0.4.0 is still running 😢

Answer 6 · 2018-12-07T18:02:45.000Z

Same issue here. Can't use at all until this is resolved. My system has 32 GB of RAM, and it still doesn't work. Haven't let it run for an hour, but after several minutes I still have plenty of RAM left.

Answer 7 · 2018-12-13T17:07:15.000Z

I've spend few hours debugging the package and it seems that here :
https://github.com/prisma/graphqlgen/blob/03336f01d1257643c4b24577a397df44bd00076a/packages/graphqlgen/src/generators/common.ts#L228 it turns into the endless loop.

When i add Mutation to my schema with the same name as one in the prisma.graphql it runs until it fails.

Answer 8 · 2018-12-13T17:08:14.000Z

@mentrie Thanks for the suggestion. But if I downgrade, then I'm hit with the #280 bug 😢

that has been fixed. I've downgraded to 0.3.0 and it worked even with the schema that fails on 0.4.0

Answer 9 · 2018-12-13T17:55:33.000Z

@mentrie Thanks for the suggestion. But if I downgrade, then I'm hit with the #280 bug 😢

that has been fixed. I've downgraded to 0.3.0 and it worked even with the schema that fails on 0.4.0

@woss It's still an issue for me in 0.3.0. This project looked promising, but it now appears under maintained if not abandoned. I've given up and gone back to Javascript to actually get things built... pesky deadlines. :)

Answer 10 · 2018-12-19T01:48:15.000Z

Anyone have a schema that leads to this issue that they can share?

Answer 11 · 2018-12-19T05:02:58.000Z

@jasonkuhrt I have figured out its problem, it has nothing to do with the size of the schema, but an error when using the imported Input Type.
like this example : https://github.com/prisma/graphql-prisma-typescript/blob/master/src/schema.graphql

when you add some Mutation using imported Input Type it will stuck:

# import * from "./generated/prisma.graphql"
type Mutation {
  addPaymentMethod(
    data: PaymentCreateInput
  ): MutationResult!
  book(
    data: BookingCreateInput
  ): MutationResult!
}

Answer 12 · 2018-12-20T00:44:15.000Z

This bug renders graphqlgen unusable for me. Running with version 3 I get a bunch of TS errors while running on latest gets stuck.

I was more specific with my imports from Prisma generated code.

// I used
# import Post, User, UserCreateInput, UserUpdateInput from './generated/prisma.graphql'

// VS
# import * from "./generated/prisma.graphql"

node -v
v11.4.0

npm -v
6.4.1

OSx 10.12.6

prisma -v
prisma/1.22.2

https://ibb.co/0DjYqkG

Any hacky workaround ideas for now? A nightly I can test?

Answer 13 · 2018-12-20T01:15:50.000Z

Quick update: in VS code, using cmd + . then selecting add all missing imports adds all missing imports - who would've thought?

Answer 14 · 2018-12-20T01:31:32.000Z

@Victorkangsh iiuc:

This issue occurs in conjunction with graphql import
I should be able to repro with
https://github.com/prisma/graphql-prisma-typescript/blob/master/src/schema.graphql
But a more minimal repro can be had by importing one schema (A) into another (B) and using A types in B Mutation fields args.

Did I get that right?

Answer 15 · 2018-12-20T07:32:54.000Z

@jasonkuhrt I think the problem is nested InputType.
I use this config:

when I try this:

it's not work, and i copy in schema:

still stuck.
But when I remove nest Type , it worked:

Answer 16 · 2018-12-20T07:38:36.000Z

graphql import has no err , I test this:

all success

Answer 17 · 2018-12-20T13:31:45.000Z

@jasonkuhrt

Anyone have a schema that leads to this issue that they can share?

https://github.com/vdiaz1130/graphqlgen-issue310

I copied over Prisma generated stuff and the schema (you wont be able to regen Prisma unless you configure all of that).

Answer 18 · 2018-12-20T14:15:10.000Z

Anyone have a schema that leads to this issue that they can share?

gist

Answer 19 · 2018-12-20T15:56:09.000Z

To provide some more context.
I'm getting a runtime error after ~1hr from flatten method in utils.js.
The error is a Range Error

Added console.log to the array and the length does grow exponentially.

function flatten(a, b) {
    a.length > 100000 && console.log('length', a.length);

    return a.concat(b);
}

I could see that the type it's trying to parse is Mutation

Would highly appreciate your help with this one

Answer 20 · 2018-12-22T20:27:39.000Z

@mentrie Thanks for the suggestion. But if I downgrade, then I'm hit with the #280 bug 😢

Experiencing the same issue. Any updates on this?

Answer 21 · 2018-12-27T15:20:14.000Z

I worked around this by not importing from Prisma generated. I simply copied over my schema and removed all the Prisma specific things (@relation, @default, etc). Also using version 0.3.0 of graphqlgen.

If in your Prisma generated code you have an error on the Model interface, delete the import and reimported via VS code "cmd + ." It should import from a ''prisma-client-lib/dist/types' - Fix this before you generate with graphqlgen.

After generating with graphqlgen, If you get missing imports from your generated code and you are using VS Code, go next to a type, class, interface etc, then hit "cmd + ." Select "add all missing imports".

Answer 22 · 2019-01-01T13:26:05.000Z

Before considering this issue resolved I'd like to hear from multiple users if its their experience that the issue is resolved too.

Answer 23 · 2019-01-01T19:18:00.000Z

Thanks @jasonkuhrt !

Is there a particular release we should try? I still see 0.4.0 as the current.

Answer 24 · 2019-01-01T21:09:46.000Z

Before considering this issue resolved I'd like to hear from multiple users if its their experience that the issue is resolved too.

I added the line you added in the pull request to my local code.
It's been running for ~90 mins without stopping.
Looks like it's still reproducing.

Answer 25 · 2019-01-02T02:08:36.000Z

@LawJolla I'm not doing releases yet myself I'm afraid (might start to this or next month though) but maybe you could pull down from git repo (but build steps and all...). Given the diff is trivial what @atardadi did might be easiest.

@atardadi thanks for the feedback! Good to know...

-- Update

Yeah, this issue is not resolved I'm pretty sure.

I just pushed a commit that increased performance of the surrounding function by 6x (effect measured and tracked in benchmark history). It's just a microbenchmark improvement. The existing integration benchmarks did not improve.

Sorry for false hope, still in the woods. I am however continuing to get a better grasp of codebase. Hopefully we'll close this soon!

Answer 26 · 2019-01-02T02:15:48.000Z

@vdiaz1130 thanks for the repo repro. I'd like to extract the minimal:

model file(s)
schema file(s)

needed, and repro with the graphqlgen test suite.

But, can you advise how I should get prisma-client-lib installed and setup exactly? Even better might be a repro that doesn't need that dep to begin with (would simplify a bit I suppose).

Answer 27 · 2019-01-02T10:24:55.000Z

My project can be generated by #373
Only the last problem left: #278

Answer 28 · 2019-01-03T02:19:55.000Z

@Victorkangsh ok thanks for confirming, interesting... I think then, since this issue was originally created by you we should close it. Others have raised other issues here but going by your feedback, they are different issues. As such I would like to see them tracked as separate issues and have focused discussion.

Does anyone strongly object?

Answer 29 · 2019-01-03T15:01:34.000Z

@Victorkangsh ok thanks for confirming, interesting... I think then, since this issue was originally created by you we should close it. Others have raised other issues here but going by your feedback, they are different issues. As such I would like to see them tracked as separate issues and have focused discussion.

Does anyone strongly object?

I don't mind, although I have the same issue - once my schema grew large it doesn't generate it for me.
We could try publishing a new version and see if that solves it for me.
Otherwise, I will create a separate issue

Answer 30 · 2019-01-03T16:04:03.000Z

We could try publishing a new version and see if that solves it for me.

I haven't been doing that personally yet, though I think making a pre-release would be fine without much prep, however I don't think I have the required npm rights yet.

Answer 31 · 2019-01-08T18:02:21.000Z

@Victorkangsh was there a separate issue created for this?