FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

[schuettc]

Issue Report

When attempting to run repolinter.js via node, getting this error.

Expected Behavior

repolinter.js should complete

Actual Behavior

repolinter.js failed with the following error:

<--- Last few GCs --->

[96417:0x128008000]   468543 ms: Mark-sweep 16116.2 (16423.6) -> 16106.2 (16427.1) MB, 4611.4 / 0.3 ms  (average mu = 0.287, current mu = 0.004) allocation failure scavenge might not succeed
[96417:0x128008000]   477428 ms: Mark-sweep 16122.0 (16427.1) -> 16111.9 (16432.1) MB, 8867.6 / 0.3 ms  (average mu = 0.120, current mu = 0.002) allocation failure scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

Steps to Reproduce the Issue

node bin/repolinter.js lint ../repo-to-lint -r ../ruleset.json

Other notes

Have increased "max-old-space-size" to 16384 by:

export NODE_OPTIONS="--max-old-space-size=16384"

With no effect.

Have attempted using node v18.16.0 and v16.19.0

Using M1 Mac.

[zhaoyuheng200]

I've seen this issue before and it happened on a repository node_modules folder with symlink in it.
Assuming that's a nodejs repository you are linting, can you try remove node_modules or any thing similar and lint again?

[schuettc]

Tried that with no change. The original repo was created with projen. Any issues there?

[zhaoyuheng200]

Since symlink was the root cause of my issue at the time, I just tied the problem to that.
From here I can only think of 2 things to do:

1. List all symlinks in the project by ls -lR /path/to/folder | grep ^l, ref: link
2. link each dir in every layer until you locate the dir/file that's causing the issue. I'm only recommending this on the belief that there is nothing wrong with repolinter and there is a symlink there somewhere, so increase space size won't solve your problem.

[zhaoyuheng200]

btw if you can share your ruleset, that can be helpful.

[hyandell]

Noting that you both work at AWS; so perhaps contacting each other internally will allow you to share the rules/codebase in question for better replication.

[hakanson]

I was able to track this down to a set of "type": "file-not-contents" rules. My project was a monorepo with node_modules and some rust compiled to WASM. Too many files were being scanned.

I avoided this error two different ways:

1. Delete these rule definitions (used to isolate rule)
2. Run repolint on a local git clone (eg. git clone ~/src/project newproject

It would be nice if anything in the .gitignore could be excluded from globsAll option for this rule.