On test re-run, Wallaby is rewriting `import` to `require`
vjpr opened this issue · 31 comments
Issue description or question
I have a simple test that looks like this in <projectCacheDir>/original:
import test from 'ava'
import index from './index'
//import foo from '@vjpr/obs.foo'
test('test', async t => {
console.log('hi2345678910')
//await index()
//t.assert(foo === 'foo')
t.assert(true)
})On initial run, in <projectCacheDir/instrumented I get the following which works as expected:
var $_$c = $_$wp(146);
import test from 'ava';
import foo from '@vjpr/obs.foo';
$_$w(146, 0, $_$c), test('test', async t => {
var $_$c = $_$wf(146);
$_$w(146, 1, $_$c), $_$tracer.log('hi2', '', 146, 1);
$_$w(146, 2, $_$c), t.assert(foo === 'foo');
});
$_$wpe(146);But then when I make a change and re-run the tests, import 'ava' is converted to a require:
var $_$c = $_$wp(146);
import foo from '@vjpr/obs.foo';
var test = ($_$w(146, 0, $_$c), require('ava')); // <-----------------
$_$w(146, 1, $_$c), test('test', async t => {
var $_$c = $_$wf(146);
$_$w(146, 2, $_$c), $_$tracer.log('hi23', '', 146, 2);
$_$w(146, 3, $_$c), t.assert(foo === 'foo');
});
$_$wpe(146);Only ava has this behavior. If I rename it to ava1, then it's an import.
This causes:
ReferenceError: require is not defined in ES module scope, you can use import instead
This file is being treated as an ES module because it has a '.js' file extension and './repos/vjpr/packages/obs/cli/package.json' contains "type": "module". To treat it as a CommonJS script, rename it to use the '.cjs' file extension.
at file:///Users/Vaughan/Library/Application%20Support/JetBrains/IntelliJIdeaVaughan/system/wallaby/projects/3947423c19a8649a/instrumented/repos/vjpr/packages/obs/cli/src/index.test.ava.js?update=1632845623648&wallaby=true:2:13
at ModuleJob.run (node:internal/modules/esm/module_job:183:25)
at async Loader.import (node:internal/modules/esm/loader:178:24)
at async importModuleDynamicallyWrapper (node:internal/vm/module:437:15)
What could be causing this?
Does wallaby do any import rewriting?
I also tried wrapping ava in my-ava, it didn't rewrite anymore, but now I do not get any inline console messages printed in IntelliJ.
Is ava4 + wallaby working with native es modules? Or do I need to transpile them to CJS?
Also, when compiling with SWC, are ranges really necessary? Would this break IntelliJ inline messages?
Does wallaby do any import rewriting?
Yes - By default for commonjs modules, Wallaby re-writes import test from 'ava' to use require to fix a bug/issue when using Babel.
When Wallaby's esm loader is used the import statements are not touched but because you have a custom loader in place, this logic was missing. We have released an updated version of Wallaby core (v1.0.1153) which should work for you with your custom loader.
I also tried wrapping ava in my-ava, it didn't rewrite anymore, but now I do not get any inline console messages printed in IntelliJ.
I think another way you may have been able to solve the problem is with:
import { test } from 'ava';Is ava4 + wallaby working with native es modules? Or do I need to transpile them to CJS?
Yes, we recently added support (in the last 2 weeks). You may also transpire them to CJS if you want. We had not expected anyone to use a custom loader, which I think is what has caused the problem in your case. We're keen to understand why you have this and to see a template for your ultimate setup. If we can automatically handle what you're doing we will (or at least update our docs for others who want to do the same thing).
We have released an updated version of Wallaby core (v1.0.1153) which should work for you with your custom loader.
What changes were made?
I still have to add this hook to my --experimental-loader and still don't get IDE line annotations.
// noinspection JSUnusedGlobalSymbols
export async function getSource(url, context, defaultGetSource) {
if (url.endsWith('ava/entrypoints/main.mjs')) {
return {
source: `
export default function() {
const runner = global.$_$tracer.avaRunner;
var avaModuleExports = runner.test || runner.chain.test || runner.chain;
return avaModuleExports.apply(this, arguments);
};
export function test() {
const runner = global.$_$tracer.avaRunner;
var avaModuleExports = runner.test || runner.chain.test || runner.chain;
return avaModuleExports.apply(this, arguments);
};
`,
};
}
return defaultGetSource(url, context, defaultGetSource);
}
We're keen to understand why you have this and to see a template for your ultimate setup.
I will setup a repro project now.
What changes were made?
We added support for using your own experimental loader, and not the Wallaby ava default.
We will wait to see the repro project, thanks.
Here is the repro: https://github.com/vjpr/issue-wallaby-ava
Just install with pnpm install.
Then run the test wallaby run configuration.
It's using swc as a wallaby compiler.
Also, when compiling with SWC, are ranges really necessary? Would this break IntelliJ inline messages?
Sorry - missed this at the end of your first message. Yes - the ranges are necessary, it is how Wallaby maps your code. I've been taking a look at how to get the ranges with SWC. It may be possible with an SWC Visitor plugin (https://swc.rs/docs/usage-plugin). I think it'll take me another 30-60 minutes or so to determine if it will work or not and will get back to you.
I have managed to get the ranges working by updating your swc compiler (see below). Right now, it's not how I would leave it in terms of efficiency, the rangeByIdxToLineColumn implementation is brute force for now and I think you could/should improve it. So Wallaby now starts to report things properly for me, but when I change the code, I lose coverage.
I'm going to try see why that's happening now.
import * as swc from "@swc/core";
import { Visitor } from "@swc/core/Visitor.js";
class RangeCapturer extends Visitor {
getAllFuncs(toCheck) {
const props = [];
let obj = toCheck;
do {
props.push(...Object.getOwnPropertyNames(obj));
} while ((obj = Object.getPrototypeOf(obj)));
return props.sort().filter((e, i, arr) => {
if (e != arr[i + 1] && typeof toCheck[e] == "function") return true;
});
}
constructor() {
super();
this.ranges = [];
this.getAllFuncs(this).forEach(funcName => {
if (funcName.startsWith('visit')) {
const oldVisit = this[funcName];
this[funcName] = function () {
if (arguments.length > 0) {
if (arguments[0] && arguments[0].span) {
this.ranges.push([arguments[0].span.start, arguments[0].span.end]);
}
}
return oldVisit.apply(this, arguments);
};
}
});
}
}
import { dirname } from "path";
////////////////////////////////////////////////////////////////////////////////
const swcConfig = {
test: ".tsx?$",
sourceMaps: true,
jsc: {
target: "es2020",
parser: {
syntax: "typescript",
tsx: true,
decorators: true,
dynamicImport: true,
},
baseUrl: "./src",
paths: {
//'@src/*': ['*'],
"@src/*": ["./src/*"],
"#src/*": ["./src/*"],
"#packages/*": ["./packages/*"],
},
},
module: {
//type: 'commonjs',
type: "es6",
},
};
////////////////////////////////////////////////////////////////////////////////
function rangeByIdxToLineColumn(source, idx) {
const content = source.substring(0, idx).replace(/\r\n/g, '\n').split('\n');
const line = content.length;
const column = content[content.length - 1].length;
return [line, column];
}
export default function (w, { shouldLog } = {}) {
return (file) => {
const { type, path, content, config } = file;
const filename = path;
if (shouldLog) console.log("Compiling:", filename);
const plugins = [];
const opts = {
sourceRoot: dirname(filename),
filename,
//jsc // Config options can be added here.
jsc: { ...swcConfig.jsc },
//plugin: swc.plugins(plugins),
};
const res = compile(filename, content, opts);
//console.log({filename, content, res})
return res;
};
}
////////////////////////////////////////////////////////////////////////////////
function compile(filename, code, opts) {
const defaultSourceMap = true;
//const defaultSourceMap = 'inline' // originally
const rangeCapturer = new RangeCapturer();
const finalOpts = {
...opts,
sourceMaps:
opts.sourceMaps === undefined ? defaultSourceMap : opts.sourceMaps,
plugin: (m) => rangeCapturer.visitProgram(m),
};
const output = swc.transformSync(code, finalOpts);
const ranges = rangeCapturer.ranges.map(([start, end]) => {
const startLineColumn = rangeByIdxToLineColumn(code, start);
const endLineColumn = rangeByIdxToLineColumn(code, end);
return [...startLineColumn, ...endLineColumn];
})
console.log('ranges', ranges);
return {
map: output.map,
code: output.code,
ranges
};
}I believe the problem is related to known behavior in SWC where it seems to append to span objects on subsequent executions. swc-project/swc#1366
Trying to work out how to fix it and then I think everything will be working well.
Thanks! Will try it out tomorrow.
I'm still not clear on what ranges are though even after reading the docs, and how they are used for mapping.
I assumed they were just something for code coverage.
Are they suppose to represent statements or expressions or...? How do they relate to source maps?
Will prob need to write in Rust for efficiency. The SWC JS plugin system is being deprecated in the future.
I can't get it working in JavaScript-land. It looks like once the native binary for swc is called, it reuses an internal buffer and no longer reports offsets in a way that can be understood by a JavaScript consumer. Perhaps you can get it working by writing something in rust, but unfortunately that's outside my skillset to help you with.
I'm still not clear on what ranges are though even after reading the docs, and how they are used for mapping.
JavaScript source-maps are lossy which means that if your code is transpiled before Wallaby runs, we can't rely on them to identify how the transformed code maps to your original code (it provides series of original start line, start column to transformed start line, start column). Each range represents the start and end (lines, cols) for each node in your original unmodified abstract syntax tree.
The range mappings (along with source maps) are what allow Wallaby to perform some of its more advanced features (such as inline console log). Inside Wallaby they are used to identify which lines of your code executed and where errors should be reported, inline values displayed, etc.
You could possibly create a substitute for ranges using source maps and some heuristics but I think there will be a number of errors and weird behaviors because of the lossy nature of source maps. It's normal for JavaScript parsers to include start/end positions for nodes in the AST (e.g. https://astexplorer.net) but in this case, swc is misreporting on subsequent executions.
The only other thing I can think that might work is to spawn a new process each time you need to compile using swc. On my machine this takes about 40ms per file. Wallaby will cache the files until they are changed so perhaps it's not a bad option if you really need/want swc. I would think at that point though, you may find an alternative like babel is actually faster.
Each range represents the start and end (lines, cols) for each node in your original unmodified abstract syntax tree.
From the wallaby compiler docs:
* ranges = [
* [ 1, 0, 1, 22 ], // whole statement
* [ 1, 12, 1, 15 ], // c() execution path
* [ 1, 18, 1, 21 ] // d() execution path
* ];
So if I understand correctly, the ranges are a depth-first search of the AST?
How does Wallaby know what is the format of my "unmodified AST"? What if SWC uses a different AST than Wallaby? Won't that confuse Wallaby?
For example using the @babel/parser,
How does it know to ignore the VariableDeclarator, VariableDeclarator, ConditionalExpression and to just use the CallExpression?
Should each range just be an execution path rather than nodes?
Program (0, 22)
- VariableDeclaration
- VariableDeclarator
- ConditionalExpression
- CallExpression (12, 15)
- Identifier
- CallExpression (18, 21)
- Identifier
You could possibly create a substitute for ranges using source maps and some heuristics
So I am attempting to do this with the source maps,
https://github.com/mozilla/source-map#sourcemapconsumerprototypeeachmappingcallback-context-order
const consumer = await new SourceMapConsumer(rawSourceMap)
const ranges = []
const spans = consumer.eachMapping(m => {
console.log(m)
const {originalLine, originalColumn} = m
ranges.push([originalLine, originalColumn])
})Mapping {
generatedLine: 1,
generatedColumn: 30,
lastGeneratedColumn: null,
source: 'xxx.ts',
originalLine: 1,
originalColumn: 30,
name: null
}
But I only get the originalLine, originalColumn, but still unsure about how to convert to ranges.
I guess the source map has no knowledge of the AST. So essentially, I would have to parse my original source to an AST (but which one?), but then the speed gains of swc parsing are lost at this point, and I would only benefit from the speed gains from transformation.
I think what I might try is adding an attribute to the source maps that swc generates called ranges.
{
"version":3,
"sources":["repos/vjpr/packages/obs/cli/src/fix-monitors/render-to-html.ts"],
"names":["renderToHtml"],
"mappings":"8BAA8B,YAAY,GAAG,CAAC;AAI9C,CAAC",
+ "ranges: [[x,x,x,x], [x,x,x,x]]
}
So if I understand correctly, the ranges are a depth-first search of the AST?
Yes - this is how Wallaby processes the ranges; I do not believe that the order matters though.
How does Wallaby know what is the format of my "unmodified AST"? What if SWC uses a different AST than Wallaby? Won't that confuse Wallaby?
Wallaby doesn't know this and nor does it need to. Wallaby actually parses the swc transformed code again after swc has processed it in order to determine live values, code coverage, add debugger support, etc. (there's a lot that Wallaby does). Wallaby uses a combination of the source maps along with the original ranges to determine values to show, etc. The source mapping in this case always maps to the start range correctly and then Wallaby's has extensive heuristics to determine which values to select and how to process them. So in effect it is using a combination of the original ranges along with the new/transformed AST + source maps to correctly process your file.
How does it know to ignore the VariableDeclarator, VariableDeclarator, ConditionalExpression and to just use the CallExpression?
This logic is part of Wallaby's heuristics when processing the swc transformed file. It doesn't matter that ranges are returned for these cases, and in fact, Wallaby uses some of those too (e.g. for VariableDeclarator, Wallaby knows to output the value that was assigned to the variable).
So I am attempting to do this with the source maps,
But I only get the originalLine, originalColumn, but still unsure about how to convert to ranges.
You would have to come up with some heuristics on how to process this... I'm not entirely sure what to suggest, perhaps you could map back to original source and try and determine where the token ends. Perhaps you could also sort the generated to original mappings and come up with another way to automatically generate the end position. Either way, it's not going to be perfect and will result in some weird behaviors.
I guess the source map has no knowledge of the AST. So essentially, I would have to parse my original source to an AST (but which one?), but then the speed gains of swc parsing are lost at this point, and I would only benefit from the speed gains from transformation.
Correct. I'm not sure what this performance looks like.
I think what I might try is adding an attribute to the source maps that swc generates called ranges.
I'm guessing you will need to fork swc to do this? If you were to take this approach, perhaps it's better to try and fix / change this behavior: swc-project/swc#1366?
I also don't imagine that the swc team would accept a pull request that enhances the source-map format. I'm not sure what the best approach is to integrate "single-swc parse" support to provide the original ranges.
Depending on what you are trying to do and what testing framework you want to use, there's another option for you to consider. We (the Wallaby team) actually use Jest with swc for some of our projects using @swc/jest. Unlike creating a custom compiler, this works in the context of jest because of how jest transformers work (effectively transforms can be fed into one another). While jest may be a little slower to start up, generally the TypeScript compilation time is much larger and once jest is started with Wallaby, feedback is fast.
You may be interested to read this blog post: https://wallabyjs.com/blog/optimizing-typescript.html
TL;DR:
Medium-size project (22,000 LOC, 135 files)
- Jest +
ts-jest:13.37 secondscompile time - Jest +
ts-jest(isolated modules):9.09 secondscompile time - Jest +
babel(typescript transform):2.26 secondscompile time - Jest +
swc:0.461 secondscompile time
For this project, Wallaby test execution time (approximately 700 tests) with swc without any cached files is 4.45 seconds, and 2.5 seconds after tests have been run before. Single test execution after a change is usually < 100ms.
Without understanding how you want to set up your project, I would consider opting for something a little more standard (e.g. jest with swc) vs. writing my own compiler, modifying swc code-base, adding ranges, complex Wallaby configuration, etc.
^^^ Haven't read your previous post, will respond soon.
I have opened a PR with SWC to traverse the AST and return ranges. See: swc-project/swc#2320.
I also found the bug that incremented the index, which I have noted to the maintainer here: swc-project/swc#1366 (comment).
Here is the visitor code I am using: https://github.com/swc-project/swc/pull/2320/files#diff-b1a35a68f14e696205874893c07fd24fdb88882b47c23cc0e0c80a30c7d53759R1116-R1138
Maybe you can suggest how this visitor should look? Should I just visit every expression? Or every node?
How do I know its working?
I just see this error which I think means the ranges are not correct:
2021-09-30T02:17:27.782Z workers Failed to map the stack to user code, entry message: false, stack: AssertionError
at ExecutionContext.assert (/xxx/node_modules/.pnpm/ava@4.0.0-alpha.2_supports-color@9.0.1/node_modules/ava/lib/assert.js:959:11)
at file:///Users/Vaughan/Library/Application%20Support/JetBrains/IntelliJIdeaVaughan/system/wallaby/projects/3947423c19a8649a/instrumented/repos/vjpr/packages/obs/cli/lib/index.test.ava.js?update=1632968247772&wallaby=true:6:27
at Test.callFn (/xxx/node_modules/.pnpm/ava@4.0.0-alpha.2_supports-color@9.0.1/node_modules/ava/lib/test.js:578:21)
at Test.run (/xxx/node_modules/.pnpm/ava@4.0.0-alpha.2_supports-color@9.0.1/node_modules/ava/lib/test.js:591:23)
at Test.run (/Users/Vaughan/Library/Application Support/JetBrains/IntelliJIdeaVaughan/system/wallaby/wallaby/runners/node/ava@1.0.0/initializer.js:14:6586)
at Runner.runSingle (/xxx/node_modules/.pnpm/ava@4.0.0-alpha.2_supports-color@9.0.1/node_modules/ava/lib/runner.js:270:33)
...reading your response now.
Maybe you can suggest how this visitor should look? Should I just visit every expression? Or every node?
I think you should visit every node; if you just do expressions you'll find some parts of Wallaby don't work properly anymore. For example, if using the debugger, you can select a function parameter and Wallaby will output its value. If you were only outputting expressions, this wouldn't work. The example I provided yesterday (using all visit functions) seemed to work until the subsequent execution because of swc span behavior.
Wallaby doesn't know this and nor does it need to.
Interesting. I had a feeling there was some magic going on there.
So it should be enough to just provide an unordered array of ranges of every node?
Perhaps I should use the visit_span from swc...https://github.com/swc-project/swc/blob/6a41e9a0bea739e55ca3f61e230b9ab07d6b6f3e/common/src/syntax_pos.rs#L17-L34
Depending on what you are trying to do and what testing framework you want to use, there's another option for you to consider.
The issue I have is I am using pnpm and it doesn't play well with jest-haste-map because of its symlinking to a virtual store of node_modules.
So it should be enough to just provide an unordered array of ranges of every node?
I think it would be ideal to provide in the same way that wallaby is (depth-first tree traversal of AST) but I've just gone through our code base and it doesn't seem like this matters. If you run into problems, we would be happy to investigate for you.
Perhaps I should use the visit_span from swc
Yes - this what I had originally thought the JS code would do yesterday but the start/end changes on subsequent executions as you know...
The issue I have is I am using pnpm and it doesn't play well with jest-haste-map because of its symlinking to a virtual store of node_modules.
Fair enough... I'm not familiar enough with pnpm to help.
I'm interested to see what the swc story for visiting the AST ends up being. From what I read of their issues, they want to do away with the JavaScript API (which makes sense from a performance perspective) but I'm not sure what it then looks like for you to walk the AST in Rust and still integrate with other tooling written in JavaScript.
Hmmm... just thinking of another work-around. The position seems offset by the source length of previous transforms so you should be able to track this and account for it over time.
I'm going to have a quick play in the repo that you provided yesterday and see if I can get that working.
I'm interested to see what the swc story for visiting the AST ends up being.
I think the idea is that every tool would be re-written in Rust. Allowing JS code to traverse the AST would just slow it down and add too much maintenance. Rust is very tricky for newcomers to write plugins so we will see if this works.
I am interested to follow Rome which is now doing their own Rust-based parser/compiler/linter.
The position seems offset by the source length of previous transforms
Yep, exactly. I thought of the same, but the API is being deprecated, and it would be slow. But I guess before I can get a PR merged it would be a good workaround.
OK, so I got it working with the code below. The important part is this:
function compile(filename, code, opts) {
const defaultSourceMap = true;
//const defaultSourceMap = 'inline' // originally
const rangeCapturer = new RangeCapturer();
+ let spanStart;
const finalOpts = {
...opts,
sourceMaps:
opts.sourceMaps === undefined ? defaultSourceMap : opts.sourceMaps,
plugin: (m) => {
+ spanStart = m.span.start;
return rangeCapturer.visitProgram(m)
},
};
const output = swc.transformSync(code, finalOpts);
const ranges = rangeCapturer.ranges.map(([start, end]) => {
const startLineColumn = rangeByIdxToLineColumn(code, start - spanStart);
const endLineColumn = rangeByIdxToLineColumn(code, end - spanStart);
return [...startLineColumn, ...endLineColumn];
})
return {
map: output.map,
code: output.code,
ranges
};
}Again, in my code, the idx to line/column mapping is brute force for now but you can optimise that if you want to go with this approach. It'll let you use SWC as it exists right now.
I'll close the ticket out for now as I think this gets everything working but obviously you may like to vary your approach.
packages/swc-compiler/index.js
import * as swc from "@swc/core";
import { Visitor } from "@swc/core/Visitor.js";
class RangeCapturer extends Visitor {
getAllFuncs(toCheck) {
const props = [];
let obj = toCheck;
do {
props.push(...Object.getOwnPropertyNames(obj));
} while ((obj = Object.getPrototypeOf(obj)));
return props.sort().filter((e, i, arr) => {
if (e != arr[i + 1] && typeof toCheck[e] == "function") return true;
});
}
constructor() {
super();
this.ranges = [];
this.getAllFuncs(this).forEach(funcName => {
if (funcName.startsWith('visit')) {
const oldVisit = this[funcName];
this[funcName] = function () {
if (arguments.length > 0) {
if (arguments[0] && arguments[0].span) {
this.ranges.push([arguments[0].span.start, arguments[0].span.end]);
}
}
return oldVisit.apply(this, arguments);
};
}
});
}
}
import { dirname } from "path";
////////////////////////////////////////////////////////////////////////////////
const swcConfig = {
test: ".tsx?$",
sourceMaps: true,
jsc: {
target: "es2020",
parser: {
syntax: "typescript",
tsx: true,
decorators: true,
dynamicImport: true,
},
baseUrl: "./src",
paths: {
//'@src/*': ['*'],
"@src/*": ["./src/*"],
"#src/*": ["./src/*"],
"#packages/*": ["./packages/*"],
},
},
module: {
//type: 'commonjs',
type: "es6",
},
};
////////////////////////////////////////////////////////////////////////////////
function rangeByIdxToLineColumn(source, idx) {
const content = source.substring(0, idx).replace(/\r\n/g, '\n').split('\n');
const line = content.length;
const column = content[content.length - 1].length;
return [line, column];
}
export default function (w, { shouldLog } = {}) {
return (file) => {
const { type, path, content, config } = file;
const filename = path;
if (shouldLog) console.log("Compiling:", filename);
const plugins = [];
const opts = {
sourceRoot: dirname(filename),
filename,
//jsc // Config options can be added here.
jsc: { ...swcConfig.jsc },
//plugin: swc.plugins(plugins),
};
const res = compile(filename, content, opts);
//console.log({filename, content, res})
return res;
};
}
////////////////////////////////////////////////////////////////////////////////
function compile(filename, code, opts) {
const defaultSourceMap = true;
//const defaultSourceMap = 'inline' // originally
const rangeCapturer = new RangeCapturer();
let spanStart;
const finalOpts = {
...opts,
sourceMaps:
opts.sourceMaps === undefined ? defaultSourceMap : opts.sourceMaps,
plugin: (m) => {
spanStart = m.span.start;
return rangeCapturer.visitProgram(m)
},
};
const output = swc.transformSync(code, finalOpts);
const ranges = rangeCapturer.ranges.map(([start, end]) => {
const startLineColumn = rangeByIdxToLineColumn(code, start - spanStart);
const endLineColumn = rangeByIdxToLineColumn(code, end - spanStart);
return [...startLineColumn, ...endLineColumn];
})
return {
map: output.map,
code: output.code,
ranges
};
}Great, thanks for you help!
Did you test this in IntelliJ?
I'm only seeing inline annotations at the top of the file.
For this file my ranges look like:
ranges [
[ 1, 0, 6, 2 ], [ 1, 0, 6, 2 ], [ 1, 0, 1, 22 ],
[ 1, 0, 1, 22 ], [ 1, 0, 1, 22 ], [ 1, 17, 1, 22 ],
[ 1, 7, 1, 11 ], [ 1, 7, 1, 11 ], [ 1, 7, 1, 11 ],
[ 1, 7, 1, 11 ], [ 3, 0, 6, 2 ], [ 3, 0, 6, 2 ],
[ 3, 0, 6, 2 ], [ 3, 0, 6, 2 ], [ 3, 0, 6, 2 ],
[ 3, 0, 3, 4 ], [ 3, 0, 3, 4 ], [ 3, 0, 3, 4 ],
[ 3, 0, 3, 4 ], [ 3, 5, 3, 11 ], [ 3, 5, 3, 11 ],
[ 3, 13, 6, 1 ], [ 3, 13, 6, 1 ], [ 3, 24, 6, 1 ],
[ 3, 24, 6, 1 ], [ 4, 2, 4, 19 ], [ 4, 2, 4, 19 ],
[ 4, 2, 4, 19 ], [ 4, 2, 4, 19 ], [ 4, 2, 4, 13 ],
[ 4, 2, 4, 13 ], [ 4, 2, 4, 13 ], [ 4, 2, 4, 9 ],
[ 4, 2, 4, 9 ], [ 4, 2, 4, 9 ], [ 4, 2, 4, 9 ],
[ 4, 10, 4, 13 ], [ 4, 10, 4, 13 ], [ 4, 10, 4, 13 ],
[ 4, 14, 4, 18 ], [ 4, 14, 4, 18 ], [ 5, 2, 5, 17 ],
[ 5, 2, 5, 17 ], [ 5, 2, 5, 17 ], [ 5, 2, 5, 17 ],
[ 5, 2, 5, 10 ], [ 5, 2, 5, 10 ], [ 5, 2, 5, 10 ],
[ 5, 2, 5, 3 ], [ 5, 2, 5, 3 ], [ 5, 2, 5, 3 ],
[ 5, 2, 5, 3 ], [ 5, 4, 5, 10 ], [ 5, 4, 5, 10 ],
[ 5, 4, 5, 10 ], [ 5, 11, 5, 16 ], [ 5, 11, 5, 16 ],
[ 3, 19, 3, 20 ], [ 3, 19, 3, 20 ], [ 3, 19, 3, 20 ]
]
Same behavior with your JS visitor, as well as Rust visitor.
Maybe it's that the source maps are wrong because of that bug...
Did you test this in IntelliJ?
I didn't but it won't make a difference.
You may need to force your Wallaby cache to reset. Try changing your wallaby.ava.js config file (add a whitespace) and restart to see if it helps.
Ok the issue seems to be that my source maps coming from swc were messed up. Everything was on line 1. Must have been something from my hacking on @swc/core.
Would it be possible to share the ast -> ranges code used in wallaby.compilers.babel?
I would prefer all my code to only be transformed with SWC, and was thinking of just parsing with Babel or ESBuild to generate the ranges. Getting ranges from SWC will not be possible until their Rust plugin API is ready.
UPDATE: Wrote my own babel compiler because I needed async ESM configuration (#2821).
Just used @babel/traverse to generate the ranges.
Hope this works...although maybe there are some edge-cases your impl covers. The benefit is that I can use this to parse the AST and generate ranges while using SWC as the transformer.
import {transformFileAsync, parseAsync} from '@babel/core'
import traverse from '@babel/traverse'
const ast = await parseAsync(src, opts)
function getRangesFromBabelAst(ast) {
let ranges = []
traverse(ast, {
enter(path) {
//console.log(path.node)
const loc = path.node.loc
if (!loc) return
ranges.push([
loc.start.line,
loc.start.column,
loc.end.line,
loc.end.column,
])
},
})
return ranges
}Also a question:
From https://wallabyjs.com/blog/optimizing-typescript.html:
Wallaby uses multiple worker processes to run your tests in parallel whereas TypeScript compilation is limited to a single process. For most testing frameworks that Wallaby supports, compilation from TypeScript to JavaScript occurs in a single process
Whenever I start Wallaby it starts a ton of processes. Is this the "processor pool" that is mentioned? Do these not compile files in parallel? What are these processors doing?
Is there a way to limit them too...it usually starts 10+ node processes using 100% CPU.
2021-10-01T00:42:57.941Z workers Parallelism for initial run: 1, for regular run: 1
...
2021-10-01T00:42:57.962Z project Starting processor pool (if not yet started)
2021-10-01T00:42:58.042Z project File babel.config.js is not instrumented by core
2021-10-01T00:42:58.043Z project Preparing to process pnpm-workspace.yaml
2021-10-01T00:42:58.043Z project Starting processor pool (if not yet started)
2021-10-01T00:42:58.081Z project Preparing to process repos/vjpr/packages/obs/cli/src/fix-monitors/web/render.ts
2021-10-01T00:42:58.081Z project Starting processor pool (if not yet started)
2021-10-01T00:42:58.084Z project Preparing to process repos/vjpr/packages/obs/cli/src/server/router.ts
2021-10-01T00:42:58.084Z project Starting processor pool (if not yet started)
2021-10-01T00:42:58.085Z project Preparing to process repos/vjpr/packages/obs/cli/package.json
2021-10-01T00:42:58.085Z project Starting processor pool (if not yet started)
2021-10-01T00:42:58.085Z project Preparing to process repos/vjpr/packages/obs/cli/src/cli/index.ts
2021-10-01T00:42:58.085Z project Starting processor pool (if not yet started)
2021-10-01T00:42:58.086Z project Preparing to process repos/vjpr/packages/obs/cli/src/index.test.ava.ts
2021-10-01T00:42:58.086Z project Starting processor pool (if not yet started)
Wallaby uses multiple worker processes to run your tests in parallel whereas TypeScript compilation is limited to a single process.
This is the case by default, unless you are using isolatedModules, babel, or swc.
Whenever I start Wallaby it starts a ton of processes. Is this the "processor pool" that is mentioned? Do these not compile files in parallel? What are these processors doing?
With a custom compiler, yes, these are compiling in parallel.
Is there a way to limit them too...it usually starts 10+ node processes using 100% CPU.
We do not currently have a setting for this. The number of processes used is equal to os.cpus().length. On initial startup, if the files need to be compiled, then Wallaby tries to compile them as quickly as possible (hence 100% CPU). After initial startup, Wallaby remembers that they have been compiled before (even on subsequent restarts) and you shouldn't see the high CPU use again.
Are you seeing something different? E.g. high CPU on every start?
If it is behaving as expected (i.e. is only really a first time starting on project or after config change), I'm assuming that you've noticed the problem because you've been writing a compiler and it's probably OK to leave as is. If not, please let us know.
Are you seeing something different? E.g. high CPU on every start?
Nope. It's working as expected.