nodejs/node

access the imported packages cache like require.cache

ganeshkbhat opened this issue · 8 comments

ganeshkbhat commented

Affected URL(s)

https://nodejs.org/api/esm.html#esm_no_require_cache

Description of the problem

How do i access the imported packages cache like require.cache?

I am trying to do something like this:

c.mjs

const tester = import("./test.js");
import * as fs from "fs";
import console from 'console';

// !!import.meta.cache is not an API as per docs - 
// https://github.com/nodejs/help/issues/2806
// How do i access the imported packages cache like require.cache
if (!!import.meta && !!import.meta.cache) {
    for (let p in import.meta.cache) {
        console.log(p);
        // console.log(trim(p));
    }
}

c.js

const tester = require("./test.js");

// current access of require.cache i am using
if (!!require.cache) {
    for (let p in require.cache) {
        console.log(p);
        // console.log(trim(p));
    }
}


function trim(p) {
    var re = /(.*?).js/;
    var basename = path.basename(p);
    var moduleName = re.exec(basename)[1];
    return [moduleName, p];
}

test.js

var path = require('path');
var c = require('fs');
var f = require('child_process');

This question comes after

https://stackoverflow.com/questions/74215997/how-do-i-do-console-log-in-esm-or-es6-file-without-having-a-package-json-in-the?noredirect=1#comment131031974_74215997

nodejs/help#2806 (comment)

nodejs/help#2806

Note:
I have read this:

I am aware of this docs - https://nodejs.org/api/esm.html#esm_no_require_cache

A simple hack is also fine.

aduh95 commented

The ESM cache is not exposed, and I believe this is per spec.

A simple hack is also fine.

In this case, you may like the following:

hack-loader.mjs:

globalThis.loadedURLs = [];

export function load(url, context, next) {
  globalThis.loadedURLs.push(url);
  return next(url, context);
}

c.mjs

const tester = import("./test.js");
import * as fs from "fs";
import console from 'console';

if (globalThis.loadedURLs) {
    for (let p of globalThis.loadedURLs) {
        console.log(p);
        // console.log(trim(p));
    }
}

If you run node --experimental-modules ./hack-loader.mjs c.mjs, you should get the result you are looking for. Note that this is a very hacky solution, and it will stop working as soon as we manage to get the loaders off-thread, so definitely not a long term solution.

I'm going to close this as I believe there's no bug to fix, but feel free to continue the discussion and ask more questions.

  • 👍1
ganeshkbhat commented

i have one more question. while this is a hack, does this help in capturing runtime dynamic imports (modules) when using a loader like this. can i use this hack in my project?

i am currently using a require/ import regex to check number of import statements, the modules and objects being accessed. here is the code base i am using for regex access:

  • import object from "module" - /import(?:[\s.*]([\w*{}\n\r\t, ]+)[\s*]from)?[\s*](?:["'](.*[\w]+)["'])?/gm
  • import("module") - /import\((?:["'\s]*([\w*{}\n\r\t, ]+)\s*)?["'\s](.*([@\w_-]+))["'\s].*\);$/mg + /(?:import\('?"?)(.*?)(?:'?"?\))/
  • require("module") - /(?:require\('?"?)(.*?)(?:'?"?\))/

https://github.com/ganeshkbhat/get-imports/blob/main/index.js

do you think it is wise to add this hack in the docs? it may help a lot of people with unnecessary "..." across comments. like https://stackoverflow.com/questions/74216420/access-the-imported-packages-and-modules-cache-like-require-cache

  • 👍1
ganeshkbhat commented

however, on a second, i believe this is a real opportunity area of exposing the import modules cache like the require.cache along with the ast. i am wishing to make a ast proposal to tc39 for scripting environments line nodejs, bun, etc even if not exposed in browsers. #45158 (comment) . however, i wished the imports cache was also included to be exposed in the server scripting environment.

aduh95 commented

i am currently using a require/ import regex to check number of import statements, the modules and objects being accessed. here is the code base i am using for regex access:

You are more likely to get better results using a JS parser, such as acorn, otherwise you would have false positive (e.g. import statements in strings, in comments, etc.) and if the spec changes the syntax you would have to rethink your regex.

however, on a second, i believe this is a real opportunity area of exposing the import modules cache like the require.cache along with the ast. i am wishing to make a ast proposal to tc39 for scripting environments line nodejs, bun, etc even if not exposed in browsers.

I don't see we could do that, AFAIK it's not something we have access to, making a request to the TC39 would indeed be the way to go, however I'd be surprised if it hasn't been discussed before.

@aduh95 Seems like the hack of loaders is not somehow working.

What is not working? What command are you using? What output do you see? What were you expecting to get instead?

  • 👍1
ganeshkbhat commented

I am looking at this. https://tc39.es/process-document/ what recommendations do you have for this. I probably need help to address the nodejs part of it. I would like if I have some subject matter exertise from you here.

ganeshkbhat commented

@aduh95 The normal loader commands are node --experimental-loader=./loader.mjs filename.mjs and I was expecting the output from console.log("Url + globalThis.loadedURLs: ", url, globalThis.loadedURLs) and console.log("globalThis.loadedURLs: ", p) using the https://github.com/ganeshkbhat/get-isesm/blob/main/loader.mjs and https://github.com/ganeshkbhat/get-isesm/blob/main/demos/loader.test.mjs respectively.

icetbr commented

This landed on node 22: --experimental-require-module. It allows mocha --watch to work with ESM modules, which was broken feature due to require.cache.

Here is the person who found the "fix"
https://joyeecheung.github.io/blog/2024/03/18/require-esm-in-node-js/

The docs
https://nodejs.org/api/modules.html#loading-ecmascript-modules-using-require

Here is where I first saw the merge
#51977 (comment)