jspm/jspm.io

Loading modules from endpoints

Closed this issue · 45 comments

probins commented

As mentioned on es6 module loader issue, I'm having problems accessing jspm.io directly from the System loader:

If I try for example System.import('https://npm.jspm.io/es6-promise'); it returns the first file correctly with export * from "npm:es6-promise@0.1.1/dist/commonjs/main";. However, it then says "XMLHttpRequest cannot load npm:es6-promise@0.1.1/dist/commonjs/main.js. Cross origin requests are only supported for HTTP". However, if I then try to access with http, I get an empty response.

Similarly, if I try to access a file directly with System.import('xxx',{address:...}).

The msg I reported on the es6 loader issue was probably due to my requesting the wrong address, and Chrome was reporting this in a misleading way.

guybedford commented

It is trying to load npm: as a protocol, thinking it doesn't have secure access to the npm: protocol.

Set up "npm" as a path for this to work, with:

  System.paths['npm:*'] = 'https://npm.jspm.io/*.js';
probins commented

so 'paths' is a replacement for the 'endpoint' config, plus wildcard support.

Afraid I'm finding switching from jspm loader to System isn't as straightforward as I'd anticipated :-(

guybedford commented

Yes exactly.

What are you getting caught on?

guybedford commented

The config for System to work like jspm is:

  System.paths['~/*'] = System.baseURL + '*.js';
  System.baseURL = 'https://registry.jspm.io';
  System.paths['npm:*'] = 'https://npm.jspm.io/*.js';
  System.paths['github:*'] = 'https://github.jspm.io/*.js';
probins commented

this in turn means:

  • I'm reliant on non-standard (yet?) paths
  • I cannot simply do <module src="npm.github.io..."> as I planned

Is there no way round this? Can it not return export * from "https://npm.jspm.io/...";? It significantly reduces the attraction of jspm as a module repository if I have to set up paths like this in every page before a <module>.

probins commented

changed issue title

guybedford commented

The following should work:

  <script type="module" name="npm:underscore" address="https://npm.jspm.io/underscore.js"></script>

I may still bring the paths configurations back into the loader, as you are right, it would make for a cleaner upgrade path. This way changing internal APIs wouldn't affect user code.

Working on this stuff!

probins commented

It is trying to load npm: as a protocol, thinking it doesn't have secure access to the npm: protocol.

to put it another way: if the server doesn't know what to do with the protocol, then surely the protocol/url path config should be on the server not on the client

guybedford commented

By configuring the paths in the loader itself, this becomes a non-issue (which will be the case in jspm).

I was simply trying to put together a loader others can use that wasn't immediately tied to jspm by default, in the name of openness. But perhaps this isn't necessary.

So one would load a loader, that would just work with npm:underscore paths etc out the box.

probins commented

my ideal would be the ability to fetch modules from jspm's server using the <module> tag that will work with browser-implemented loaders, without the need for a custom loader. Ok, for the moment, I have to use a polyfill, but that's not a long-term need.

probins commented

The following should work:

<script type="module" name="npm:underscore" address="https://npm.jspm.io/underscore.js"></script>

same problem: "cannot load npm:underscore@1.5.2/underscore.js"

guybedford commented

Yes, with the paths configuration.

guybedford commented

Ultimately, it should be possible to do something like this:

<html>
  <script> System.paths['npm:*'] = 'https://npm.jspm.io/*.js'; </script>

  <module name="npm:underscore"></module>

Without the paths config, one would effectively be loading module names that are URLs, which is not supported by the spec, unless there is a spec change here. Happy to describe the problem further - it is not one I introduced at all.

guybedford commented

That said, for module names to work in a registry, I am very much for a paths configuration system, as I would much rather write:

  import "npm:underscore";

than

  import "https://npm.jspm.io/underscore.js"

It's one or the other, for everyone on jspm.

probins commented

but that hardwires 'npm:' into module names. As I see it, I should be able to do:

<script type="module" name="underscore" src="https://npm.jspm.io/underscore.js"></script>

to preload the module, and then refer to it as:

import "underscore";

I don't see the need for any config

guybedford commented

Names need to be hardwired as they are the true globals of dependency management. We need to all refer to the same underscore and know that that can be trusted. The jspm naming system is specially chosen to work in a global dependency scenario, so that dependencies can be shared between entirely different packages.

Consider that npm:lib might be a completely different library to bower:lib. The jspm registry becomes a way to ensure that global names are the same (underscore is typically the jspm registry), but we still need the ability to refer to specific registries in the edge cases.

If underscore is dependent on fs, we mean npm:fs not fs in the jspm registry.

guybedford commented

The only way to avoid making paths entirely necessary, would be to stop supporting multiple registries and only support a single jspm server for all modules:

  https://registry.jspm.io/module-name.js

This would require every dependency loaded to be in the jspm registry. Perhaps we could create npm modules as npm/underscore at:

  https://registry.jspm.io/npm/underscore.js

And instead move to a syntax of:

  import "npm/underscore";

It's a big shift, for a complaint over a single line of code, which actually offers a lot of flexibility.

guybedford commented

One still has to set the baseURL regardless:

<script>
  System.baseURL = 'https://registry.jspm.io';
</script>

So I'm not sure what the real gain is here at all.

guybedford commented

Even if I did return export * from "https://npm.jspm.io/..."; this would cause an error in the System loader, by design of the System specification as described currently (jorendorff/js-loaders#55 (comment)).

guybedford commented

Sorry, I'm getting defensive based on the technicalities.

In reality, I see it working this way:

  1. ES6 modules not supported in browsers - use SystemJS, either hosted from jspm:

      <script src="https://jspm.io/system.js"></script>

    Or a self install:

       <script src="system.js"></script>
   <script>
        System.baseURL = 'https://registry.jspm.io';
        System.paths['npm:*'] = 'https://npm.jspm.io/*.js';
   </script>

    Note that it is only the self install that needs paths specified.

  2. ES6 modules supported in browsers, but not all - use SystemJS on older browsers only:
    jspm will automatically feature detect and use SystemJS only if necessary, falling back to the standard window.System:

       <script src="https://jspm.io/system.js"></script>

    If self-hosting, we can do something like:

       <script>
       if (!window.System)
          document.write('<' + 'script src="system.js"><' + '/script>');
   </script>
   <script>
        System.baseURL = 'https://registry.jspm.io';
        System.paths['npm:*'] = 'https://npm.jspm.io/*.js';
   </script>

  3. ES6 modules supported in all browsers. On jspm we just do:

        <script src="https://jspm.io/system.js"></script>

    jspm system.js now only includes the paths config for registries, and the patch for plugin and map support.

    If self hosting, we only now need to provide jspm paths.

I think this sort of path makes a lot of sense, and don't see any issues as we are providing flexibility all the way towards the ideal use case.

If you have further concerns I'd be happy to hear them.

probins commented

yeah, that's a neat solution. I really didn't like the idea of having to define the paths on each page, but if I can load an adapted version of System.js from jspm.io, that removes that need. There remains the case when there's no longer any need to support legacy module types - but that's not an urgent requirement :-) and is a simple add-on.

One further issue here: if I have the paths all set up with standard system.js, I can do for example System.import('github:enyo/dropzone') but doing System.import('skrollr') fails, and System.import('hammer') tries to load https://github.jspm.io/EightMedia/hammer.js.js. Shouldn't this work?

What about cdn:?

guybedford commented

System.import('hammer') should work if the baseURL is set to the jspm registry (System.baseURL = 'https://registry.jspm.io/'). Similarly for skrollr. These should both be working (as they are both in the list of libraries tested and working on jspm!), if not I will look into it.

probins commented

just tried again: System.import('intro') does work, but System.import('skrollr') and System.import('hammer') don't. To be precise, the initial file load is ok, but for example https://github.jspm.io/Prinzhorn/skrollr.js produces an empty file, so possibly this is a different issue?

probins commented

and I see that if I load from the registry like this, I get an entry for both 'underscore' and 'npm:underscore', so presumably other modules can import from either.

guybedford commented

Ok I will look into those two.

Yes exactly, the registry allows the generic form of the name to be used.

probins commented

and I can't work out how to load from cdnjs either, even if I add a path

guybedford commented

I've got skrollr and hammer.js working now. This is actually due to a dependency conflict between Traceur and the ES6 Module Transpiler for builds, so I've had to temporarily disable ES6 builds on the server until this can be resolved.

To use cdnJS, try:

  System.paths['cdnjs:*'] = 'https://cdnjs.cloudflare.com/ajax/libs/*.js';

I'm actually going to start leaving it out of jspm, because one always has to manually include shim config to make any libraries work through cdnjs. Rather, I'm moving towards shim being included in the global files themselves of the form:

  "global";
  "import jquery";
  "exports my.global";
  // ... global script code

You are still welcome to add the path in your own apps though.

probins commented

The config for System to work like jspm is:

  System.paths['~/*'] = System.baseURL + '*.js';
  System.baseURL = 'https://registry.jspm.io';

well, not really. Previously, I was setting baseURL to lib/ or dist/ but this no longer works. I'm wondering whether it wouldn't be better to have both a baseURL and registryURL, as with jspm-loader

probins commented

and another issue here is that I want to change over to using address/src= but if I do this with relative addressing - src="./mylib/mymodule.js" - then it tries to load this from the registry.

guybedford commented

RegistryURL and baseURL have switched again, so if your previous config was:

System.registryURL = 'https://registry.jspm.io';
System.baseURL = 'lib/';

Then the new config would be:

System.baseURL = 'https://registry.jspm.io';
System.paths['~/'] = 'lib/.js';

On 9 January 2014 17:51, Peter Robins notifications@github.com wrote:

The config for System to work like jspm is:

System.paths['~/'] = System.baseURL + '.js';
System.baseURL = 'https://registry.jspm.io';

well, not really. Previously, I was setting baseURL to lib/ or dist/ but
this no longer works. I'm wondering whether it wouldn't be better to have
both a baseURL and registryURL, as with jspm-loader

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3#issuecomment-31945063
.

probins commented

but that won't be compatible with <module src="">. Browsers know what src="lib/mymodule.js" means; they don't know what ~/lib/mymodule.js means. If I use ~ in src now, I'll have to change it when browsers support natively. If I use relative addresses now, I have to set baseURL to 'lib/' or whatever, but then the registry lookups won't work. I would have to use full urls, which makes having lib and dist difficult. If I set baseURL to the registry, then it will look for the relatively addressed modules on the registry, which won't work either.

guybedford commented

For now, use dynamic imports:

<script>System.import('~/lib/mymodule')</script>

Later on, when the spec is updated to work properly, you will be able to
use the module tag:

But at the moment this still isn't specified properly to do the fetch, as
discussed previously.

On 10 January 2014 00:55, Peter Robins notifications@github.com wrote:

but that won't be compatible with . Browsers know what
src="lib/mymodule.js" means; they don't know what ~/lib/mymodule.jsmeans. If I use ~ in src now, I'll have to change it when browsers support
natively. If I use relative addresses now, I have to set baseURL to 'lib/'
or whatever, but then the registry lookups won't work. I would have to use
full urls, which makes having lib and dist difficult. If I set baseURL to
the registry, then it will look for the relatively addressed modules on the
registry, which won't work either.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3#issuecomment-31986108
.

probins commented

System.baseURL = 'https://registry.jspm.io';
System.paths['~/'] = 'lib/.js';

that will be relative to the registry, though. I can fiddle around with the string

System.paths['~/*'] = System.paths['~/*'].replace('*', 'lib/*'); // or dist/

though it's probably easier to just not use system-jspm.js but set my own paths

probins commented

If I use ~ in src now

which I can't do anyway, as the paths config applies to names not src/address. For relative ('local') addresses to work, baseURL can't be set to the registry. So I come back to my original suggestion of keeping a separate path config for registryURL

guybedford commented

No, the paths targets are URLs!

So System.paths['~/'] = 'lib/.js';

Means map the any "~/..." module name into the "lib/....js" URL. If
anything isn't behaving as expected though let me know.

On 10 January 2014 14:22, Peter Robins notifications@github.com wrote:

System.baseURL = 'https://registry.jspm.io';
System.paths['~/'] = 'lib/.js';

that will be relative to the registry, though. I can fiddle around with
the string

System.paths['/_'] = System.paths['/'].replace('', 'lib/_'); // or dist/

though it's probably easier to just not use system-jspm.js but set my own
paths

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3#issuecomment-32023217
.

guybedford commented

I still don't understand why you are using addresses at all, it is a flawed
concept in the spec.

Use names!

On 10 January 2014 14:41, Peter Robins notifications@github.com wrote:

If I use ~ in src now

which I can't do anyway, as the paths config applies to names not
src/address. For relative ('local') addresses to work, baseURL can't be set
to the registry. So I come back to my original suggestion of keeping a
separate path config for registryURL

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3#issuecomment-32024242
.

probins commented

Means map the any "~/..." module name into the "lib/....js" URL

relative to baseURL. So if I do

System.paths['~/*'] = 'lib/*.js';
System.import('~/myModule');

it looks for https://registry.jspm.io/lib/myModule.js

guybedford commented

Ahh, yes of course.

This URL being relative to the baseURL may still change depending on the
spec.

I would advise using absolute paths, or system-jspm to ensure the upgrade
path works.

So the issue is that it isn't easy to set the ~ path because it becomes
relative to the baseURL instead of the current page?

Can you not just do:

System.paths['~/'] = "/lib/.js"

?

Thanks - will make sure to document this path should be specified in this
way.

On 10 January 2014 16:14, Peter Robins notifications@github.com wrote:

Means map the any "~/..." module name into the "lib/....js" URL

relative to baseURL. So if I do

System.paths['/'] = 'lib/.js';
System.import('/myModule');

it looks for https://registry.jspm.io/lib/myModule.js

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3#issuecomment-32030070
.

probins commented

I still don't understand why you are using addresses at all, it is a flawed
concept in the spec.

I don't agree. For me, separating load from dependency definition in the browser is a feature not a flaw (though I'm not sure all the use cases have been fully thought through). Modules defining a dependency (import from xxx) should be agnostic about how that module is loaded and where it comes from. That way, apps/web pages are free to preload modules whenever and from wherever they like. If modules are bundled, then the filename may well bear no resemblance to the individual module name: if bundle xyz.js includes module a, then modules should 'import from a' not require that 'a' should somehow be converted to a url. Different sites will probably bundle modules in different ways.

As these bundles probably don't need to be referred to again, they can be anonymous, and I can load them with System.module({address: xxx}) once the issue with that is resolved. In the meantime, I can fake this with:

System.fetch({address: 'xxx.js', metadata: {}}).then(function(source){System.module(source);})

Same result as import() but they don't clutter up the module registry.

guybedford commented

There is a distinction between development and production, just like
RequireJS.

In production there needs to be some way to indicate that a module name
should be loaded from a specific bundle.

This will be the next addon to SystemJS, using a syntax like:

System.bundles = {
'my-bundle': ['jquery', 'github:some/module', ...]
};

Then whenever there is a require to a name that corresponds to bundle, the
bundle itself would be loaded first, itself corresponding to a module name.

In terms of being able to fetch from an address, I personally would prefer
it if URLs were permitted as module names.

This way, one could simply do:

System.import('http://www.some-site.com/file.js').then(function(m) { ...
});

URLs can be distinguished by the existence of a protocol (something://) to
make this exception work.

The address argument just feels clumsy to me.

On 11 January 2014 13:53, Peter Robins notifications@github.com wrote:

I still don't understand why you are using addresses at all, it is a
flawed
concept in the spec.

I don't agree. For me, separating load from dependency definition in the
browser is a feature not a flaw (though I'm not sure all the use cases have
been fully thought through). Modules defining a dependency (import from
xxx) should be agnostic about how that module is loaded and where it comes
from. That way, apps/web pages are free to preload modules whenever and
from wherever they like. If modules are bundled, then the filename may well
bear no resemblance to the individual module name: if bundle xyz.js
includes module a, then modules should 'import from a' not require that 'a'
should somehow be converted to a url. Different sites will probably bundle
modules in different ways.

As these bundles probably don't need to be referred to again, they can be
anonymous, and I can load them with System.module({address: xxx}) once the
issue with that is resolved. In the meantime, I can fake this with:

System.fetch({address: 'xxx.js', metadata: {}}).then(function(source){System.module(source);})

Same result as import() but they don't clutter up the module registry.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3#issuecomment-32094065
.

probins commented

The address argument just feels clumsy to me.

yes, I get the impression this was tacked on to the spec at the end. I'm not sure about urls as names though. I think I would prefer a different function loadFromUrl(xxx) or whatever

probins commented

I'm now doing this in a different way: keeping baseURL relative and setting up a jspm endpoint:

System.baseURL = System.baseURL + 'lib/'; // can be changed to dist for production
System.paths['jspm:*'] = 'https://registry.jspm.io/*.js';

then if I need something from jspm, I do !jspm:json or jspm:underscore. I quite like this, as it's consistent with the other endpoints. I think it's better if you mainly want to access internal modules and only sometimes externally from jspm. Either way, the system's flexible enough to be able to handle different ways of loading things.

I think all the issues here are now resolved, so I'll close this issue.

guybedford commented

See the proposal I just made here -
jorendorff/js-loaders#100, which would resolve
this quite nicely.

For a long time I went with the idea of the jspm path for the jspm
registry ('jspm:jquery' etc), but the feedback I got was that users really
do just want to write:

import "jquery";

and not:

import "jspm:jquery";

So the current method I use is to rather define the local code path as
"~/..." and then allow easy requires.

This is also most similar to the way one write NodeJS requires:

require('fs') // library code
require('./local-module') // local code

The only difference is that "~/..." is used instead of "./...", but if the
proposal above is accepted, then this would be identical.

If you are downloading your own libraries to a lib folder, then certainly
make that the baseURL. But then perhaps consider using the jspm CLI to
manage the lib folder and updates (update coming soon honest!).

On 13 January 2014 01:47, Peter Robins notifications@github.com wrote:

I'm now doing this in a different way: keeping baseURL relative and
setting up a jspm endpoint:

System.baseURL = System.baseURL + 'lib/'; // can be changed to dist for production
System.paths['jspm:_'] = 'https://registry.jspm.io/_.js';

then if I need something from jspm, I do !jspm:json or jspm:underscore. I
quite like this, as it's consistent with the other endpoints. I think it's
better if you mainly want to access internal modules and only sometimes
externally from jspm. Either way, the system's flexible enough to be able
to handle different ways of loading things.

I think all the issues here are now resolved, so I'll close this issue.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3#issuecomment-32138457
.

probins commented

I'm moving towards shim being included in the global files themselves

how are proposing to do this? On an ad hoc file by file basis?

guybedford commented

Currently, when shim configuration is provided in the package.json, it is
added in to files when downloading with the CLI or over the CDN.

There was a stage where instead, configuration was being injected into the
main config file by the CLI, but I've moved away from this.

The global shim config with a System.shim['bootstrap'] = ['jquery'] sort of
thing still works though if custom shims are needed though.

On 14 January 2014 13:38, Peter Robins notifications@github.com wrote:

I'm moving towards shim being included in the global files themselves

how are proposing to do this? On an ad hoc file by file basis?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3#issuecomment-32257770
.