Resource.textContent() not implementable in Browser

[quintesse]

Right now Resource.textContent() isn't implemented in the Browser (nor is .size obviously) because in the browser we can only deal with external data asynchronously.

Right now that means that a module like ceylon.locale cannot be used at all in a browser.

Futures aren't part of the language module so it seems to me that the only thing we can do is to add asynchronous versions of the two members:

• textContentAsync(encoding, callback) and
• sizeAsync(callback)

The documentation should then state that it would be preferable to always use those for best compatibility across all possible platforms.

Wdyt @gavinking @chochos ?

[chochos]

Is this a problem similar to the readline thing? Isn't the node.js version also async? In JS we can detect at runtime whether it's running on a browser and use a different implementation...

[quintesse]

Don't ask me the things you should know @chochos ;)

But there could indeed be other places where we would need an async version, readline is a good example. Any others you can think of?

[chochos]

Yeah that damn readline.

And sorry about the previous comment, it's been some time since I looked at that code; turns out, the node.js version has a way to read files synchronously.

[xkr47]

So you don't read a module as a package along with all its resources then?

[chochos]

Not with all its resources. Those are loaded on demand.

[gavinking]

So you don't read a module as a package along with all its resources then?

So, yeah, I agree, that's the easy solution, and it's the solution I've partly implemented.

However, the current implementation works by just copying text data into strings embedded in the .js file. Which I guess works well enough for stuff like .properties files and maybe very small xml and json files, but starts to fall apart for large text data files, and for anything that's not plain text.

So a second shot at this would add a layer of indirection, and eagerly populate the data fields from files on the server while executing the .js file (i.e. when loading the module).

A third option is to add the APIs proposed above for loading resources asynchronously, but I gotta say it's just not clear to me that that is really a useful model for the typical sorts of things we call "resources". And, in that case, we already give you the option of getting the URI from the Resource and going and fetching it from the server yourself in your own code.

So I dunno if it's really even offering that. For simple .properties files and the like, loading the data with the module seems quite reasonable to me.

[gavinking]

i.e. for example, getting a Locale instance in a callback seems a little crazy to me. But perhaps I'm wrong about that, and it's the sort of thing people are used to these days.

[gavinking]

So a second shot at this would add a layer of indirection, and eagerly populate the data fields from files on the server while executing the .js file (i.e. when loading the module).

Oh, wait, looks like there's no good way to do this after all—I had thought there was an API in requirejs, but now I can't find one.

[gavinking]

So my approach of embedding in the .js file doesn't really work that well for ceylon.locale because it results in a 3.3MB file size. There needs to be a way to filter down all those 162 language variants to something more sane.

I guess we can handle this at build time by having two different resource directories, resource-jvm and resource-js.

Or, yeah, the only other solution I can see would be to make a JS-specific version of the locale() function which accepts a callback. (Yew!)

[gavinking]

So my approach of embedding in the .js file doesn't really work that well for ceylon.locale because it results in a 3.3MB file size.

Well, it's <190K gzipped, but still large.

[FroMage]

I've nothing against callbacks: they allow other libs to built on top and provide support for RX for example.

[gavinking]

It's not that I'm against callbacks; just that it seems amazingly inconvenient for resources.

[chochos]

What if we pack them like we pack the metamodel info? Could be one separate file with all the resources, or one file per resource. The compiler can even optimize so that all smaller resources can be stuffed in one file and large resources be in separate files.

So, a resource file gets compiled as a common js module that exports a string with said resource as text...

[gavinking]

Sure, that would be better I suppose. But how does the meta model file actually get fetched from the server? I have never understand that.

[chochos]

A call to require

[gavinking]

At the time when you load the module I guess?

[chochos]

No, it's lazy. There's a function in the main module that internally returns the big model object if it's already loaded, or loads it the first time via require. Come to think of it, that might be unnecessary since require caches the loaded module anyway...

So, we can wrap a resource in a common js module that exports a single value, then load it via require and returns that value. As I mentioned before, we can optimize this to stuff several small resources in a single js module, and leave large resources on separate modules. The main difference with leaving the resources as they are and loading them manually (as we do in node backend) is that require works synchronously...

[gavinking]

So it does a synchronous request to the server?

[chochos]

Yes. It fetches the module from the server, once, and caches it locally for subsequent requests.

[chochos]

I just did a quick handwritten PoC for this and it works fine. I have a common JS module with this:

(function(define) { define(function(require, ex$, module) {

function loadResource(name) {
  var res=require('shit/0.1/resources');
  return res[name];
}

//MethodDef run at caca.ceylon (1:0-11:0)
function run() {
  console.log("Hey");
  console.log(loadResource("uno"));
  console.log(loadResource("dos"));
  console.log(loadResource("uno"));
  console.log(loadResource("dos"));
}
ex$.run=run;
});
}(typeof define==='function' && define.amd ? define : function (factory) {
if (typeof exports!=='undefined') { factory(require, exports, module);
} else { throw 'no module loader'; }
}));

And the resources.js file contains this:

(function(define) { define(function(require, ex$, module) {

console.log("Loading module resources");

ex$.uno="Esta es la cadena uno";
ex$.dos="And this is string 'dos'";

});
}(typeof define==='function' && define.amd ? define : function (factory) {
if (typeof exports!=='undefined') { factory(require, exports, module);
} else { throw 'no module loader'; }
}));

So when you run the run function in the module, you get this:

Hey
Loading module resources
Esta es la cadena uno
And this is string 'dos'
Esta es la cadena uno
And this is string 'dos'

Which means the module is loaded lazily, just once, asynchronously, via require.

[chochos]

So to implement this, then:

1. The compiler must process the module resources, packaging them as CommonJS modules
2. Modify the Resource implementation in JS so that it uses require to fetch the resource contents
3. If we want to optimize for space, we can pack resources under a certain size in files up to a certain size, so that for example you can have 10 small resources and 2 larges ones and you end up with 3 files (one for the 10 small resources, and each large resource in its own file). And we'd need some index in the module to know where each resource is found, to call require with the proper module for a requested resource.

[chochos]

However, binary files could be a problem, and perhaps we want to leave the original resources available as well, in case they are used directly in URL's in web apps for instance.

[gavinking]

However, binary files could be a problem

Yes, I agree, this should be only for text files.

So to implement this, then [...]

That all sounds just perfect @chochos.

[chochos]

How do we detect text files?

[gavinking]

I have some rubbish heuristics for that in the code already.