resources in .car archive
Closed this issue · 44 comments
[@gavinking] The compiler should copy resources in the source directory into the .car.
[Migrated from ceylon/ceylon-compiler#881]
[Closed at 2013-10-26 20:56:22]
[@quintesse] So first thing is how to define where the resources are stored. It could simply be a resource folder next to the source folder we already use for code. Add an extra option to the compiler to override the default, eg --resource.
The problem I see with resources is how to handle the difference between JVM and JS backends.
For the JVM backend there's a single .car archive that could simply be a unification of the (compiled) contents of the source and resource folders.
But the JS backend has a single .js file containing the code and probably no way to deal with archives.
So for the JS backend we could just simply copy the contents of the resource folder right next to the JS file (or inside a separate resource folder)... but that would mean to have duplicate resources, they'd be available both as separate files for JS and inside the .car file for the JVM. That doesn't seem ideal. What do do about that?
[@gavinking] Is a separate folder really necessary, or is it just an inconvenience?
I mean, if it's not xxx.ceylon, xxx.java, or .xxxx, it's a resource, right?
[@tombentley] Well, until we do some kind of JS interop, at which point we have to add *.js to the exclusion filter, except such a js file might legitimately be a resource for a JVM module (e.g. javax.scripting/Rhino).
[@quintesse] Besides if we're talking about resources for web apps for example, HTML, CSS, JS, images it would be a nuisance to have to filter out the things that are not resources (and the JS files? Are they part of the web app and therefore resources or are they Ceylon code and should be filtered? Just as @tombentley just mentioned)
The separation is also one of the things I think Maven did right, if I may be allowed to say something positive about Maven without being lynched :)
[@tombentley] Quick, grab your pitch forks!
if I may be allowed to say something positive about Maven
That's it. We're kicking you out of Ceylon.
[@quintesse] Anyway, the actual folder layout is not something I feel too strongly about, but I have no idea how to handle the JVM/JS difference here. I'm pretty sure we need them packed at least because the JVM runtime needs to be able the easily download the resources, I don't think we want them downloaded one at a time.
So that means that either we have them duplicated in a folder for JS (not a very acceptable option IMHO) or we let the container/server decide if it should extract the resources so browsers can get at them.
And that can also mean two things: either they download the .car with all the JVM stuff and extract only the resources, or we make a separate resources archive (which might be a better option if you think about a future with possibly even more backends than we have right now).
That's it. We're kicking you out of Ceylon.
Well at least I got out alive ;)
[@matejonnet] I see two types of resources, static web content (html, css, js, jpg, etc) and configuration files (log properties, web endpoint mappings, etc).
Static web content can be in a separate archive, or even not in archive at all, we can specify path, where to look for them. Out of archive is simpler, but not so practical to transfer our app.
Configuration files should be in car as they could be module specific. We can pack in all *.properties fies. Or are there any other preferences to specify configurations?
Configuration files should be in car as they could be module specific
Well we still have the problem where the JS backend maybe wants to have access to those files as well. So do we download the .car with Java classes just because it contains a resource it needs? Maybe better to just have a separate resource download perhaps? On the other hand we might say that the size of the .class files in general is small enough to not care about that and not worth the trouble of introducing yet another archive file.
Or are there any other preferences to specify configurations?
Well we tend to prefer git-like conf files, which is what we use for the tool chain. I'm currently busy implementing a parser for that in Ceylon (which is unfortunately not as easy as I thought ;) )
I see two types of resources
Thing is that even though it's best for a web server to have the static "serveable" resources available as separate files I don't think a repository server is the best choice to do that (it does do it for module documentation btw, but I think that's a use-case where it does make a lot of sense to be able to serve those files directly).
So in my opinion it makes perfect sense to pack the resources in some kind of archive and have the web server, that downloads the module from the repository, decide that it wants to unpack it to some kind of location for easy access. That way we don't have to make that decision. We just have to care about easy up-downloadability / manageability for Herd.
And like JEE does with its WEB-INF folder we might have a PUBLIC folder within the resources to distinguish between private and public files, it's a "low-tech" solution but it works fine I think.
[@gavinking] So we need to decide exactly how the program is actually going to access these resources.
FTR, @matejonnet, I don't think we're primarily talking about stuff like HTML or images, we're talking about shit like properties files, data files, etc.
Scenario 1: JVM
We need to provide an API for the program to load its resources. If that's going to be built on top of Java's Class.getResource(), then obviously the resources need to be in the .car. And we need to expose Class.getResource() via some API in ceylon.language.
If not, then whether they are packaged into an archive in the module repo, or thrown into a directory in the repo, the module architecture needs to know about them and support them and treat them as a whole separate thing, and we need to expose an API in ceylon.language for accessing them. That will be some work to implement!
Scenario 2: JavaScript
A JavaScript client isn't going to want to get its resources for a .car. Nor it there really an module runtime as such in the JavaScript runtime. So it's going to just get its resources by hitting the module repo directly over HTTP or something, I guess. Well, I'm not sure about the case of Node - should that be direct filesystem access? So my question is: on a JavaScript client, are there convenient APIs for opening up zip archives? 'Cos if not, then the resources are going to need to be just lying in a directory.
I'm much more focussed on what works for a web browser client here than on what works just in Node.
[@quintesse] Just a remark: because while I think we'll be able to set up an infrastructure that will support clients downloading software packages and resources once (or at least a limited amount of times) I don't think we'll be able to handle the requirements of a popular app that needs to download its stuff repeatedly for each and every client. We just need one mention on Slashdot to bring down our whole infrastructure. (Right now a Slashdot might bring down our web server after an interesting blog item has been published, but it wouldn't bring down Herd)
If we do go ahead with something like that we might need some serious hardware.
(You might say it's just a webserver, it can handle things we throw at it, but imagine that any application within Herd might be considered it's own web application, which means that one Herd server might potentially host a large number of popular applications. I know right now we would be happy to have so many users that we have to worry about that, but once we make the decision to go down that road it would be difficult to change it)
[@matejonnet] What about putting single "config" file in a .car that will be used for all configurations (httpd, log, etc)?
Especially if we create a git-like format, different sections can be nice separated.
@quintesse
Do you have any work done on git-like parser? Should I take it?
What about putting single "config" file in a .car that will be used for all configurations (httpd, log, etc)?
I would strongly prefer to avoid inventing configuration file formats for now. We should stick to expressing things within the language. Config file hell came scarily close to killing Java dead.
[@quintesse] I do agree with @gavinking here. We wrote a config file parser/reader/writer for Ceylon that handles an almost 100% git-like syntax which is fine for simple things. And I would like to have it ported to Ceylon so we have something people can use when they need something simple.
But like Gavin I think we should try to avoid putting any dependency on file formats in the language/SDK.
I would find it much more reasonable that a file format would be part of some kind of OpenShift support module (unless it could really be shown that it has value outside of that use-case) although that file format could then use the config file loader we have made available.
[@matejonnet] We can drop my ceylon.util.properties, they are not required by OpenShift.
I thought it would be easier for users to define httpd web endpoints in prop file instead of writing in code.
I thought it would be easier for users to define httpd web endpoints in prop file instead of writing in code.
I don't see why it would be. When they use code, the tooling helps them write the right thing. When they write a properties file they have to guess, and they find out about mistakes at runtime.
[@quintesse] Personally I prefer to have some kind of configuration object, that makes it strongly typed and like Gavin says will help the user. Later if we have some kind of simple/generic config file support a user can decide to add it's own config file format. In general I think that the way we do configuration depends very much on the framework that uses it. We should not try to "push" people in a certain direction at this point. So if you really don't need it right now I'd remove it. Maybe make it part of the demo to show how easy you can add file format support.
Besides that it will make us focus on making the API as easy as possible to use... because there's no other way :)
[@matejonnet] Agree! Will remove properties related code from a httpd.
Do you have any work done on git-like parser? Should I take it?
@matejonnet I was porting the CeylonConfig, ConfigReader and ConfigWriter that we have in com.redhat.ceylon.common. I have some code, not much. If you're interested, and have nothing better to do, get in touch with me by email and I'll tell you more about it.
[@matejonnet]
@quintesse
I thought to take it, to speed it up, if we use it for httpd, as we decided to go without file config, I will go with other things ;)
I also find it interesting, I'll ping you, when I have some extra time.
[@gavinking] How about if we let you specify resources to include in the .car as part of the module descriptor, for example:
include("../resource/*.xml")
module org.example.whatever '1.0' {}
[@matejonnet] As we decided to go without properties files, there is no rush to resolve this issue. We can move it out of M5.
For static web content users can define in code which folder to use. I'll do the same on OpenShift, define a folder for static files.
[@FroMage] Moved to 1.0
[@gavinking] I really don't like how this one keeps slipping. It really doesn't sound like it should be that hard to me. I think we're overcomplicating it.
[@gavinking] I mean, right now there is no way to get a .properties file into a .car short of asking Ant to unzip the car and rebuild it.
[@FroMage] Well what you described yourself seems like a complicted problem that we haven't begun to solve. We need these things to be in a zip for the JVM and unpackaged for JS. The CMR needs to know about this, so does Herd. And that's without any logic as to what should end up in there like most build tools provide. I'm happy to just stuff any resource folder contents in the car, but as you said, it requires support from the language module and remember the language module doesn't deal with IO at all so what sort of support would it give? And that wouldn't work for JS.
[@quintesse] Thing is there are several options:
- One option is to just put all the resources in the
sourcefolder. When we create the.carfile we filter out.ceylonfiles and put everything that's left in with the compiled.classand.jsfiles.
This works just fine for .car files on the JVM side. But for the JS side it's a bit more complicated because it can't access those files. I still think the simplest thing is to make the individual files accessible on the server by extracting the .car.
Problems I see with this: right now having a .car file means there is a Java implementation for that module. There would be no way to say: this .car is only here to hold some JS resources.
There's also a minor issue with not being able to filter out the Ceylon .js files because they could actually be resources used by some web application.
- The second option is to create a separate resource file,
foo-1.0.0.resorfoo-1.0.0-resources.zipwhich would still get exploded on the server so JS based code can access the files. It would also get added to the JVMs classpath so Java's resource handling can get at it. - A third option is a mixture of the two. For real easy JVM-only resources that simply get added to the
.caryou add them to thesourcefolder, but they'll be inaccessible to JS. For shared resources we go with option 2. - Another hybrid option is where all resources are put in the
.carfor easy access and easy download while they also get added to a separate downloadable file/zip like in option 2 just for the JS side. Obvious disadvantage is the duplication of resources.
All these options could either use a separate resource folder or mix in everything with the sources. We'd just have to filter correctly.
(NB: Option one is actually the one that gives the most problems because suddenly we can't just say anymore: .car means having a Java implementation, .js means having a JavaScript implementation.)
So I agree we should go for something simple, but I don't agree we over-complicate things, it is complicated.
[@quintesse] Btw, the 3rd option, a hybrid where the resources mixed in with the sources would get added to the .car while the ones in the resource folder would get added to a shared downloadable zip file would give us the possibility to implement this in 2 phases: first we do the resources-in-source-folder that get added to the .car, solving our most immediate problems, while we could do the rest at some later stage.
[@gavinking] Look, for now, surely we can let ceylon compile accept a glob with the resources you want to include in the car.
ceylon compile blah blah blah --resources resources/**/*.properties
[@quintesse] Sure we can, but then we very well might have to change it in some incompatible way in the future. If you don't mind that then it's okay.
And can we decide on resource vs resources for once and for all? We have source and modules, so what do we go for, singular or plural? Or flip a coin?
[@quintesse](ok now in english)
@FroMage This sounds as something that should be not too hard to do if we keep it limited to just adding the contents of a resources folder to the .car file. I took a quick look at the JarOutputRepositoryManager and it seemed pretty straight forward, but it wasn't immediately clear how we could get the required information up to that point. If you could give me some hints I'd be willing to pick this up (assuming you're not going to fight Gavin over moving it to M6 of course hehe)
[@FroMage] Maar, dit is niet genoeg omdat: you still need a way to access those resources at runtime.
[@quintesse]
;)
Sure we'd need at some time to make a Ceylon resources API, but for now people (Gavin) can use Java interop. It at least makes it possible. Right now there's just no way (except for unpacking and repacking the .car like Gavin said).
[@gavinking] As this issue nears its first birthday, it is currently blocking us from implementing ceylon.locale. I am hereby shutting down the above lively discussion of how we can solve the problem using AbstractResourceLocatorFactorys and cloud-based virtual filesystem abstractions, and requesting that somebody add the ability to include resources in a .car. This incredible innovation will make it much easier to write useful programs in Ceylon.
[@quintesse] Come on, there's been none of that here. The only thing we discussed is "how the heck do we make this work for the JS backend?" :)
Other than that I've already said before I'd like to do this, so if there are no objections...
[@tombentley] Well, I think I need a break from annotations before they drive me crackers, so I can do that. For the API part, I was thinking something like this:
"Resolves a resource URI (in the 'resource:' URI scheme) to a URL from which the resource can be obtained."
shared native String resolveResource(String resourceUri);
The caller could pass the URL thus obtained to stuff in ceylon.net or whatever. This isn't ideal because of the lack of type safety, but it does provide a way of having something in the language module without dragging in IO, and it hides how/where the resources are located for a given platform.
[@tombentley] OK, I see @quintesse has assigned himself, so I'll just go crackers then ;-)
[@quintesse] Sorry @tombentley but I need to do something else besides fixing issues as well... it's been like... forever? ;)
[@quintesse] Ok, the whole system of putting resources into CAR files is now done (whipping up a first working example wasn't too bad, but making it all work right was quite a bit of work in the end).
So the final design is very similar to the handling of source folders: either you add one or more --resource=/some/path entries on the command line or you leave it out and you get the default resource folder in the current directory.
Then if you compile, let's say ceylon compile mymodule you'll not only get all the sources (well the resulting class files) that were found in source/mymodule but you also get all the resource files found in resource/mymodule. A "resource files" is just any file, there's nothing that makes them special in any way and no processing is done on them, they just get added to the final CAR file.
Handling of single file names instead of compiling modules works exactly the same as for source files. You can do ceylon compile resource/mymodule/image.jpg and only that file will be put/updated in the CAR.
Multiple resource folders are handled a bit differently from sources. Where for sources all source files encountered in all the source folders will be compiled (even if they have the same name, let's say source1/mymodule/test.ceylon and source2/mymodule/test.ceylon). Detection of duplications is only done at the level of declarations, not files.
But for resources we cannot have duplicate files. So the rule I've implemented now is that only the first of such files encountered in the list of folders will be used, all other duplicates will be ignored. The ordering of the folders is as given on the command line.
There are still a couple of things that need to be done but I'll open new issues for them. Closing.
[@gavinking] Great!
Tako, is this documented in ceylon help and the subcommands list?
[@gavinking] And is it supported in the ant task?
[@quintesse] Ah ant tasks, good one!
Ceylon help is automatic, but we probably should have some extra explanation on the website. (Does this need to be in the spec, or does it not deal with things like this?)