Command to copy zipped sandbox to cabal's local sandbox
Closed this issue · 14 comments
Is it possible for Halcyon have a command to copy the sandbox in /app to the local .sandbox?
Note: I'm really new to Haskell, so I don't really know what I'm doing, but when developing, I've noticed cabal build runs a lot faster than halcyon build -- reusing the halcyon sandbox for cabal would be nice.
Related to #47, this might be a duplicate.
Currently, it’s not possible to copy the Halcyon sandbox to another location, because Cabal sandboxes are not relocatable (haskell/cabal#462, haskell/cabal#1490, haskell/cabal#1542, haskell/cabal#2302).
However, all you need for cabal build (and related commands, such as cabal test, cabal repl, and cabal run) is to point cabal-install to the Halcyon sandbox, which you can do in one of three ways:
-
Specify a command-line option:
$ cabal --sandbox-config-file=/app/sandbox/cabal.sandbox.config build -
Set an environment variable:
$ export CABAL_SANDBOX_CONFIG=/app/sandbox/cabal.sandbox.config $ cabal build -
Create a symlink:
$ ln -fs /app/sandbox/cabal.sandbox.config . $ cabal build
The possibility of doing so should be made clear in the documentation. This is now tracked as mietek/halcyon-website#2.
Let me know if this helps.
Will that sandbox be messed up if you run a different project? In that case I should just run halcyon build again to get it fixed?
(trying now)
Ok I had to delete the /app/sandbox to get constraints to resolve (I just built Hoogle; my project is a Snap application), and now it has to rebuild the entire sandbox... oh well. I think I tried symlinking before but it felt wrong because the sandbox could be messed up by something else.
Is making this simpler just a matter of improving cabal?
Halcyon keeps a single sandbox only, suitable for the project for which you most recently ran halcyon build (or halcyon install), because of the implications of Cabal sandboxes not being relocatable.
Therefore, yes — if you want to build a different project, you should run halcyon build at least once, so that it can restore (or rebuild) the right sandbox.
For Halcyon to restore the sandbox (as opposed to rebuilding it), the sandbox must’ve already been built, archived, and uploaded to private storage — which is only possible if you’ve set up private storage first.
Unfortunately, improving Cabal is no simple matter, which is why Halcyon attempts to improve on Cabal from the outside.
Can you check if halcyon build --no-archive is much slower for you than cabal build?
Running time <command>:
halcyon build takes 32 seconds; probably half of that is "Stripping app"
halcyon build --no-archive takes 31 seconds; same regarding the stripping
cabal build takes 4.6 seconds
Executed in that order without any cabal clean's in between.
I'm using IntelliJ as the IDE, but I'm building on the command line because the Haskell support is flakey (ghc-modi crashes a lot). Right now I'm setting up private storage, so that's from-scratch rebuilds should be ok.
Is there an option to exclude certain things from the sandbox tar? It's picking up the old .cabal-sandbox (just deleted it), .git, and some other things that it probably shouldn't. Something like .gitignore syntax. Sorry if this should be its own issue.
EDIT:
I just rearranged my project files so extraneous stuff wouldn't get caught in the sandbox collection process. Got my sandbox from 55MB to 3.7MB and it runs a lot faster. It still picks up the dist directory and uploads that to S3 though.
It sounds like you’re talking about build directory archives, which are named halcyon-build-$LABEL.tar.gz, as opposed to sandbox directory archives, named halcyon-sandbox-$HASH-$LABEL.tar.gz.
A build directory starts as a copy of your source directory, so yes — it’s a good idea to avoid unnecessary source files.
Right now, there’s no option to avoid copying certain source files — only an option to avoid hashing them, HALCYON_EXTRA_SOURCE_HASH_IGNORE (#10). Please open a separate issue if you’d like this option to be added.
You can disable the creating and uploading of archives with HALCYON_NO_ARCHIVE, or just the uploading, with HALCYON_NO_UPLOAD.
Currently, it’s not possible to disable stripping the app. This is now tracked as part as #54.
Ok thanks for the clarification. I am of the opinion that it should leave out dist by default, as by definition it should change (even if you don't check with a hash). I'll create an issue for it sometime.
I did discover the NO_UPLOAD option -- saves a 3 MB upload every time and I'm now getting 6 seconds halcyon builds -- thanks.
The purpose of archiving a build directory, dist included, is to restore it the next time a build is performed, saving time by not requiring the app to be rebuilt from scratch. This is useful in workflows which don’t retain the source directory between builds.
There is an additional subtlety — a source directory may include a partial dist, which may be required for a build to complete successfully. For example, happy includes pre-processed files in dist, because processing these files would require happy to be already installed.
It occurred to me that rather than an option to avoid copying certain files, your use case would be better served by being able to keep build products in the source directory (#14).
Ok, please excuse my ignorance -- I wasn't aware of how halcyon worked. So... it copies the source dir (the working directory) to a temporary dir, and does the build there? (and copies the build artifacts somewhere else if you used install). And the source hash checks whether it needs to actually do the copy?
Looking through Halcyon source, it looks like it's a bit of a pain to exclude things from the archive, unless they're not copied in the first place. At the moment, now that I've removed the extraneous things from my source dir, I'm ok with living with dist in the build archive. I just won't upload it each time. I'm used to relatively small Java projects using maven/sbt, where doing a clean rebuild is no big deal, and with the size of my project so far, (as long as the sandbox is kept intact) a clean rebuild is trivial.
I would probably better off if the build could be executed in the source dir (or copy the artifacts back there), because my acceptance tests expect the executable to be where cabal would put it. However at the moment, I think a once off halcyon build, then using cabal build normally will suffice.
With 3419002, you can now disable stripping the app by setting HALCYON_APP_NO_STRIP to 1, or by specifying --app-no-strip on the command-line.
$ cabal clean && halcyon build --purge-cache
13.068s
$ cabal clean && halcyon build --purge-cache --app-no-strip
12.077s
There's not really much difference (it downloaded an old build dir from S3). If I was in the mood to save one second, I'd also turn of self-updating while I'm at it.
But I'm realising now that Halcyon was not originally designed for live development work. It's a tool for build servers/Heroku to get maximal performance in not-so-often but repeated builds. Not for running every half-minute in a coding cycle.
It's useful for prepping the dependencies etc at the beginning of a the coding session though. Thanks for your work.
You’re welcome.
While you’re correct that Halcyon grew out of the Haskell on Heroku project, supporting incremental development was my goal from the start. In fact, at one point, I was able to complete remote builds, on Heroku, in under 30 seconds — from git push to deployed release.
I consider anything which gets in the way of incremental development to be a bug. Please continue opening issues.