Compiling scripts transparently

Question

Compiling scripts transparently

zmeyc opened this issue 6 years ago · 10 comments

I'm considering replacing lot of small bash scripts with swift scripts, but they're too slow to start:

$ time ./script
Promise(Hi!)

real	0m0.548s
user	0m0.358s
sys	0m0.173s

Running compiled ones is fast though:

$ time ./Script
Promise(Hi!)

real	0m0.016s
user	0m0.009s
sys	0m0.006s

I could hack a solution with 'swift sh eject'-ing scripts, building them then copying binaries to /usr/local/bin, but possibly this can be supported transparently by swift-sh? I.e. it can maintain a cache of compiled scripts and check if compiled version already exists and is not older that script file, and if so, run it instead. What do you think, is it possible / viable?

Answer 1 · 2019-03-14T04:55:53.000Z

We already have a cache.

So yeah, we could already do this.

Though ideally SwiftPM itself would do this, but we can implement this for now, PR welcome, sources are easy enough to understand and I don't have the time right now.

Answer 2 · 2019-03-23T00:57:16.000Z

I've made some improvements that I will push shortly, times are all for a pre-built unchanged script:

	time
swift-sh-1.8.4	0.421s
swift-sh-1.8.5	0.175s
swift-run 5.0	0.109s
direct	0.021s

I can do more I believe, since we can check if the script mtime is less than the compiled binary mtime unchanged and then just exec which is how make works, but seems a little risky, but I guess if it's good enough for make it should be good enough for us.

Answer 3 · 2019-03-23T01:04:40.000Z

K got it down to 0.040s with the above. Further optimizations will be hard. There isn’t a lot more happening that isn’t Swift itself. In theory I'm sure more could be gained, but we are talking about an unnoticeable amount of time now at least. Obviously we’d prefer to have negligible overhead, so PRs welcome.

Answer 4 · 2019-03-23T01:37:17.000Z

@mxcl Very cool! Thank you very much, it really boosts usability.

Answer 5 · 2019-03-23T02:49:01.000Z

Notably this introduces a bug that is only present for the Swift 4.2 -> Swift 5.0 transition. I'm not sure as a result if I should fix it.

Because these are not ABI compatible, thus the binaries will fail to run after Xcode is upgraded.

The only fix is to record the swift-tools-version in a file that is then checked before deciding to just run the binary.

However once Swift 5 is installed this no longer becomes necessary… ever again.

I'll think on it.

Answer 6 · 2019-03-23T02:49:48.000Z

I guess I'll make a quick update that disables this optimization if swift < 5. Sucks a bit, but people having broken scripts and no idea what to do about it sucks more.

Answer 7 · 2019-03-23T02:52:02.000Z

Hmm needs to be more than this, I need to detect if a script was built against <5 and now swift is >=5.

Answer 8 · 2019-03-23T03:30:49.000Z

The full solution is to detect the ABI version requirements from the binary somehow and compare that against the current swift version in the path.

Querying swift for it’s version via swift --version is 20ms. Not sure we can skip this unless we can cache that via mtime querying again and that's complicated considering /usr/bin/swift is just a shim.

Answer 9 · 2019-03-23T05:54:21.000Z

So basically I need:

Fast active swift version lookup
Fast ABI version lookup for binaries†

† could just assume the generated manifest tools version is this, it’s not exact, but if the manifest version is different then either there's a bug or someone edited it by hand.

Otherwise I have to push this patch that makes the exec time closer to 50–60ms.

Answer 10 · 2019-03-23T06:19:17.000Z

Also for reference, I built a simple swift app that execs the same pre-built script I was testing with above and it ran between 32–35ms over multiple runs.