notes after trying to use external-program
Closed this issue · 7 comments
Hello.
I once tried to use external-program, and collected some notes about problems I saw.
-
sb-ext:run-program on Windows doesn't accept :environment nor :env keyword arguments.
external-program always passes the :environment argument to sb-ext:run-program,
therefore external-program:run always fails on SBCL Windows with an error about
illegal keyword passed to sb-ext:run-program, event if we don't pass :environment to
external-program:run -
Quoting the program arguments against shell interpretation is
inconsistent.For example on CCL (external-program:run "some-other-lisp.exe" '("--eval" "(+ 1 2)"))
doesn't work because of spaces in the expression;
''("--eval" ""(+ 1 2)"") works.But on CLISP quoting is not necessary, and
(external-program:run "some-lisp" '("--eval" "(+ 1 2)"))
works fine.The issue is complicated by the fact that external-program docs do not
specify how arguments are interpreted.I may be wrong, but I find more convenient the approach similar
to Unix' execve, i.e. arguments are just strings which are not interpreted
anyhow and work as if they just passed as the argv parameter of the called
program function main (i..e the how it works on CLISP now). -
On CLISP the method external-program:run
incorrectly interprets the exit status
returned by CLISP function ext:run-program.The rules of ext:run-program:
case 1
if :input or :output were T,
than ext:run-program return value
NIL means the program exited successfully
with status 0. Othersise ext:run-program
returns a positive integer.
case 2
if :stream was specified for both
input and output, 3 streams are returned.The external-program:run method interprets
the ext:run-program return value absolutely
differently. -
(minor) the API doc is not very clear what is the format
of exit status returned by external-program:run.
My assumption is that it should be in the same form
as returned by external-program:process-status.
Note, none of these issues is real blocker for me; just reporting here for your information.
On 27 May 2012, at 6:55, Anton Vodonosov wrote:
Hello.
I once tried to use external-program, and collected some notes about problems I saw.
- sb-ext:run-program on Windows doesn't accept :environment nor :env keyword arguments.
external-program always passes :environment argument to sb-ext:run-program,
therefore external-program:run always fails on on SBCL Windows with error about
illegal keyword in call to sb-ext:run-program, event if we don't pass :environment to
external-program:run
Ah, thank you. I have no easy way to test on Windows, so I didn't realize the behavior was inconsistent. (I also realize that external-program's tests don't work on Windows, but I'm not really sure how to make portable tests for this library. Perhaps I just have to duplicate them all.)
I have hopefully committed a fix that warns if the user tries to pass :ENVIRONMENT or :REPLACE-ENVIRONMENT-P in SBCL on Windows.
- Quoting the program arguments against shell interpretation is
inconsistent.For example (external-program:run "some-lisp" '("--eval" "(+ 1 2)"))
doesn't work because of spaces in the expression.
''("--eval" ""(+ 1 2)"") works.That's how args are treated by external-program:run called from CCL.
Is this also on Windows? I can't seem to replicate this on OS X. My guess is that it's a bug in CCL on Windows. In any case, I added a test to make sure that arguments work the sane way (as in your CLISP example).
- On CLISP the method external-program:run
incorrectly interprets the exit status
returned by CLISP function ext:run-program.The rules of ext:run-program:
case 1
if :input or :output were T,
than ext:run-program return value
NIL means the program exited successfully
with status 0. Othersise ext:run-program
returns a positive integer.
case 2
if :stream was specified for both
input and output, 3 streams are returned.The external-program:run method interprets
the ext:run-program return value absolutely
differently.
:STREAM is not a valid value for INPUT or OUTPUT in EXTERNAL-PROGRAM:RUN (you never get a process object, so there would be no way to access said stream), so we only need to deal with the first case, which I think we do correctly. In EXTERNAL-PROGRAM:START, we handle the :STREAM results (and, as the CLISP docs fail to mention, in the case where WAIT is NIL and neither INPUT nor OUTPUT is :STREAM, it just returns NIL, which we also handle just fine).
As an aside, these complicated result conditions are exactly the reason I wrote External Program. (run-program … :wait t) and (run-program … :wait nil) are clearly two different functions, yet every implementation conflates them.
28.05.2012, 03:21, "Greg Pfeil" reply@reply.github.com:
I have hopefully committed a fix that warns if the user tries to pass :ENVIRONMENT or :REPLACE-ENVIRONMENT-P in SBCL on Windows.
Could you push it? You seem to forget.
- Quoting the program arguments against shell interpretation is
inconsistent.For example (external-program:run "some-lisp" '("--eval" "(+ 1 2)"))
doesn't work because of spaces in the expression.
''("--eval" ""(+ 1 2)"") works.That's how args are treated by external-program:run called from CCL.
Is this also on Windows? I can't seem to replicate this on OS X. My guess is that it's a bug in CCL on Windows. In any case, I added a test to make sure that arguments work the sane way (as in your CLISP example).
Yes, Windows. Maybe a CCL bug (unlike execve, Windows function CreateProcess accepts the program command line as a single string and the program is expected to parse it to arguments. To provide the same semantics as on unix, CCL should quote the arguments somehow)
:STREAM is not a valid value for INPUT or OUTPUT in EXTERNAL-PROGRAM:RUN (you never get a process object, so there would be no way to access said stream), so we only need to deal with the first case, which I think we do correctly.
Indeed, it is handled correctly now (I was looking at the version on previous quicklisp).
Ah, sorry. My local repo wasn't tracking github, so it told me there were no unpushed changes (since there was no repo it expected to push to). It's pushed now.
Also, I can't really test the #+win32 bit, since I don't have Windows, so please let me know if it's broken.
Also note that there is an issue open against CCL on the argument escaping: http://trac.clozure.com/ccl/ticket/858
28.05.2012, 23:56, "Greg Pfeil" reply@reply.github.com:
Also, I can't really test the #+win32 bit, since I don't have Windows, so please let me know if it's broken.
#+win32 in convert-enviroment seems, OK, but run-program fails in a further step - windows does not have /usr/bin/env.
What do you mean by the comment
;;; Always use /usr/bin/env in SBCL to prevent "file not found" errors (could
;;; work around this more easily if it weren't a SIMPLE-ERROR).
When these "file not found" errors happens? Maybe other workaround is possible?
Ah, yes, I knew I was breaking Windows compatibility when I did that. There are a couple other places that break Windows compatibility as well (see EMBED-ENVIRONMENT and MAKE-SHELL-STRING) – I would be happy to incorporate Windows-friendly variants (perhaps I should put in environment-ignoring warnings in the mean time).
The problem in SBCL is that the PATH passed in the environment is not used when searching for the program. But, come to think of it, maybe that's not something External Program should be trying to manage. The PATH in the environment should just be used for the program that is run. I'll have to look into that and see how easy it is to make the behavior consistent across implementations.
The problem in SBCL is that the PATH passed in the environment is not used when searching for the program. But, come > to think of it, maybe that's not something External Program should be trying to manage.
I think externa-program should not. One reason it's difficult to implement, and also in my opinion it's just not necessary. I do not even sure that PATH of the current lisp process needs be considered (but as implementation does it, it's OK).
IMHO it would be sufficient and useful to put th idea with /usr/bin/env may to the API doc - people really need the PATH search can easily do it if they are on unix.