Dynamic Eval for Fortran
jacobwilliams opened this issue · 17 comments
So, here's a thought. This would be amazing:
integer :: i,j
i = 1
call eval('j = i + 1') ! some sort of new intrinsic
write(*,*) j ! writes 2So, something akin to Python's eval.
Now, we can already write function parsers (see here and here, and we outlined a more modern approach here). But, these are limited and require you to manually pass in the variables you want your function to have access to (there is no access to the current Fortran scope). An intrinsic eval could:
- Have full access to everything in the current scope
- Be able to execute arbitrary Fortran code
Moving Fortran more into the realm of dynamic languages would be game-changing I think.
I don't believe this is feasible for a compiled language. It would effectively require the fortran run time to carry around a copy of its own compiler, that understands how to interface with the code it generated. Additionally, the mechanics of making it work, and fail properly, are incredibly complex. What happens if you put invalid code in there? You can't check that at compile time, because the string could be generated at run time. What if the code includes something that fails? I.e. call eval('read('Hi', '(I0)') var'). What if the code includes variables that aren't in scope? Can you put variable definitions in there? I'ts just too complex.
I didn't say it would be easy. 😄 Just trying to get us to think big, and not just about minor updates.
A vendor could choose to only allow this feature if the compiler was installed on the machine maybe? Same way you have to have Python installed to run Python (or Julia installed to run Julia). Others could include it in the runtime. The standard could be silent on this (as it is silent on so many other things).
Errors would be run time errors. It would probably need to return some kind of status code or message (or raise an exception if we ever actually get that).
@jacobwilliams , do you know of any compiled language which has this capability? Python and Julia are interpreted languages, so when eval is invoked, the interpreter is already running and has full access to the current environment. It would be a pretty incredible feature, but I don't think it fits in with Fortran's goals.
This could be done in the future with LFortran. It can already just in time compile, so it is effectively interpreted, just like Julia works. But whether it should be done is another question --- I really like that Fortran can be fully and statically compiled. The idea behind LFortran is to keep this feature, but allow Julia/Python like interactive development. I wanted to implement something like eval that you described, because it would make it feel more like Python, which could be useful for interactive development, the main target of LFortran. But for production use, I would prefer if Fortran didn't have this feature. So how to reconcile this?
@jacobwilliams what are some use cases where you would like to use this feature?
@everythingfunctional No, I don't know of a compiled language that has this. But wouldn't it be amazing if Fortran had it!
@certik Personally, I would use this like crazy, even in production. I've spent countless hours and written tens of thousands of lines of code to give my users the ability to dynamically add computations to a compiled Fortran application (this includes a DLL plugin architecture, hand-written embedded parsers, callbacks to Python, etc). I could probably replace all that with this. It would be glorious.
Another trivial (but important) use case would be config file input to programs. If they could contain actual Fortran code, that would be amazing (e.g., the way some people use Lua for this right now).
Very excited to know that maybe it's at least a possibility for LFortran.
@jacobwilliams ok, interesting. I created an issue for this: https://gitlab.com/lfortran/lfortran/issues/149.
(The eval functionality is mostly already implemented in LFortran, it's just not exposed from Fortran itself, only from Python and now C++, as that is how the read eval loop is implemented.)
This could be done in the future with LFortran. It can already just in time compile, so it is effectively interpreted, just like Julia works. But whether it should be done is another question --- I really like that Fortran can be fully and statically compiled. The idea behind LFortran is to keep this feature, but allow Julia/Python like interactive development. ..
By the way, does the Fortran standard require a processor to statically compile programs? My understanding is no; that, as written, it doesn't care whether a processor is a Python/Julia type of interpreting one or a static compiler.
FWIW, eval type functions can be considered unsafe, since they can be a way to inject code into an executable. I'd probably prefer that something like this not exist if it's not necessary.
From the top of my head, Rust, Go, C#, Lisp are examples of compiled languages that can eval. I bet there are many more. Compiled binary needs to have a runtime included to be able to eval() as it runs. LFortran seems like a good candidate for such a thing.
Yep, HPC applications may not need this... although I could see using a Fortran config file input as being useful. Think about namelists as being a very very restricted kind of Fortran code, and then allow any arbitrary Fortran code to be in there. Think of the possibilities. :)
Yes, it could be considered unsafe... but so are a lot of things. execute_command_line could also be considered unsafe, since it can basically do anything. People just have to be aware of what there are doing. I think this eval idea is safer than loading a DLL, since at least you have access to the code you are running.
And certainly, those of us building interactive applications would benefit tremendously from this. It also has the potential to expand the use cases of Fortran, which is what we need to be thinking about.
I could definitely see some of the Big Corp compilers not implementing this, but that's OK (intel hasn't implemented unicode, but the standard allows that option). The interface could be similar to execute_command_line (which I think also has the option of returning a -1 exit code for the alleged "processors" that don't have the concept of a command line).
Indeed. The advantage of not including eval is that you can ahead of time compile everything. By including it, the Fortran runtime library has to include the full Fortran compiler (with LLVM etc.) and indeed any code can then be compiled and executed at runtime.
It also needs to have (a representation of) the original program's scopes and symbols and types and IMPLICIT rules and generic interfaces, so that the names in the evaluated statement can be resolved at runtime. This would be a big deal.
It also needs to generate code in such a way that any symbol that might appear in an eval is available at runtime (essentially -g0 mode), so performance of the compiled code around the eval will suffer.
I often see requests for an expression evaluator, and if one limits it to that, it's at least feasible. To make this work at all you would need to identify the variables you wanted to use in advance so that the compiler could pass information about them to the evaluator. NAMELIST already has to do this sort of thing.
That said, I don't see this as an idea that would gain much traction for the standard. It's something that could be done as a library, with calls to "register" variables. If you wanted it to be magic and see all the host-scope variables, the idea would be DOA in my opinion.
Actually, my contribution in #32 is quite akin to that: lambda expressions. Also see the "experiments" directory in my Flibs project enumerating (integer) sets and the like - enum_set.f90 for instance. Not entirely an "eval()" function, but still.
@jacobwilliams , do you know of any compiled language which has this capability?
Common Lisp compilers have this capability (and they are not interpreters). You're right, @everythingfunctional, though, it does require all the compilation machinery at runtime. And I believe the evaluated code runs in a separate "environment" that doesn't have access to the lexical environment.
Greenspun's Tenth Rule: "Any sufficiently complicated C or Fortran program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp."