Building Forth (Book)
Opened this issue · 40 comments
Hi All,
I had been planning to write a book on building Forth for quite a well but never quite got around to it. That was back in 2016 before I knew Forthhub existed. Since communities like Clojure have produced books like the famous O'Reilly Cookbook series and the Rust Programming Language Book, I am going to open up the repo so all can contribute. I will be writing a fair chunk of it, editing contributions and also finding a publisher for dead tree form.
Here's the link, please feel free to bookmark it and I will set up a ebook build system soon.
The main content will cover bootstrapping Forth from a scripting language (Python, Ruby or perhaps Go) and then finally to full Forth bootstrapping. Perhaps a implementation in C too. Along the way concepts such as indirect threading will be covered. War stories, famous usages of Forth and any other Forth related content are also welcomed. For code, I am particularly interested in easy, simple implementations of the does and create
macros and also a full Forth x86 assembler. The simpler the better. Remember our audiences are not embedded programmers, this is a book for general programmers.
Of particular interest is the relation of Forth to modern stacks, i.e. what role can Forth play in facilitating unikernels and Internet of Things
For bare metal Forth OS, QEMU + this
can be used to skip the whole modern PC initialization mess.
WebAssembly would also be another great target option
There is also Pure64 to handle initialization, but the initialization is only tedious, given some architectural knowledge, whereas any more or less modern x86-64 assembly is pure madness. (I think I have two half-functional Forth assemblers for x86 where the instruction descriptions turned out to not be expressive enough in the middle of the table; 80186 is easy, though.)
Frankly, I don't think getting more targets up is that difficult, it's the first target that's the problem. For a book, you might want to choose something clean — ARMv4, maybe, or MSP430, or possibly even the JVM. I looked at WebAssembly, and it looks rather convoluted to me, but it might be the sorry state of the documentation.
Obviously anything someone would like to do to further Forth or get new people understanding it is great; I commend - and would endeavor to assist with - any project like this where I could.
That said, I personally feel that if there was Forth "book" that needed written, it would be to take Brad Rodriguez's very good, and yet very dated, Moving Forth web pages and update them both in terms of focus (the documents do bounce around a bit) and their targets; while the Z80 and 6502 are still quite popular for emulating and hobbyists, can you even get a Z8 or PDP-11 any more? Certainly teaching with code that targets C or JavaScript would help many newcomers.
His pages are very old, and I'm constantly in a minor state of panic thinking the domain could be lost at any time.
Of course, as I type this, all I can think to myself is, "if you want this, why not just do it instead of hoping someone else does?" ;-)
@massung You'll be happy to hear that the indefatigable Juergen Pintaske has reformatted Rodriguez' texts and published them in e-book format on Amazon (https://www.amazon.de/Moving-Forth-Internals-Processor-English-ebook/dp/B07BTV7QVR/) as part of his Forth Bookshelf.
@alex-shpilkin I agree with you that x86 assembler is, ah, daunting. One possible target, especially for a book such as this, would be the free RISC-V ISA (https://riscv.org/). It seems to be gaining real-world traction with Nvidia and Western Digital getting behind it for some of their projects, and there is now actual silicon out in the world that you can buy (https://www.sifive.com/products/hifive1/). This would allow you to create a bare-metal Forth with a more modern slant.
Sorry, I just remembered that Mecrisp-Forth has already been (experimentally) ported to RISC-V assembler, see https://www.forth-ev.de/article.php/20180306213428204 for the announcement (in German) and https://sourceforge.net/projects/mecrisp/files/ for the actual files.
@scotws Wow, thanks for that link! I'm so happy that is the case. While Kindle is fine, if there was a physical copy I would have clicked "buy" already. ;-)
@Brad-R would you be interested in donating content or helping out? Moving Forth would be a great guide and template for an updated book in the same vein
x86 is boring, but still we all use it. i guess there's arm and mips could be considered too, also the riscv, since the author of qemu has wrote a online emulator for it, its easy to try for us
for arm, there's rpi and many other clones, for mips just check our wifi aps. and for the internet of things, please give esp8266 a chance, i knew someone has already done(punnyforth)
also i am very glad you mentioned unikernel. its a worth to try. finally there's someone focus on cloud instead of his outdated microcontroller
give esp8266 a chance, i knew someone has already done(punnyforth)
CForth too.
@quozl thanks for completed me. but i prefer asm based implementation, although c implmented one might be more portable(which made me strugle :[ )
I am the author of an ASM-based Forth system (Forthmacs) that has been ported to, at least, 680x0, SPARC, MIPS, x86, PA-RISC, PowerPC, Alpha, SaxPY, ARM32, and ARM64. That system has served as the base of an application (Open Firmware) that shipped on several million computers from several different companies.
I'm also the author of CForth, a C-based Forth system that runs on pretty much anything.
As such, I know quite a bit about the tradeoffs between ASM and C based Forths.
Personally, I rarely use the ASM based system anymore. All the tools that I care about, with the sole exception of the interactive assembly language debugger, run the same on both, sharing most of the same source code. There was a time when an ASM Forth was quite useful to me - when my job involved debugging "first silicon" of new microprocessors. At that time, one often could not trust C compilers to generate correct code, and even when they did, it was helpful to dissect problems down to the assembly language level when you could not trust the silicon.
Those days are over. It has been nearly two decades since I have had my hands on first silicon of a new micro. I'd venture to guess that few people on this list have ever had that privilege/responsibility. Furthermore, C compilers are extremely good now, often generating better code than most humans can. At the beginning of my career, there was a time when I had found at least one code generation bug in every compiler I had ever used. Now, I can't remember the last time I found such a bug.
The turning point for me came when I needed to convert a hand-code ASM program for AVR into C. That program, which I had written, had a tightly coded inner loop that did bit timing for a pulse-position infrared modulator. It was becoming difficult to maintain because of an ever-increasing feature set. It needed to be rewritten in C for ease of maintenance, but I didn't have time to go through a full testing and qualification cycle for the new version. Realizing that the AVR C compiler generated good code, I was able to write the C code so that the compiler generated exactly the same - byte for byte - machine code as the old ASM version. So we were able to switch over to the new code base with no additional testing. From then on, new mods could be tested as time and schedule permitted.
With CForth, I have been able to target new instruction sets essentially instantly. That happened with at least ARM-THUMB (a different instruction encoding than ARM32), x86-64, xtensa (esp8266 and esp32). At this point I have just stopped caring about instruction sets. For a lot of my career they were extremely important; now they are uninteresting.
I agree with @MitchBradley. Unless you have a good reason to code in assembly, it's not worth the bother or the slow down you get by avoiding compiler optimisers. If it is worth it, then at least only in Forth words that are critical to an application, and only if the speed improves. Hand assembly just doesn't always help.
no i am not prefer asm for showoff, its just for me, using asm could understand much of the details.
also i'd saw some embed project's c code(maybe 8051), which's code organization is really ugly, i think that might be the source of bad impression of embeding c code
also i prefer tiny things, not only the executable , but also the tools, this is also the reason i like forth
dont worry, i dont hate cforth, its just if i need to gave my choice for only 1 forth, then i prefer asm implementation
Using C in a step kind of defeats the point of Forth - a self contained system that is NOT yet another variation on the Bell Labs computing vision
The only other system that has cultural similarity with Forth is Lisp and that's on another totally different abstraction level altogether. Beyond the islands of Forth and Lisp, everything else is just a sea of curly braces. I have nothing against C per se just that the whole point of Forth is that it brings something new to the table, a table that's currently only occupied by C/C++ and perhaps now Rust.
Though we could always just bring in LLVM..
C has been described as a "universal assembler". It works reasonably well for that purpose. You can use as few runtime library routines, or as many, as you wish. I have done pretty much every variant, as needs demanded, from bare-metal where my code was running starting at the reset vector, with zero dependence on any code that I didn't write, to fully-hosted systems where Forth sits on top of a huge stack of external OS code, to Forth inside the OS kernel acting as a resident debugger, to tethered systems where Forth runs on a host and takes control of an app on another processor via a tiny stub of a few hundred bytes of code.
Your idea of "the purpose of Forth" is obviously quite different from mine. For me, Forth is a way to do whatever I need to do at the moment. Sometimes it's bare metal, sometimes it's a small part of an enormous system.
@Brad-R MSP430 is great, actually—even a full commented listing of the whole kernel (a la TeXbook appendix B) should be short enough, although that’s not probably not what you’re going for. It also has very real implementations and development tools, unlike the RISC-V.
One request: if the ebook prepared by Pintaske is going to be the definitive version, could it be released in a DRM-free way? I’m fine with paying for it, but don’t like restrictions that prevent proper archiving.
@MitchBradley good point! What would be your recommendations for the book?
I think there are enough Forth implementations already. I would like to see more activity around applications.
I think there are enough Forth implementations already. I would like to see more activity around applications.
There's one more Forth option for the ESP8266:
https://github.com/niclash/forthright