onetrueawk/awk

How can we build awk under Windows?

d0vgan opened this issue · 9 comments

d0vgan commented

Ideally I'd like to build awk under Windows using the Visual Studio Developer Command Prompt to produce a native build... but how?

plan9 commented

I have no idea. I don't work with Windows systems. perhaps some other contributor will reach out to you. [i will leave this issue open for a few days]

  • 👍1
d0vgan commented

After some experiments, I partially succeeded. Here is the instruction:

  1. Get the latest awk sources using git clone https://github.com/onetrueawk/awk.git. Let's assume the sources are cloned to C:\prj\awk.
  2. Download the Bison's Binaries plus Dependencies from https://gnuwin32.sourceforge.net/packages/bison.htm. I've downloaded two archives: bison-2.4.1-bin.zip and bison-2.4.1-dep.zip.
  3. Create a folder C:\prj\awk\bison-2.4.1. Unpack the bison-2.4.1-bin.zip and bison-2.4.1-dep.zip into that folder.
  4. Under C:\prj\awk, Modify the file proto.h by adding the following at the beginning of the file (after the comments and before the first extern):
#ifndef __attribute__
/* This feature is available in gcc versions 2.5 and later.  */
# if (! defined __GNUC__ || __GNUC__ < 2 \
      || (__GNUC__ == 2 && __GNUC_MINOR__ < 5) || __STRICT_ANSI__)
#  define __attribute__(Spec) /* empty */
# endif
#endif
  1. Modify the file run.c by adding the following in the beginning of the file (after the last #include statement):
#ifndef F_SETFD
#define F_SETFD 2
#endif

#ifndef FD_CLOEXEC
#define FD_CLOEXEC 1
#endif
  1. Under C:\prj\awk, create a file build.bat with the following content (it corresponds to the makefile):
set PATH=%PATH%;.\bison-2.4.1\bin
bison.exe -d awkgram.y
cl maketab.c
.\maketab awkgram.tab.h >proctab.c
cl /O2 awkgram.tab.c b.c lex.c lib.c main.c parse.c proctab.c run.c tran.c /out:awk.exe
  1. Run Developer Command Prompt for VS and execute these commands in it:
cd /D C:\prj\awk
build.bat

As the result, I'm getting:

lib.obj : error LNK2019: unresolved external symbol _strncasecmp referenced in function _is_valid_number
main.obj : error LNK2019: unresolved external symbol _srandom referenced in function _main
run.obj : error LNK2001: unresolved external symbol _srandom
run.obj : error LNK2019: unresolved external symbol _popen referenced in function _openfile
run.obj : error LNK2019: unresolved external symbol _pclose referenced in function _closeall
run.obj : error LNK2019: unresolved external symbol _WIFEXITED referenced in function _bltin
run.obj : error LNK2019: unresolved external symbol _WEXITSTATUS referenced in function _bltin
run.obj : error LNK2019: unresolved external symbol _WIFSIGNALED referenced in function _bltin
run.obj : error LNK2019: unresolved external symbol _WTERMSIG referenced in function _bltin
run.obj : error LNK2019: unresolved external symbol _random referenced in function _bltin
run.obj : error LNK2019: unresolved external symbol _fcntl referenced in function _openfile

Any hints?

enh-google commented

given all the unix stuff awk uses, i think you'll want to use cygwin rather than visual c++. (otherwise you're going to have to implement all the unixy stuff yourself. which is easy enough for strncasecmp() or srandom()/random(), but not for popen()/pclose(), the wait() stuff, or fcntl().)

  • 👍2
wajap commented

With MSYS2 and MINGW64 one can build a native windows awk. Native means that it uses the msvcrt.dll already available in windows and does not need extra dll's. Some changes to the source are required to accomplish this.
The differences can be found in the file awk_win32.diff.gz included in this message (at the bottom).
It also has some changes to allow it to run the tests using the windows awk, but not all tests are yet included (T.*are not executed when using the windows awk).
In nix systems the source still can be compiled and with the produced awk the tests T. are used. (Was tested on WSL2/Ubuntu.)
MSYS2 can be installed using the installer found in https://www.msys2.org/. Run the installer. When ready a terminal will be launched.
If the terminal is not there from the "MSYS2" start menu choose "MSYS2 MING64".
Before installing the tools needed to compile awk, is better to update MSYS2 by the command pacman -Suy.
It may ask you to confirm to close the terminal. From the "MSYS2" start menu choose "MSYS2 MING64" to relaunch it.
Repeat pacman -Suy until everything is up to date.
Install the build tools and compiler with command pacman -S base-devel mingw-w64-x86_64-gcc.
If git is required it can be installed with pacman -S git.
Download the code from https://github.com/onetrueawk and put the contents of the zipfile to a folder of your choice (example C:\prj\awk). One can also use git: git clone https://github.com/onetrueawk/awk.git.
Put the awk_win32.diff.gz file also in the same folder.
In the terminal use cd to move to the folder where you have put the code.
The drives like C: are represented by /c/. So to go to the example use: cd /c/prj/awk.
To perform the patch run gunzip -c awk_win32.diff.gz | patch -Np1.
Now the a.exe can be created by using make -f makefile_win.
A small test can be done by moving to the folder bugs-fixed (cd bugs-fixed) and performing ./REGRESS.
This should give:
=== a-format.awk
=== concat-assign-same.awk
=== decr-NF.awk
=== fmt-overflow.awk
=== fs-overflow.awk
=== getline-corruption.awk
=== getline-numeric.awk
=== inf-nan-torture.awk
=== missing-precision.awk
=== negative-nf.awk
=== nf-self-assign.awk
=== numeric-fs.awk
=== numeric-output-seps.awk
=== numeric-rs.awk
=== numeric-subsep.awk
=== ofs-rebuild.awk
=== pfile-overflow.awk
=== rs_underflow.awk
=== space.awk
=== split-fs-from-array.awk
=== string-conv.awk
=== subsep-overflow.awk
=== system-status.awk
Files system-status.ok and system-status.OUT differ
++++ system-status.awk failed!
=== unary-plus.awk

One sees most tests are ok. The only one failing is system-status.awk. But this uses the "kill" command, which is not available in windows.
To be able to run the other tests an awk ("oldawk") is needed. A logical candidate is awk build as MSYS2 application.
In principle the source does not have to be changed for this build, but when compiling I got warnings regarding the ctype.h functions. For this reason a cast to uschar was added to the ctype.h functions.
The awk build as MSYS2 application needs the msys-2.0.dll which in a MSYS terminal is on the path.
To be able to build this one has to install gcc: pacman-S gcc.
After installation there are two gcc.exe's. One is /mingw64/bin/gcc.exe (to build the native windows programs) and one in /usr/bin/gcc.exe (to build MSYS2 programs).
The first step is to close the terminal and from the "MSYS2" start menu choose"MSYS2 MSYS". This will launch a terminal with the proper path to the gcc required.
Then save the a.exe by mv a.exe awk.exe.
Then issue make cleaner to remove files created by the previous build.
Next do make. (so without -f makefile_win)
An a.exe should have been created. If a.exe is not created, check which gcc is used using the command which gcc. This should give /usr/bin/gcc.
Rename a.exe and put on the path by mv a.exe /usr/bin/nawk.exe.
Clean up using make cleaner.
Move saved awk back mv awk.exe a.exe.
Run tests: oldawk=nawk make check > check.out 2>&1
The tt.* test times from nawk.exe seem to be about 10x bigger than real and too short from the windows native awk.
This is caused by the local time.c which is compiled as a MSYS application. Why is not clear to me.

I am busy adapting the T.* tests to be able to run with the windows native awk.

Two issues I have found during testing:

  1. The result of rand() differs in the windows version from that of the MSYS version.
    For this reason I modified t.randk and p.48b.

  2. The awk script below gives in the *nix and MSYS awk three lines, but with the windows native awk only the first line is given.
    The window versions of gawk and mawk (https://github.com/p-j-miller/wmawk2) also only give the first line.
    BEGIN {while ("echo A\necho B\necho C" | getline > 0) print}

    When the "\n" are replaced by a "&" and a "&" is put at the end of the string, it seems to work with all awk versions (including gawk and mawk).
    BEGIN {while ("echo A&echo B&echo C&" | getline > 0) print} 

awk_win32.diff.gz

  • 👍1
d0vgan commented

With MSYS2 and MINGW64 one can build a native windows awk.

This is brilliant!
The instructions are absolutely clear, the only thing I additionally needed was to execute

PATH+=:/c/msys64/mingw64/bin

right before

make -f makefile_win

because otherwise I got "-bash: gcc: command not found".
As for the last point about "oldawk" vs "nawk", I should confess I did not understand that. Is it all about the tests or about performance as well?
The ideal output for me would be a 32-bit executable compatible with Windows XP or at least with 32-bit Windows 7. Though the world moves forward and probably today no one is interested in 32-bit Windows 7. (Actually, my daily PC uses 64-bit Windows 10, but an older one has 32-bit Windows 7 and does not physically support more than 4 GB RAM).

Looking at the diff (the "awk_win32.diff.gz"), I believe it is worth to include these changes into the official master branch since they either fix variable types or Windows-specific things under the corresponding #ifdef.
Maintainers, please consider these these changes!

wajap commented

You can avoid setting the path by choosing the right option from the MSYS2 start menu: MSYS MINGW64
The prompt should be user@PCname MINGW64

The "oldawk" vs "nawk" is concerning the tests and is not relevant for building awk.

The executable which is built is 64 bit.
For a 32 bit awk you have to do:
pacman -S mingw-w64-i686-gcc
From the MSYS start menu use MSYS MINGW32 this gives user@PCname MINGW32 as the prompt.
To check use which gcc this should give /mingw32/bin/gcc
If that is ok make -f makefile_win will build a 32bit awk.

  • 👍1
d0vgan commented

For a 32 bit awk you have to do: pacman -S mingw-w64-i686-gcc

The 32-bit build is compatible with Windows XP! (Just tested that under VirtualBox).

wajap commented

The patch above is for version 20221215.
For version 20230911 ( or https://github.com/onetrueawk/awk/archive/refs/tags/2ndEdition.zip ) use the one below:
awk_win32.diff.gz

plan9 commented

I hope everyone is happy with the contributions on this discussion. closing the issue.