migrating GCC program from PDCURSESMOD 4.2.0 to 4.3.3 unexpected results
chuck-haatvedt opened this issue · 29 comments
I have a WYSIWYG screen painter program that I wrote using MinGW with GCC version 6.3.0 and PDCursesMOD 4.2.0, I recently built GNUCOBOL with OSDN version of MinGW which comes with GCC 9.2.0 and I downloaded the source for PDCursesMOD 4.3.3, the build was successful.
Should I be able to compile my paint.c program with GCC 9.2.0 and PDCursesMOD 4.3.3 and expect the same behavior as I had with the GCC 6.3.0 and PDCursesMOD 4.3.3 ?
With GCC 6.3.0 & PDCursesMOD 4.2.0 I have used putwin function to save a window to a file. However after compiling with GCC 9.2.0 & PDCursesMOD 4.3.3 I found that I can now longer load the window using getwin function.
paint.c ===> source code
ACCTMNU.PDC_WINDOW ==> file created with putwin on GCC 6.3.0 & PDCursesMOD 4.2.0
paint-keys.pdf ===> documentation for usage of paint program.
Should I be able to compile my paint.c program with GCC 9.2.0 and PDCursesMOD 4.3.3 and expect the same behavior as I had with the GCC 6.3.0 and PDCursesMOD 4.3.3 ?
Yes, as long as you pass the same set of options during make. Which options do you use?
If this isn't the case then I'd say that's a bug.
Note: GnuCOBOL's cobcrun --verbose --info will show at least some of the defines used for compiling and most important: any mismatch it sees (obviously only the "screen io" parts are relevant here).
More important note: The question is different from the title, which is not about "recompilation of PDCursesMod 4.3.3 with a newer GCC version" but about "recompile with a newer PDCursesMod" - what is your actual question?
For the later one: hopefully you do not get the exact same results everywhere (as otherwise it wouldn't make sense to update PDCursesMod), putwin and getwin should work identical in general.
Sorry about the confusion Simon, the issue is regarding the new version of PDCursesMOD 4.3.3 not behaving the same as the previous version 4.2.0 .... However there is also the change in GCC as with Arnolds build they were using MinGW with GCC 6.3.0 and the new build guide uses MinGW from the OSDN website which included GCC 9.2.0
The program that I'm encountering different behavior is paint.c which does not involve the gnucobol compiler at all. In fact I do not use the GNUCOBOL compiler when compiling it although I know that I could as it would just invoke the C compiler anyway.
`Copy "PDCursesMod-4.3.3.tar.gz" into your C:\MinGW\MSYS folder using Windows commands,
and then use the following bash shell commands to uncompress it:
cd /mingw/MSYS
tar xzf PDC*.tar.gz
rm PDC*.tar.gz
After you have unpacked the tarball, you would use the following commands to build PDCurses:
cd PDC*/wincon # note changes
make -f Makefile INFOEX=N CHTYPE_64=Y DLL=Y
Then use the following command to verify that a "pdcurses.dll" exists in the
"C:\MinGW\MSYS\PDCursesMod-4.3.3\wincon" folder.
ls *.dll`
I do see two possible problems here.
First, the window is written out and read in as a stream of chtypes. If you created some files containing window data built with 32-bit chtypes and read them in with 64-bit chtypes, you'll get garbage; the input and output chtypes have to match. So if you made those files with WIDE on or off, then read them in with WIDE off or on, you'll have troubles.
Similarly if you switched from 64-bit to 32-bit chtypes, or vice versa.
Second, the WINDOW structure changed in 4.3.3 (added _pminrow through _smaxcol; see commit 41da0b2). putwin() and getwin() write out that structure "raw". So a file created using 4.2 will defiinitely fail with 4.3.3, no matter which flags you use.
There is a version byte within the file that is supposed to be updated whenever the WINDOW struct changes. Except I didn't know about it and didn't update it. Had I done so, getwin() would have returned ERR.
So, to summarize, I should increment DUMPVER and, I think, add in a byte or two to specify wide vs. narrow and 64-bit vs. 32-bit chtypes, so that files compiled with one set of flags won't be read with a different set of flags. It'd also be good if sizeof(WINDOW) was written to the file and checked; that would have caught this issue even if DUMPVER wasn`t incremented... let me get back to you on this one.
One point I should check on, though : am I correct that this problem occurs when reading "old" files with "new" code? If you call putwin() and then getwin(), both from the same program and version, does that work?
Thanks for inspecting this! Yay, versioning issue. In theory we should get out a fixed 4.3.4 asap :-(
Question @Bill-Gray: When now doing the changes outlined above, wouldn't it be possible in getwin() to explicit:
- map from "real version 1" to version 2?
- map from version 2 32bit to version2 64bit chtypes and possibly even vice versa?
Bill, with the 4.2.0 version i added a printf("size of chtype is %d \n", sizeof(chtype));
it printed 8 so i'm guessing that it is using chtype=64 which is what I did when i build pdcursesmod 4.3.3 and that matches the build instructions which Arnold Trembley has for when he build pdcursesmod 4.2.0
your explanation about the change in window format in 4.3.3 explains why the getwin() failed with the file created by 4.2.0 putwin()
however there are other issues, like some keystrokes are not recognized by the 4.3.3 version Ctrl + C for example. also when using Shift + End to highlight to the end of the line, it goes outside the window to the terminal console...
also the reverse video attribute appears to change the foreground and background colors as well..
here is the instructions I followed...
`Copy "PDCursesMod-4.3.3.tar.gz" into your C:\MinGW\MSYS folder using Windows commands,
and then use the following bash shell commands to uncompress it:
cd /mingw/MSYS
tar xzf PDC*.tar.gz
rm PDC*.tar.gz
After you have unpacked the tarball, you would use the following commands to build PDCurses:
cd PDC*/wincon # note changes
make -f Makefile INFOEX=N CHTYPE_64=Y DLL=Y
Then use the following command to verify that a "pdcurses.dll" exists in the
"C:\MinGW\MSYS\PDCursesMod-4.3.3\wincon" folder.
ls *.dll`
i'll download version 4.2.0 and build pdcursesmod with that using OSDN MinGW and see that resolves the issues... I'll report back to here with the results...
if that works, which is unknown at this time, then I'll proceed with testing 4.3.0 and each succeeding versions until I find where it breaks...
Also, I'm building a WYSIWYG paint program and if the format of the WINDOW file created by putwin() changes and is not usable by a later version, it would be nice to know that .... even better would be if there were some type of conversion program.
One point I should check on, though : am I correct that this problem occurs when reading "old" files with "new" code? If you call
putwin()and thengetwin(), both from the same program and version, does that work?
when calling putwin() and getwin() from the same program and version it works as expected... at least on 4.2.0
i've encountered other issues with 4.3.3 not capturing some keystrokes the same as in 4.2.0 so i've not tried the putwin() and getwin() on 4.3.3 .... i've only tried getwin() on 4.3.3 using a file created but putwin() on 4.2.0 and that failed as you mentioned earlier...
@GitMensch , yes, this'll result in a new release.
At least in theory, we could read "old-format" files and, with decent reliability, set the six new values reliably. For that matter, the six values in question are only used for pads and sub-pads; we could set them to zero in such cases and say "sorry, you can't read old-format sub-pads".
The file size of an old-format file would tell us if we were looking at 64-bit or 32-bit chtypes.
I'm thinking about how to handle the 32-bit/64-bit wide/narrow chtype differences. The solution I would see would be that the files would always contain 64-bit chtypes; compile with 32-bit chtypes, and the values are expanded upon writing and "shrunk" upon reading. We'd do a check to see if any bits got lost in reading. This could happen if, for example, the character value was >= 256 in a wide-character build and you tried to read the result in a narrow-character build. It could also happen if you stored the overlined, dimmed, or strikeout attributes set and tried to read them into a 32-bit build; or if you saved a window with more color pairs than the program reading the file could handle.
At present, I think this would get us into a real hairball of code, with lots of different configurations to test and many opportunities for error and overly complicated code. Still thinking it through.
The man page for ncurses' getwin() and putwin() says that wide and non-wide builds of ncurses use different output formats, but that you can transfer data between them. Interestingly, only top-level windows are used, evading the entire subwindow problem.
We will share with ncurses the limitation that our color table isn't stored as part of the window.
@chuck-haatvedt , I'll have to get back to you on the Ctrl-C and Shift-End issues. I'm currently testing WinCon and WinGUI in Wine (no "real" Windows available to me right now). On that, Ctrl-C always breaks (a Wine thing.) Shift-End gets no response at all, which seems strange.
The reverse-video attribute should simply swap foreground and background. I gather that's not what you're seeing?
Hello Bill,
the window I saved had a solid black background and a bold green foreground.
I tested versions 4.3.0, 4.3.1 and 4.3.2a and all of them had the same issue with the colors and Ctrl-X, Ctrl-C, Shift left and right arrow key and some of the other Ctrl - keys that I use. I use the PDC_get_key_modifiers() function to interpret the modifier key combinations. For example the right arrow key would wrap to the beginning of the next line when it reached the right window boundary. However the shift right arrow key would continue beyond the right boundary of the window even though there is code to check for that.
only the 4.3.0 version was able to load the window file created by the 4.2.0 putwin() function. However the colors changed to blue instead of green and the reverse video actually changed the forground to red and the background to white (my memory is a bit foggy on the exact colors but it was not just a reverse video).
Hi Chuck,
I tracked down a Win1 0 box, and found that Shift-End was detected in WinCon. This was tested using testcurs; go to the "Input Test" section and hit Shift-End. Shift-arrows also worked (though if you mean Shift-arrows on the numeric keypad, those do get translated; Shift-Left becomes 4, for example.)
Ctrl-C and Ctrl-V are now used for copy/paste (commit 9f487d4). Used to be Shift-Ctrl, and still is on non-Windows builds. But there were Windows people who'd never heard of this Shift-Ctrl-C or V thing.
It's entirely possible that some keys are not returning what they did a couple of years back. I vaguely recall some issues in "upstream" PDCurses that required William McBrine to modify the behavior, and PDCursesMod followed suit.
It sounds to me as if you're running into much more misbehavior than I'd have expected. But checking to see what testcurs shows in the input test would be a good next step.
On the color change front... when you re-loaded the window, had you set color pairs in the same manner? (putwin() will store the fact that a given character has color pair 3, say, but it doesn't store the fact that color pair 3 means a white foreground and black background.)
On PDCursesMod, an uninitialized color pair will be shown as red text on a blue background (blue on red if A_REVERSE is set). See the init_col demo for an example (it intentionally draws some text with an uninitialized color pair just to test that out).
Similarly, if you'd had color pair 3 set to be red on black when you called putwin(), but white on yellow when you called getwin(), you'd have exactly this sort of issue.
At least in theory, we could read "old-format" files and, with decent reliability, set the six new values reliably. For that matter, the six values in question are only used for pads and sub-pads; we could set them to zero in such cases and say "sorry, you can't read old-format sub-pads".
The file size of an old-format file would tell us if we were looking at 64-bit or 32-bit
chtypes.I'm thinking about how to handle the 32-bit/64-bit wide/narrow
chtypedifferences. The solution I would see would be that the files would always contain 64-bitchtypes; compile with 32-bitchtypes, and the values are expanded upon writing and "shrunk" upon reading. We'd do a check to see if any bits got lost in reading. This could happen if, for example, the character value was >= 256 in a wide-character build and you tried to read the result in a narrow-character build. It could also happen if you stored the overlined, dimmed, or strikeout attributes set and tried to read them into a 32-bit build; or if you saved a window with more color pairs than the program reading the file could handle.
If you add "use only big-endian in the output format and, if necessary, convert during export/import" that would be fully inter-operational!
At present, I think this would get us into a real hairball of code, with lots of different configurations to test and many opportunities for error and overly complicated code.
So, what is the actual plan for "soon"?
"Soon" turned out to be "now". It's slightly uglier than I'd like, but not actually all that bad.
Commit 6824e49 revises the on-disk format to store the WINDOW structure (and everything except chtype values) in ASCII. (The binary representation isn't guaranteed to match between compilers, much less OSes, wide vs. narrow, or 32-bit vs. 64-bit chtypes.) An ASCII representation should also improve our odds of getting backward compatibility when things change in the future.
chtypes are always stored in eight bytes. On 32-bit builds, we pad them out on writing and shrink them down on reading. I don't (at least yet) check to make you haven't done something like creating a file with colors or characters or attributes not supported in your current configuration. Nor am I byte-swapping on big-Endian platforms yet (should be easy to do, but I've no way of testing it).
I also have not tried to get it to read the previous all-binary format. The odds of getting that to work are remote. In particular, there is no hope of distinguishing 32-bit narrow from 32-bit wide chtypes in the old format.
Hi Chuck,
I tracked down a Win1 0 box, and found that Shift-End was detected in WinCon. This was tested using
testcurs; go to the "Input Test" section and hit Shift-End. Shift-arrows also worked (though if you mean Shift-arrows on the numeric keypad, those do get translated; Shift-Left becomes 4, for example.)Ctrl-C and Ctrl-V are now used for copy/paste (commit 9f487d4). Used to be Shift-Ctrl, and still is on non-Windows builds. But there were Windows people who'd never heard of this Shift-Ctrl-C or V thing.
It's entirely possible that some keys are not returning what they did a couple of years back. I vaguely recall some issues in "upstream" PDCurses that required William McBrine to modify the behavior, and PDCursesMod followed suit.
It sounds to me as if you're running into much more misbehavior than I'd have expected. But checking to see what
testcursshows in the input test would be a good next step.
Bill, does this mean that my C code can no longer detect the Ctrl + C, Ctrl + V and Ctrl + X key combinations ? currently my code detects them and is coded to take certain action based on a user entering them. If the C code can no longer detect them would the "raw()" function allow me to do that ? Currently I use cbreak();
I use a Microsoft Ergonomic Natural 4000 keyboard and it has a set of the 4 arrow keys separate from the numeric keypad, those are what I'm using not the ones on the numeric keypad...
Also regarding colors, I do programmatically initialize the 17 colors and also the color_pairs before calling getwin() to load the window file.
if you can compile and run my paint program on 4.2.0 first and observe the colors and the user guide for key commands / functionality the differences are pretty clear.
if it would be helpful I could join you on SKYPE to walk thru the issues I'm encountering.
Perhaps I should have premised this discussion by clarifying that I'm not a C programmer but rather a retired IBM mainframe COBOL programmer. So my C code is not what an experienced C programmer would create.
PS.. I did manage to build GNUCOBOL with PDCursesMod 4.2.0 so at least I have a working environment
let me know if you were able to test with the paint.c program which I provided ...
Also if there is any testing you would like me to do, please let me know.
here is a screen shot of what should appear when loading the ACCTMNU window.
Bill,
I may have had an issue with the way I built PDCurses in MINGW, please wait and don't spend any more of your time on this until I can rebuilt my pdcurses environment and do some more additional testing.
I apologize for wasting your time on this...
thanks,
Chuck Haatvedt
Hi Chuck,
No worries... actually, I had "day job" issues crop up which kept me from looking further at this anyway (and probably will keep me from it for a few days). I'll await word.
Two things, though. It occurs to me that we can load "old format" window images, on the assumption that chtype in the old-format file was the same as that in the new-format file. Generally speaking, that'll be a safe assumption; you probably haven't switched between 32-bit and 64-bit chtypes, or from wide to 8-bit characters, between the time you made those files and now. In other words, the old code could still read those files (with some minor modifications to check for the WINDOW struct having changed).
Dunno if being able to load "old format" images actually matters for you, but if it does, don't give up hope quite yet.
And second... I've not had a chance to investigate the ability to catch Ctrl-C and friends in WinCon. I'm pretty sure they default to being mapped to the copy and paste functions. But I think you can just say "PDCursesMod, please don't intercept Ctrl-C and Ctrl-V and assume they're for cut-and-paste; instead, use the following characters". I think calling
PDC_set_function_key( FUNCTION_KEY_COPY, 0);
PDC_set_function_key( FUNCTION_PASTE, 0);will disable keyboard cut/paste, by mapping both functions to non-existent keys; after that, getch() will return Ctrl-C and Ctrl-V. (And changing those numbers to 0x03 = Ctrl-C and 0x16 = Ctrl-V should set them back again.) Haven't had a chance to try it, though.
Bill with the above PDC_set_function_key statements I was able to get version 4.3.3 working correctly...
however I did have to change from using cbreak() to raw()
for example here is what happens with raw() when putting the cursor on column 0 (the left edge) and hitting Shift + END
changing raw() to cbreak() and repeating the same process... Note that the reverse video changes the colors and the highlighted text extends beyond the right edge of the window.. Also with cbreak() the Ctrl + C key now aborts the program...
Bill, thanks for your patience on helping someone with very little C knowledge get this working. I very much appreciate it.
So I have a usable program by changing cbreak() to raw() .
However I suspect there is something unexpected in the behavior change between raw and cbreak. I did not expect it to change colors on reverse video attribute and write outside the window boundary.
As to converting old files created by putwin(), that is probably not needed now as I have not deployed this suite of application to the public.
However in the future, I would like to be able to update PDCURSESMOD to the most recent version while still being able to use WINDOW files created by versions after 4.3.3 if possible.
If that is not possible another option would be to have a conversion program to convert WINDOW files created by putwin() to the current format. This would allow for an upgrade path for my application in the future
I would like to build my GNUCOBOL 3.2dev compiler with PDCursesMOD 4.3.3 however until I hear back from you about the raw() / cbreak() behavior, I will wait on that.
Yup, you should use raw() for this. From MANUAL.md :
raw() and noraw() toggle raw mode. Raw mode is similar to cbreak
mode, in that characters typed are immediately passed through to the
user program. The difference is that in raw mode, the INTR, QUIT,
SUSP, and STOP characters are passed through without being
interpreted, and without generating a signal.
My recollection is that we had an issue with this involving WinCon, and the fact that you could hit Ctrl-C and not get it to send a signal. See wmcbrine/PDCurses@793556fff and commit f524be6.
However in the future, I would like to be able to update PDCURSESMOD to the most recent version while still being able to use WINDOW files created by versions after 4.3.3 if possible.
I do expect that future changes ought not to break window files. The new scheme of having the header written out in ASCII rather than binary ought to cover most "breaking" cases I can think of. I would invoke the long-term stability of PDCurses and PDCursesMod (after some decades, there's not much reason to make breaking changes)... an argument I could make more strongly if it wasn't for the fact that I did make exactly such a change when I added six integers to the WINDOW struct.
But it's still the case that we really try to maintain backward compatibility to the Stone Age.
That does leave us this odd issue with A_REVERSE and writing outside the window boundary. I'll (still) have to get back to you on that one. Just spent a couple of days wrestling with a nasty day-job bug, and I don't think I'm quite done yet... I can answer the other issues off the top of my head, but not this one! (Looks like it may have something to do with the select/cut/paste capability, though, which is a "screen" thing rather than a "window" thing.)
Bill, since that is exactly what my code is doing, select / cut / copy / paste.... as long as it works in "raw()" mode, I would not spend much time on this since the code is working as expected with RAW mode.
I don't know if you have a little temporary program to convert from 4.2.0 WINDOW format to the current 4.3.3 WINDOW format, however if you do would you mind sharing it ?
if you don't have one then don't spend any time on it as I can recreate the couple of windows that we've been using for testing.
thanks
Chuck Haatvedt
I may try for a separate converter routine. The problem (at present) is that implementing getwin(), in either its old or new forms, requires access to library internals. So the only way I was able to do it was to modify getwin() itself, to read either old-format or new-format files.
UnZIP the following, and use the resulting scr_dump.c in place of the one currently in pdcurses/scr_dump.c :
and you can then getwin() your old (binary-formatted) file, then putwin() it to a spandy-new ASCII one. Some care is taken to ensure that changed WINDOW structs don't cause grief.
I'd rather not carry around the extra code involved for reading old-format files in the library. It's slightly bulkier and messier than I'd really want. However, a separate converter utility remains a possibility, should we eventually decide we'd like to have one... but the current workaround ought to fill the immediate requirement to just convert a few files.
Bill,
don't bother with any logic to convert old WINDOW files. I was able to use the HxD hex editior program to insert 24 bytes of binary zeros at offset 0x64 of the old WINDOW file and after that I was able to use it successfully in version 4.3.3 so all is well... Nothing needed for old file conversion.
plus I only have a few old format WINDOW files to convert and I can do that with HxD pretty easily now.
Hi Chuck -
Well, yes and no. At some point, you may want to use the latest code, with ASCII screen files. For that, you'll need to be able to read in your binary files and output them as ASCII. But it should be a one-time conversion; as noted, the new format ought to be (much more) resistant to breakage than the previous binary format was.
That does leave us this odd issue with
A_REVERSEand writing outside the window boundary. I'll (still) have to get back to you on that one.
With the "compatibility" thing being clear now - this is the one that is still open (and likely to be for some days), right?
Not sure, so asking: what is the state of this issue?
Hmmm... looking through this, it really became three issues :
- Storing/restoring windows, with a breaking binary change. I think we call that 'fixed'.
- Capturing 'special' keys. This works if you call
raw(), which is as intended. - The only unresolved bit I see is the
A_REVERSEwriting outside the window boundary. But on further examination, I think I simply misunderstood what's going on here.
Looking at Chuck's screenshots from July 21, it looks as if he clicked on the left edge of the window, hit Shift-End, and got the entire line highlighted. That's exactly what should happen. PDCurses and PDCursesMod do text selection based on the entire screen; that part of the code doesn't understand about windows.
This is obviously a problem in any case where the window doesn't happen to conveniently span the entire screen horizontally. You'd sort of want a 'windowed select' mode : when you click the mouse, we determine which window you're in and the highlighting and text selection occurs within that window. But I don't expect we'll do that any time soon. (We do have a leg up on it, now that we actually maintain a list of windows. We did that to be standards-compliant and delete windows when we should and not when we shouldn't. But it would also help for this "limit selection to the window" problem.)
So we have two parts fixed and this highlighting/selection part basically a WONTFIX, unless somebody has a "Eureka" moment and comes up with a plausible way of doing it. Overall, I think we can call this issue 'closed as completed'.