Nullflush -z mode doesn't work
Closed this issue · 11 comments
Nullflush -z mode still doesn't seem to work reliably, at least not when run from APy.
And, this breaks: apertium-fra-cat.
Any update on this issue?
It seems to be an issue with flushing. It only prints the previous text, and not the one just sent.
How to reproduce it.
# tty 1
mkfifo /tmp/fifotest
sleep 1000 > /tmp/fifotest & # so it keeps the pipe open
../src/lsx-proc -z data/out.bin < /tmp/fifotest
# tty 2
echo -e "^take<vblex><pres>$ ^it<prn><obj>$ ^out<adv>$ [][\n]\0" > /tmp/fifotest # nothing gets printed in tty1
echo -e "^take<vblex><pres>$ ^me<prn><obj>$ ^out<adv>$ [][\n]\0" > /tmp/fifotest # outpout of take it out gets printed on tty 1
echo -e "^take<vblex><pres>$ ^him<prn><obj>$ ^out<adv>$ [][\n]\0" > /tmp/fifotest # output of take me out gets printed on tty 1
When that happens (after first text, no ouptut exists), code is waiting in
https://github.com/apertium/lttoolbox/blob/master/lttoolbox/fst_processor.cc#L867
This has been fixed as part of apertium/lttoolbox@75a62df, but now we need new versions of lttoolbox to be published, so apertium-separable is aware of the fix.
@TinoDidriksen can you do that?
@TinoDidriksen Have you pushed new lttoolbox?
Not yet, another fix is pending to be released together with this fix.
Still a problem on apertium-oqaa with fra→cat. Here's the pipeline up until lsx-proc:
apertium@oqaa:~$ echo -e 'ai[][\n]\0ai[][\n]\0' | lt-proc -w -z '/home/apertium/tarballs-build/share/apertium/apertium-fra-cat/fra-cat.automorf.bin' | cg-proc -z -w '/home/apertium/tarballs-build/share/apertium/apertium-fra-cat/fra-cat.rlx.bin' | apertium-tagger -g -z '/home/apertium/tarballs-build/share/apertium/apertium-fra-cat/fra-cat.prob' | apertium-pretransfer -z
^avoir<vbhaver><pri><p1><sg>$[][
] ^avoir<vbhaver><pri><p1><sg>$[][
]
then lsx-proc removes the words:
apertium@oqaa:~$ echo -e 'ai[][\n]\0ai[][\n]\0' | lt-proc -w -z '/home/apertium/tarballs-build/share/apertium/apertium-fra-cat/fra-cat.automorf.bin' | cg-proc -z -w '/home/apertium/tarballs-build/share/apertium/apertium-fra-cat/fra-cat.rlx.bin' | apertium-tagger -g -z '/home/apertium/tarballs-build/share/apertium/apertium-fra-cat/fra-cat.prob' | apertium-pretransfer -z | lsx-proc -z '/home/apertium/tarballs-build/share/apertium/apertium-fra-cat/fra-cat.autosep.bin'
[][
] [][
]
A test with just that step on fra-cat.autosep.bin.zip:
$ echo -e '^avoir<vbhaver><pri><p1><sg>$[][\n]\0^avoir<vbhaver><pri><p1><sg>$[][\n]\0' | lsx-proc -z fra-cat.autosep.bin
[][
][][
]
where if you show NUL as ^@ it looks like:
[][
]^@[][
]^@
^@
apertium-separable:
Installed: 0.3.2+g2337465824-1xenial1
Actually, the whole lsx method is a huge mess. I've been trying to look at where the problem is, and it seems that the original processor was never meant to support null-flush.
Fran added partial support for it (in apertium/lttoolbox@82c4537) but the actual code was always looking for feof(input), and never for \0. The fix I did last year worked for a few cases, but was causing the problem described above.
It seems that this fixes it: apertium/lttoolbox@85fbf5c
but it's an ugly hack, and we should probably rewrite the whole method (and, honestly, the whole file, if we want it to be maintainable).
$ echo -e '^avoir<vbhaver><pri><p1><sg>$[][\n]\0^avoir<vbhaver><pri><p1><sg>$[][\n]\0' \
| lsx-proc -z fra-cat.autosep.bin
^avoir<vbhaver><pri><p1><sg>$[][
]^avoir<vbhaver><pri><p1><sg>$[][
]
Seems to work =D
Could you add that to tests? (Especially in case someone ever cleans up the code …)