Improve webwork generation; don't abort when a single problem fails
oscarlevin opened this issue · 13 comments
Alex suggests there might be an -a flag for core pretext that controls whether the webwork representations file is still built even if a problem fails. Setting this right and giving a big old warning about broken webwork would be a good idea.
This looks like the line in CLI that is causing -a to be used:
So that's what the abort_early means!
So now the question is: should we just make this false always, or add an option? I'm leaning toward just never aborting early for CLI use. @StevenClontz , thoughts?
Hm. I have no opinions here one way or the other, and apologize for waiting two weeks to say so.
@Alex-Jordan , I tried to change abort_early=False, but this does not seem to help create the webwork-representation file. The CLI uses a try/except around its call to the core command. The exception that is thrown has this message: "Opening and ending tag mismatch: A line 5 and webwork, line 10, column 11 (, line 10)"
With abort_early=True, it also reports a warning, but gives more detail about where the issue is. Ideas?
Are you able to go into the PROSE dropin? I'm in there. Just in case we can talk whatever this is through easily.
I'm a bit lost because I've lost track of what project is trying to be built here, where the building of the representations file fails. We thought that abort_early=True was causing someone's single broken WW problem file to make the whole thing stop. What project was that, or which project are you currently testing this with?
I think that's right. I just mocked up some incorrect webwork problems amid some I knew worked to test out switching it to false. So I don't know if this is actually still needed or desired, but didn't want to lose track of it.
What was the nature of how you broke the problems?
I tried it with this broken problem from the OPL: openwebwork/webwork-open-problem-library/OpenProblemLibrary/UMass-Amherst/Abstract-Algebra/PS-Relations/Relations6.pg, which is what originally got me back on this issue. Also tried a PreTeXt authored problem missing a semicolon, and a source problem pointing to a non-existent OPL problem (i.e., the path was wrong). Same complaint about mismatched open/close tag in each case.
I put that problem into the sample chapter (replacing the OPL problem that is in there as a demonstration) and when I run pretext/pretext/pretext -v -c webwork -p [it's pub file] [sample chapter source file] (note no -a flag for abort early) then it runs as expected with the message below about how it skips over this one.
...
PTX:INFO : building representations of server-based WeBWorK problem: webwork-32
PTX:INFO : sending webwork-32 to server to save in /Users/alex.jordan/pretext/examples/webwork/sample-chapter/generated/webwork/webwork-representations.xml: origin is 'server'
PTX:ERROR: WeBWorK problem Library/UMass-Amherst/Abstract-Algebra/PS-Relations/Relations6.pg with seed 8435 does not have a statement tag
Maybe it uses something other than BEGIN_TEXT or BEGIN_PGML to print the statement in its PG code
Use -a to halt with returned content
PTX:INFO : building representations of server-based WeBWorK problem: webwork-36
PTX:INFO : sending webwork-36 to server to save in /Users/alex.jordan/pretext/examples/webwork/sample-chapter/generated/webwork/webwork-representations.xml: origin is 'server'
...
The reps file is created, and for that problem it has:
<webwork-reps version="2" ww_major_version="2" ww_minor_version="17" xml:id="extracted-webwork-32" ww-id="webwork-32">
<static seed="8435" source="Library/UMass-Amherst/Abstract-Algebra/PS-Relations/Relations6.pg" failure="statement">
<statement>
<p>PTX:ERROR: WeBWorK problem Library/UMass-Amherst/Abstract-Algebra/PS-Relations/Relations6.pg with seed 8435 does not have a statement tag
Maybe it uses something other than BEGIN_TEXT or BEGIN_PGML to print the statement in its PG code
Use -a to halt with returned content
</p>
</statement>
</static>
</webwork-reps>
which is valid XML. The particular message here "Maybe it uses something other than..." is only triggered if the server returns valid XML, but there is no statement tag in the tree.
I'm not able to see what happens by using the CLI, since I'm just using the CLI distribution version that has abort_early=True. So of course when I try, the reps file is not built.
Can you confirm what happens when you try with abort_early=False where this is the only issue with a problem? Next I will look at the next kind, with broken perl (missing semicolon).
The missing semicolon in a problem does fail in a bigger way than I expected, with that lxml.etree.XMLSyntaxError: Opening and ending tag mismatch: A line 5 and webwork, line 10, column 11 message. So I need to look into core and catch something better with try/except.
PreTeXtBook/pretext#2132 should fix all this with broken perl in a problem or using a nonexistent file.
I am not sure that PTX core has an issue with the case where the only bad problem is like using Library/UMass-Amherst/Abstract-Algebra/PS-Relations/Relations6.pg. That produces valid XML but is missing a statement tag. And it seems like pretext core handles that one OK right now. Let me know if you confirm the CLI with abortEarly=False still does not produce good messaging in that case.
Fantastic. I guess I must have had one of the other types of errors too when I tested setting abort_early to False. I see that it works as expected for that problem. PR to the CLI incoming, and on next release, we will have the better desired behavior