Test infrastructure: Add a custom editor for DAR tests
gregg-miskelly opened this issue · 0 comments
Problem
It can be painful to create tests in DAR. This proposal seeks to improve two problems with authoring/maintaining DAR tests:
- Make it easier to turn the actual responses from a debug adapter into C# code for a DAR-based test
- When a DAR test fails with a log entry like the following, make it easier to understand why the 'actual' and 'expected' don't match.
Example log entry:
DebugAdapterRunner.DARException : Test command execution failed...
Failure = Expected response not found before timeout.
Expected =
1. Not Found: {"success":true,"body":{"stackFrames":[{"id":1000,"name":"[External Code]"},{"id":1001,"name":"^This\\ will\\ fail\\ to\\ match$"},{"id":1002,"name":"Debuggee\\.Main\\(string\\[]\\ args\\)"}]}}
Actual Responses =
1. {"success":true,"message":null,"request_seq":5,"command":"stackTrace","body":{"stackFrames":[{"id":1000,"name":"[External Code]","line":0,"column":0,"presentationHint":"subtle"},{"id":1001,"name":"[Exception] Exceptions4Debuggee.exe!AsyncTest.C(string messageFormat) Line 51","source":{"name":"debuggee.cs","path":"C:\\dd\\Concord\\src\\vsdbg\\VsDbg-UITests\\Debuggees\\Exceptions4Debuggee\\debuggee.cs","sourceReference":0,"checksums":[{"algorithm":"SHA256","checksum":"b2ef5210e5727249981edef3380f6f4d34a1cb776683ec762cc21096f6d81a4a"}]},"line":51,"column":9,"endLine":51,"endColumn":87,"instructionPointerReference":"0x00007FFE0AA619D4","moduleId":1001},{"id":1002,"name":"[Exception] Exceptions4Debuggee.exe!AsyncTest.B(string messageFormat) Line 43","source":{"name":"debuggee.cs","path":"C:\\dd\\Concord\\src\\vsdbg\\VsDbg-UITests\\Debuggees\\Exceptions4Debuggee\\debuggee.cs","sourceReference":0,"checksums":[{"algorithm":"SHA256","checksum":"b2ef5210e5727249981edef3380f6f4d34a1cb776683ec762cc21096f6d81a4a"}]},"line":43,"column":9,"endLine":43,"endColumn":32,"instructionPointerReference":"0x00007FFE0AA615CE","moduleId":1001},{"id":1003,"name":"[Exception] Exceptions4Debuggee.exe!AsyncTest.A(string messageFormat) Line 36","source":{"name":"debuggee.cs","path":"C:\\dd\\Concord\\src\\vsdbg\\VsDbg-UITests\\Debuggees\\Exceptions4Debuggee\\debuggee.cs","sourceReference":0,"checksums":[{"algorithm":"SHA256","checksum":"b2ef5210e5727249981edef3380f6f4d34a1cb776683ec762cc21096f6d81a4a"}]},"line":36,"column":9,"endLine":36,"endColumn":32,"instructionPointerReference":"0x00007FFE0AA6122E","moduleId":1001},{"id":1004,"name":"[External Code]","line":0,"column":0,"presentationHint":"subtle"},{"id":1005,"name":"Exceptions4Debuggee.exe!AsyncTest.Run(System.Func<string> messageFormatFactory) Line 25","source":{"name":"debuggee.cs","path":"C:\\dd\\Concord\\src\\vsdbg\\VsDbg-UITests\\Debuggees\\Exceptions4Debuggee\\debuggee.cs","sourceReference":0,"checksums":[{"algorithm":"SHA256","checksum":"b2ef5210e5727249981edef3380f6f4d34a1cb776683ec762cc21096f6d81a4a"}]},"line":25,"column":13,"endLine":25,"endColumn":45,"instructionPointerReference":"0x00007FFE0AA60DF4","moduleId":1001},{"id":1006,"name":"[External Code]","line":0,"column":0,"presentationHint":"subtle"}],"totalFrames":7},"running":false,"refs":null,"seq":24,"type":"response"}
Command = {"args":{"threadId":26140,"startFrame":0,"levels":20},"Name":"stackTrace","ExpectedResponses":[{"Response":{"success":true,"body":{"stackFrames":[{"id":1000,"name":"^\\[This\\ will\\ fail\\ to\\ match]$","presentationHint":"subtle"},{"id":1001,"name":"^\\[Exception]\\ Exceptions4Debuggee\\.exe!AsyncTest\\.C\\(string\\ messageFormat\\)\\ Line\\ 51$","source":{"name":"^debuggee\\.cs$","path":".+\\\\Exceptions4Debuggee\\\\debuggee\\.cs$"},"line":51,"column":9,"endLine":51,"endColumn":87},{"id":1002,"name":"^\\[Exception]\\ Exceptions4Debuggee\\.exe!AsyncTest\\.B\\(string\\ messageFormat\\)\\ Line\\ 43$","source":{"name":"^debuggee\\.cs$","path":".+\\\\Exceptions4Debuggee\\\\debuggee\\.cs$"},"line":43},{"id":1003,"name":"^\\[Exception]\\ Exceptions4Debuggee\\.exe!AsyncTest\\.A\\(string\\ messageFormat\\)\\ Line\\ 36$","source":{"name":"^debuggee\\.cs$","path":".+\\\\Exceptions4Debuggee\\\\debuggee\\.cs$"},"line":36},{"id":1004,"name":"^\\[External\\ Code]$","presentationHint":"subtle"},{"id":1005,"name":"^Exceptions4Debuggee\\.exe!AsyncTest\\.Run\\(System\\.Func<string>\\ messageFormatFactory\\)\\ Line\\ 25$","source":{"name":"^debuggee\\.cs$","path":".+\\\\Exceptions4Debuggee\\\\debuggee\\.cs$"},"line":25},{"id":1006,"name":"^\\[External\\ Code]$","presentationHint":"subtle"}]}},"Match":null,"IgnoreOrder":false}]}
Proposal
We should have a WPF application that makes it easier to author and debug DAR based tests. A rough sketch is below:
| Paste From Exception Text | Copy As C# | Update Expected from Actual |
Status: <NO MATCH FOUND | Match Found | No expected text | syntax error>
Expected:
+-----------------------------------------------------------+
| |
| <-- Editable rich text block with expected response --> |
| |
+-----------------------------------------------------------+
[X] Ignore order
Actual:
+-----------------------------------------------------------+
| |
| <-- Editable rich text block with actual response --> |
| |
+-----------------------------------------------------------+
Where:
- 'Paste From Exception Text', 'Copy As C#', 'Update Expected From Actual' are buttons
- 'Status' is a non-editable textbox
- 'Ignore order' is a checkbox
The primary function of the app would be to match up the 'Expected' and 'Actual' text and report if they matched. It should do so
by refactoring or borrowing code from DebugAdapterRunner.DebugAdapterCommand.Run.
This program should also expose three other bits of functionality:
1: The 'Paste From Exception Text' button would look at the clipboard for text of the form
Expected = ...\nActual Responses =\n1 ...[\n2. ...] and would replace the current "Expected"
and "Actual" tetxt block with that content. This should include reformatting the JSON to add indentations so that it is readable.
2: Text highlighting for regular expressions. Every string is DAR is treated as a regular expression. Every string in the 'Expected' block should be highlighted in green, if at least one match was found, or red, if no match was found. If the cursor is in one of these regular expressions, that has a match, then it should get a yellow highlight in the 'Actual' block.
3: The 'Copy As C#' would take the content of the 'Expected' text block and reformat it as C# using anonymous types. For example, if the current Expected text is:
{
"success": true,
"body": {
"variables": [
{
"name": "param1\\ \\[int\\]",
"value": "^1$"
}
]
}
}This should produce:
new
{
success = true,
body = new
{
variables = new dynamic[]
{
new
{
name = @"param1\ \[int\]",
value = @"^1$"
}
}
}
}4: Implement 'Update Expected From Actual' -- take the json from the 'actual' box, and paste it into the 'excepted' box, turning any string into a regex that would match just that value using:
static string EscapeFullString(string value)
{
return "^" + Regex.Escape(value) + "$";
}