Bug: Unexpected CasparCG crash without writing any error logs

[leftearsons]

Observed Behavior

Starting from somewhere around July 2023, CasparCG (v2.3.x-2022-04-25 onwards, with current master versions included as well) will crash unexpectedly from time to time without writing any error log messages at all! The occurrence frequency cannot be predicted and could span from one day up to one month, but we still have some servers that could operate for longer without crashing.
The only trace that is visible are some information and error logs in the Windows event viewer.
I have noticed that the issue is irrelevant of whether flash or html graphics are in use, as well as not necessarily related to the video source (would take place both with SDI and HLS video sources).
Based on our statistics though, it seems to be more frequent when loading streaming video content (HLS specifically).

I am adding the related logs from the event viewer for your reference:

ERROR LOG:

Faulting application name: casparcg.exe, version: 2.4.0.0, time stamp: 0x661ffb91
Faulting module name: ntdll.dll, version: 10.0.19041.2130, time stamp: 0xb5ced1c6
Exception code: 0xc0000374
Fault offset: 0x00000000000ff6a9
Faulting process id: 0x1a98
Faulting application start time: 0x01daadd1fa3f3c3b
Faulting application path: C:\Users\Admin\Documents\Video Service\graphics\casparcg.exe
Faulting module path: C:\WINDOWS\SYSTEM32\ntdll.dll
Report Id: 0dd7e7bd-29c1-4cc5-8982-397b89622534
Faulting package full name: 
Faulting package-relative application ID: 

INFORMATION LOG:

Fault bucket 1919053832705669865, type 4
Event Name: APPCRASH
Response: Not available
Cab Id: 0

Problem signature:
P1: casparcg.exe
P2: 2.4.0.0
P3: 661ffb91
P4: StackHash_30e5
P5: 10.0.19041.2130
P6: b5ced1c6
P7: c0000374
P8: PCH_7B_FROM_ntdll+0x000000000009DC14
P9: 
P10: 

Attached files:
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER1311.tmp.dmp
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER16CB.tmp.WERInternalMetadata.xml
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER16DB.tmp.xml
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER16D9.tmp.csv
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER16FA.tmp.txt

These files may be available here:
\\?\C:\ProgramData\Microsoft\Windows\WER\ReportArchive\AppCrash_casparcg.exe_7f05fda3b1597de05cfb3afd34279a3bcbf91e_48550a11_9a8d754b-aa01-4ecf-a645-6d8108b167be

Analysis symbol: 
Rechecking for solution: 0
Report Id: 0dd7e7bd-29c1-4cc5-8982-397b89622534
Report Status: 268435456
Hashed bucket: 692ea9158538f81dcaa1d94b1345fae9
Cab Guid: 0

INFORMATION LOG: 

Fault bucket 2148026746892973266, type 5
Event Name: RADAR_PRE_LEAK_64
Response: Not available
Cab Id: 0

Problem signature:
P1: casparcg.exe
P2: 2.4.0.0
P3: 10.0.19045.2.0.0
P4: 
P5: 
P6: 
P7: 
P8: 
P9: 
P10: 

Attached files:
\\?\C:\Users\Admin\AppData\Local\Temp\RDR1669.tmp\empty.txt
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER1679.tmp.WERInternalMetadata.xml
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER167A.tmp.xml
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER1689.tmp.csv
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER16B9.tmp.txt

These files may be available here:


Analysis symbol: 
Rechecking for solution: 0
Report Id: 92a5a7b4-a5b0-495b-bc80-e6ffd6f34651
Report Status: 268435456
Hashed bucket: 7ed4d0524476372ebdcf52f21097f4d2
Cab Guid: 0

Could it be some memory leak issue as mentioned in the above event log Event Name: RADAR_PRE_LEAK_64?

Expected behaviour

The expected behavior would be to always have CasparCG server running without crashing, or at least to always log an error cause when need to exit forcefully.

Steps to reproduce

1. Start CasparCG
2. Start running various commands with graphics and video playback
3. CasparCG would crash unexpectedly without writing any cause in the logs

Environment

• Server version from 2.3.x onwards (current version is 2.4.0 1e25c7a Stable)
• Operating system: Windows 10

Note that the issue was not present before July 2023