`Failed to get string: Utf8Error` when parsing `/private/var/db/diagnostics/Special/`

[fukusuket]

Thank you for maintaining the tool :)
Since the following WARN often appears, I will report an issue. I would appreciate it if you could confirm this WARN.

Describe the issue
A warning Failed to get string: Utf8Error appears when parsing files under /private/var/db/diagnostics/Special/ files.

Step to Reproduce

1. git clone https://github.com/mandiant/macos-UnifiedLogs.git (c857f6f)
2. cd examples/unifiedlog_parser
3. cargo build --release
4. ../target/release/unifiedlog_parser -l true -o out.csv

Expected behavior
No warnings.

Actual behavior
The following warning is printed.

Parsing: /private/var/db/diagnostics/Special/000000000000094d.tracev3
15:17:21 [WARN] [macos-unifiedlogs] Failed to get string: Utf8Error { valid_up_to: 0, error_len: Some(1) }
15:17:21 [WARN] [macos-unifiedlogs] Failed to get string: Utf8Error { valid_up_to: 0, error_len: Some(1) }
Parsing: /private/var/db/diagnostics/Special/000000000000095d.tracev3
15:17:21 [WARN] [macos-unifiedlogs] Failed to get string: Utf8Error { valid_up_to: 1, error_len: Some(1) }
15:17:21 [WARN] [macos-unifiedlogs] Failed to get string: Utf8Error { valid_up_to: 0, error_len: Some(1) }
15:17:21 [WARN] [macos-unifiedlogs] Failed to get string: Utf8Error { valid_up_to: 0, error_len: Some(1) }
15:17:21 [WARN] [macos-unifiedlogs] Failed to get string: Utf8Error { valid_up_to: 1, error_len: Some(1) }
15:17:21 [WARN] [macos-unifiedlogs] Failed to get string: Utf8Error { valid_up_to: 2, error_len: Some(1) }
15:17:21 [WARN] [macos-unifiedlogs] Failed to get string: Utf8Error { valid_up_to: 0, error_len: Some(1) }
15:17:21 [WARN] [macos-unifiedlogs] Failed to get string: Utf8Error { valid_up_to: 51, error_len: None }
15:17:21 [WARN] [macos-unifiedlogs] Failed to get string: Utf8Error { valid_up_to: 0, error_len: Some(1) }
Parsing: /private/var/db/diagnostics/Special/0000000000000986.tracev3
15:17:21 [WARN] [macos-unifiedlogs] Failed to get string: Utf8Error { valid_up_to: 0, error_len: Some(1) }
15:17:21 [WARN] [macos-unifiedlogs] Failed to get string: Utf8Error { valid_up_to: 0, error_len: None }
...

Environment

• OS: macOS Ventura version 13.4
• OS Language & Region : Japan
• Hard: MacBook Air(M1, 2020) , Memory 8GB, Core 8, Chip: Apple M1

Additional context

• As far as I have investigated, it seems that parsing fails in the following process.
  • https://github.com/mandiant/macos-UnifiedLogs/blob/main/src/chunks/statedump.rs#L106-L107
• This WARN appeared when statedump_results.unknown_data_type is 1 or 2
  • According to the document below, UTF-8 formatted string is set, but data other than UTF-8 formatted string may be set...?
  • https://github.com/libyal/dtformats/blob/main/documentation/Apple%20Unified%20Logging%20and%20Activity%20Tracing%20formats.asciidoc#212-statedump-chunk
• Except for /private/var/db/diagnostics/Special/ files, this WARN does not appeared.

[puffyCid]

yes this is currently a known issue/limitation as mentioned in RUNNING.md:
https://github.com/mandiant/macos-UnifiedLogs/blob/main/RUNNING.md#possible-issues-when-running

[WARN] Failed to get string: Utf8Error { valid_up_to: 0, error_len: Some(1) }

The parser failed to extract string metadata from a log message. This is commonly happens with log files in the Special directory. The parser currently attempts to extract strings associated with metadata on the log entry. Sometimes the metadata cannot be represented as a string

There have been some recent updates to the log format at dtformats

I can take another look and see if its possible to improve the parsing

[fukusuket]

I'm sorry, I overlooked the description of the known issue ... Thank you so much for checking it again :)

[puffyCid]

I just pushed some updates to the statedump parser based on the updated documentation at dtformats

The parser should not generate string warnings when parsing statedumps.
If you have chance to run and test that would be great.

I ran on the test files and my local system and the warnings are gone and it extracts the data successfully

[fukusuket]

I also confirmed that there is no WARN in my environment :) Thank you so much for the quick fix!!