Message duplicated if taken from nested field

[alexbakker]

Thanks for hl, it's a useful tool!

I'm dealing with a rather unfortunate log format where the message field is not located at the top level of the JSON object, but in a nested field. Luckily, hl supports reading the message from a nested field, but I found that it duplicates the message in the output while doing so. This does not happen if the message is in a top level field.

Case 1 (good)

Contents of good.json:

{"ts": "1711402897", "level": "info", "message": "test message", "field1": "value1"}

Command: hl good.json

Output:

Mar 25 21:41:37.000 |INF| test message field1=value1

This is what I'd expect to happen. The message is not shown as a field but is displayed right after the log level.

Case 2 (unexpected)

Contents of bad.json:

{"ts": "1711402897", "level": "info", "message": {"text": "test message", "nested1": "value2"}, "field1": "value1"}

Contents of config.yaml:

fields:
  predefined:
    message:
      names: [message.text]

Command: HL_CONFIG=config.yaml hl bad.json

Output:

Mar 25 21:41:37.000 |INF| test message message={ text=test message nested1=value2 } field1=value1

In this case, the message is correctly displayed after the log level, but I would not have expected it to also still be displayed in the fields.

I'm not sure if this is intended behavior, and if it isn't, whether it's even desirable to change this to avoid the duplication.

[pamburus]

This is not a desired behavior.
I am going to fix it, but the fix is not trivial.

In the meantime, I can suggest you a workaround.
Add the following configuration option:

# Settings for fields processing.
fields:
  # List of exact field names to hide.
  hide:
    - message.text

This should help.

[alexbakker]

Thanks, that does help. I tried that previously, but I had --raw-fields enabled as well. It kind of makes sense that hiding a nested field doesn't work in that case.

[pamburus]

May I ask you why you were using --raw-fields?

[alexbakker]

Just experimenting a bit with cosmetics.

[pamburus]

Yes, I see.

This is a bit off topic, but it would be great if you could share what kind of cosmetic issues you've experienced.
For example, it could be

1. You didn't like the way string values with spaces were displayed.
2. You didn't like the way string values with special characters were displayed.
3. You didn't like the way string values with newline characters were displayed.
4. All of the above, or something else?

More recently, the way string values are displayed was changed in #147 (actually, just the quotes were removed), and later an option to enable or disable quotes was added in #165.
But those quotes were always added without properly escaping the string contents, and it can get confusing if the string itself contains quotes or special characters.
So I am considering improving the way string values are displayed.
However, it is a difficult decision to find the right balance between readability and unambiguity.
I don't want to see a lot of quotes and escaping, otherwise it will look more like the original JSON file. There will probably be several different views, and some smart default that quotes when needed, adds proper escaping when needed, and tries to avoid escaping when possible, probably by automatically choosing the most appropriate outer quotes.

It would be very helpful to get any feedback, suggestions, or just thoughts on this.

[alexbakker]

The primary thing that felt a little bit awkward at first was the way hl displays nested objects, but I've actually taken a liking to it over the last couple of days, having used it a bit more. Initially I thought I wanted the keys and values of nested objects to have quotes around them, so I turned on --raw-fields, but then quickly discovered the downsides like lots of escaping of characters.

I would have to think about it more, but I think hl already strikes a pretty good balance here. It's tough to display a nested object on a single line in an easily readable way. Like you said, you don't want to see a plain JSON object.

On a more general note, I'm a big fan of the default theme of tint:

Since the keys have a fairly dim color, it's easier for me to discover the values I'm looking for., because I can usually already intuitively recognize them by their 'signature' (UUID, integer, etc). If I can't easily find a particular value, the keys are still there to help. This can already be achieved through hl's theming support, which is great! I wouldn't look at tint for an example of displaying nested objects, though.

[pamburus]

Thanks for the feedback. It helps a lot.

It looks like I guessed that you wanted to add quotes to string values. Currently, you can add single quotes with the 'formatting.add-quotes = true' configuration option. And those quotes will not add any escaping, which is both good and bad, but you can just try it and see if it feels better. I admit that strings with spaces look really ugly without quotes right now. I am researching a better approach that will try to find an optimal balance between readability and ambiguity.

Regarding nested objects, it also bothers me and when I have logs with a log of deeply nested objects it becomes unreadable almost like a json. I discovered that the way the logfmt format handles nested objects is much more readable. It simply flattens them by combining nested keys with . delimiters. And that improves readability a lot. So this is another thing on my TODO list with rather high priority, I would like to add an option to choose between current view and flat view.

Regarding tint's theme, I recently discovered it and found it useful as well. I agree that because it is mostly dimmed, it makes it easy to find errors and warnings. Inspired by this theme, I added a similar theme in the latest version called neutral. Although I have kept some values highlighted according to their types, I am not sure about this yet. You can try it and make a copy and tune it if you don't like something. And to remove special characters around the level you need to add the following options to the configuration file

# Formatting settings.
formatting:
  punctuation:
    level-left-separator: ''
    level-right-separator: ''

Unfortunately, these options are not currently part of a theme, so they should be configured separately. This is not convenient, and I plan to move many of these settings into the theme configuration file.