JSON parsing should not fail on missing fields

Problem description

When parsing a JSON message, the pipeline rule fails if a field does not exist, but it defined in select_jsonpath.

For example, this will fail:

let json_result = parse_json(to_string($message.message));
let json_fields = select_jsonpath(json_result,
            { source: "$.source",
              some_field: "$.i_dont_exist"
            });

The error message will be:

For rule 'f5_test_rule': In call to function 'select_jsonpath' at 8:22 an exception was thrown: null

I suggest ignoring missing fields by default and introducing a new optional boolean parameter to toggle this behaviour.

Environment

Graylog Version: 2.3.1
Pipeline Processor plugin version: 2.3.1

@lennartkoopmann I'm unable to reproduce this issue with Graylog 2.3.1 and 2.4.0-SNAPSHOT.

The test case I've been using is directly derived from your example:

diff --git a/plugin/src/test/resources/org/graylog/plugins/pipelineprocessor/functions/jsonpath.txt b/plugin/src/test/resources/org/graylog/plugins/pipelineprocessor/functions/jsonpath.txt
index e036970..75c4b7c 100644
--- a/plugin/src/test/resources/org/graylog/plugins/pipelineprocessor/functions/jsonpath.txt
+++ b/plugin/src/test/resources/org/graylog/plugins/pipelineprocessor/functions/jsonpath.txt
@@ -7,4 +7,7 @@ then
               author_last: "$['store']['book'][-1:]['author']"
             });
   set_fields(new_fields);
+
+  // Don't fail on missing field
+  let missing_field = select_jsonpath(x, { some_field: "$.i_dont_exist" });
 end
\ No newline at end of file

It looks like json_result was null when you tried running the JSON path extraction on it.

The parse_json() function may return null if the input wasn't valid JSON:

graylog-plugin-pipeline-processor/plugin/src/main/java/org/graylog/plugins/pipelineprocessor/functions/json/JsonParse.java

Lines 51 to 56 in 9fe21fa

    
           try { 
        
               return objectMapper.readTree(value); 
        
           } catch (IOException e) { 
        
               log.warn("Unable to parse json", e); 
        
           } 
        
           return null;

This all being said, I think the parse_json() function shouldn't return null and the json_path() function should be able to handle null as an input.

	try {
	return objectMapper.readTree(value);
	} catch (IOException e) {
	log.warn("Unable to parse json", e);
	}
	return null;