trivago/gollum

format.MetadataCopy: unexpected change in metadata if msg.data.payload get modified

mahdi-hdi opened this issue · 4 comments

Issue description

By using format.MetadataCopy the value of payload is copied to metadata['your_key']. After that any changes to the payload would be appeared in metadata['your_key'] as well.

Steps to Reproduce (for bugs)

  1. Run gollum with the following configuration.
  2. Enter aaaa:bb
  3. Expect out must be key#payload which is aaaa:bb#bb
  4. However the output is bbaa:bb#bb

Your Environment

Used versions

  • gollum version: v0.5.3
  • go version: go1.10.3 linux/amd64

Config:

default-input:
    Type: producer.Console
    Streams:
        - console

default-output:
    Type: consumer.Console
    Streams:
        - console
    Modulators:
        ## Suppose that payload is 'aaaa:bb'
        ## Copy payload to metadata['key']
        ## So metadata['key'] is now 'aaaa:bb'  and payload is 'aaaa:bb' as well
        - format.MetadataCopy:
            ApplyTo: key

       ## Split the payload and pick index 1
       ## So metadata['key'] is now 'aaaa:bb' and payload is 'bb'
        - format.SplitPick:
            Index: 1
            Delimiter: ':'

       ## In order to print value of metadata prepend it to message payload
       ## So metadata['key'] must be 'aaaa:bb' and payload be 'key#payload' 
       ## which is 'aaaa:bb#bb'
        - format.MetadataCopy:
            Key: key
            Separator: '#'
            Mode: prepend
       
       ## BUT the final payload is 'bbaa:bb#bb'
        - format.Envelope:
            Postfix: "\n"

Context and Possible Solution

package main

import "fmt"

func main() {
	payload := []byte("aaaabb")
	metadata := map[string][]byte{}
	
	//
	// format.MetadataCopy:
	//     ApplyTo: key
	//
	metadata["key"] = payload
	
	///////////////////////////////////////////////////
	// A suggested approach could be:                //
	//                                               //
	// metadata["key"] = make([]byte, len(payload))  //
	// copy(metadata["key"], payload)                //
	//                                               //
	// instead of:                                   //
	//                                               //
	// metadata["key"] = payload                     //
	//                                               //
	///////////////////////////////////////////////////
	
	
	fmt.Printf("payload: %s\n", payload)
	fmt.Printf("metadata: %s\n", metadata)
	
	//
	// Modifiying payload with other formaters
	//     
	payload = payload[:2]
	copy(payload, []byte("xx"))
	
	fmt.Printf("payload: %s\n", payload)
	fmt.Printf("metadata: %s\n", metadata)

}

Accordingly, a possible solution could be here metadata.go:22:

# Change from:
meta[key] = value

# To:
meta[key] = make([]byte, len(value))
copy(meta[key], value)

The cause is correct but I would prefer to not fix it in the core/metdata but in the plugins.
There are situations where you already operate on a copy and fixing it in core/metadata would force you to create another one. As of this I would rather do the cloning in the formatter

getSourceData := core.GetAppliedContentGetFunction(format.key)

As you already have identified the issue and have tests - could you do a PR an this (+unittest)?

Yeah of course, I'm gonna fix it as you suggested in the formatter.

Thank you!