format.MetadataCopy: unexpected change in metadata if msg.data.payload get modified
mahdi-hdi opened this issue · 4 comments
Issue description
By using format.MetadataCopy
the value of payload is copied to metadata['your_key']
. After that any changes to the payload would be appeared in metadata['your_key']
as well.
Steps to Reproduce (for bugs)
- Run gollum with the following configuration.
- Enter
aaaa:bb
- Expect out must be
key#payload
which isaaaa:bb#bb
- However the output is
bbaa:bb#bb
Your Environment
Used versions
- gollum version:
v0.5.3
- go version:
go1.10.3 linux/amd64
Config:
default-input:
Type: producer.Console
Streams:
- console
default-output:
Type: consumer.Console
Streams:
- console
Modulators:
## Suppose that payload is 'aaaa:bb'
## Copy payload to metadata['key']
## So metadata['key'] is now 'aaaa:bb' and payload is 'aaaa:bb' as well
- format.MetadataCopy:
ApplyTo: key
## Split the payload and pick index 1
## So metadata['key'] is now 'aaaa:bb' and payload is 'bb'
- format.SplitPick:
Index: 1
Delimiter: ':'
## In order to print value of metadata prepend it to message payload
## So metadata['key'] must be 'aaaa:bb' and payload be 'key#payload'
## which is 'aaaa:bb#bb'
- format.MetadataCopy:
Key: key
Separator: '#'
Mode: prepend
## BUT the final payload is 'bbaa:bb#bb'
- format.Envelope:
Postfix: "\n"
Context and Possible Solution
package main
import "fmt"
func main() {
payload := []byte("aaaabb")
metadata := map[string][]byte{}
//
// format.MetadataCopy:
// ApplyTo: key
//
metadata["key"] = payload
///////////////////////////////////////////////////
// A suggested approach could be: //
// //
// metadata["key"] = make([]byte, len(payload)) //
// copy(metadata["key"], payload) //
// //
// instead of: //
// //
// metadata["key"] = payload //
// //
///////////////////////////////////////////////////
fmt.Printf("payload: %s\n", payload)
fmt.Printf("metadata: %s\n", metadata)
//
// Modifiying payload with other formaters
//
payload = payload[:2]
copy(payload, []byte("xx"))
fmt.Printf("payload: %s\n", payload)
fmt.Printf("metadata: %s\n", metadata)
}
Accordingly, a possible solution could be here metadata.go:22:
# Change from:
meta[key] = value
# To:
meta[key] = make([]byte, len(value))
copy(meta[key], value)
The cause is correct but I would prefer to not fix it in the core/metdata but in the plugins.
There are situations where you already operate on a copy and fixing it in core/metadata would force you to create another one. As of this I would rather do the cloning in the formatter
Line 115 in 3d7deb2
As you already have identified the issue and have tests - could you do a PR an this (+unittest)?
Yeah of course, I'm gonna fix it as you suggested in the formatter.
Thank you!