dawi/jsont

Go JSON Tokenizer that also emits insignificant tokens (like whitespaces)

jsont

A JSON tokenizer that also emits insignificant tokens like whitespaces.

{ "Hello" : "World" }

Will be split into the following tokens:

|{| |"Hello"| |:| |"World"| |}|

Usage

tokenizer := jsont.NewTokenizer(strings.NewReader(jsonInput))

for tokenizer.Next() {
	token := tokenizer.Token()
	fmt.Println(token.Type)
	fmt.Println(token.Value)
}                                                                                                                                   

if tokenizer.Error() != nil {
	fmt.Println(tokenizer.Error())
}

Disclaimer

• Only consider using this tokenizer if you are intersted in whitespaces for some reason.
  Otherwise you should have a look at Go 1.5 which has a json tokenizer built in.
  • golang/go#6050
  • https://github.com/golang/go/commit/0cf48b4d919e5137ec03f1bf230fbd8720873151
• Also notice that this tokenizer does not verify if the incoming data is valid JSON or not.