golang/go

encoding/json: JSON number serialization differs from ES6/V8

cyberphone opened this issue · 12 comments

cyberphone commented

The following program (written by a complete go n00b)

package main

import "fmt"
import "encoding/json"

func main() {
    mapD := map[string]float64{"apple": 5e-6, "lettuce": 0.0}
    mapB, _ := json.Marshal(mapD)
    fmt.Println(string(mapB))
}

returns

{"apple":5e-06,"lettuce":0}

it should rather return

{"apple":0.000005,"lettuce":0}

Rationale for this request:
chakra-core/ChakraCore#149

gopherbot commented

CL https://golang.org/cl/19092 mentions this issue.

rsc commented

I spent a little while looking at this. CL 19092 suggests just changing the format to %f, but that doesn't match at least Chrome's console.log either. It matches for 5e-6, but not for larger or smaller numbers.

By manual binary search it appears that Chrome's algorithm is:

abs := math.Abs(x)
if abs == 0 || 1e-6 <= x && x < 1e21 {
   use %f
} else {
   use %e
}

This is kind of a peculiar thing to do, but if all the Javascript engines agree then I don't see significant harm to using the same algorithm for Go. However, a much stronger argument would be if the ECMA-262 spec actually documents and mandates this apparent rule. Does it?

cyberphone commented

ECMA-262 does indeed mandates this formatting but unfortunately offers two solutions for the last (unprecise) digit. V8 selected the more advanced ("correct") version.

rsc commented

Thanks @cyberphone. I have no problem with producing the correct last digit, since Go already does. However, on closer examination it looks like maybe JS doesn't get later digits correct: 999999999999999868928 is an exact float64 value. Go prints it correctly as 999999999999999868928 (%f) or as 9.999999999999999e+20 (%g/%e), while Chrome prints it as 999999999999999900000. Does ECMA-262 mandate this too? Can you point to a page in the PDF that I can read? Thanks.

cyberphone commented

@rsc If I understand this correctly 999999999999999868928 is an exact float64 value but according to the authorities on this matter (excludes me...) it is still wrong showing it because the precision is only 17 digits.

7.1.12.1 in http://www.ecma-international.org/ecma-262/6.0/ECMA-262.pdf is the reference.

rsc commented
cyberphone commented

Yes :-) A possibility could be integrating the V8 algorithm code in go "as is" and make it available through a method in "strconv" like FormatFloatES6. This is essentially how I did it in my JSON Java library.

RalphCorderoy commented

The issue has been raised on json/encoding. ECMA-262's ToString http://www.ecma-international.org/ecma-262/6.0/index.html#sec-tostring-applied-to-the-number-type describes a behaviour (two to hedge their bets) of Javascript. But it's http://www.ecma-international.org/publications/standards/Ecma-404.htm that covers JSON and that makes clear on page ii that it seeks to get away from a particular programming language's representation and move towards a human representation. Code that depends on 1e42 being represented a particular way in JSON seems broken to me. A library that deliberately discards bits so the conversion can't be reversed, more so.

cyberphone commented

@RalphCorderoy The sole purpose of this proposal is creating a foundation for "normalized" JSON objects that can safely traverse through different systems even if they carry a digital signature. ES6 also specifies ordered properties although it is not a part of JSON.

For digitally signed JSON/JavaScript, the following samples illustrate the two principal alternatives (JCS is just an example) that developers have to select from. They should be fully comparable with respect to security since they use the same algorithms (not easy to see since the IETF scheme dresses everything in Base64URL).

JSON Cleartext Signature (JCS): https://cyberphone.github.io/openkeystore/resources/docs/jcs.html#ECMAScript_Compatibility_Mode

var signedObject = {
      // The data
      statement: "Hello signed world!",
      otherProperties: [2000, true],
      // The signature
      signature: {
          algorithm: "ES256",
          publicKey: {
              type: "EC",
              curve: "P-256",
              x: "vlYxD4dtFJOp1_8_QUcieWCW-4KrLMmFL2rpkY1bQDs",
              y: "fxEF70yJenP3SPHM9hv-EnvhG6nXr3_S-fDqoj-F6yM"
          },
          value: "2H__TkcV28QpGWPkyVbR1CW0I8L4xA...CTyYfmAippJXqdzgNAonnFPVCSI5A6novMQ"
      }
};

JSON Web Signature (JWS): https://tools.ietf.org/rfc/rfc7515.txt

var signedObject = {
      payload: "eyJpc3MiOiJqb2UiLA0KICJleHAiOjEz...eGFtcGxlLmNvbS9pc19yb290Ijp0cnVlfQ",
      protected: "eyJhbGciOiJFUzI1NiJ9",
      signature: "DtEhU3ljbEg8L38VWAfUAqOyKA...gfTjDxw5djxLa8IS lSApmWQxfKTUJqPP3-Kg6NU1Q"
};

JCS was created for preserving the human representation of JSON.

joeshaw commented

Related: #6384

gopherbot commented

CL https://golang.org/cl/30371 mentions this issue.