Missing documents using import document functionality of Sync Gateway

[matteosist]

Hi, trying to figure out why some documents are missing from mobile devices i found a strange behaviour of import documents functionality.

Sync Gateway version

2.7 EE

Operating system

Ubuntu 18.04

Config file

{
  "adminInterface": "127.0.0.1:4985",
  "interface": "0.0.0.0:4984",
  "logging": {
    "console": {
      "log_level": "debug",
      "log_keys": ["HTTP", "Import"]
    }
  },
  "databases": {
    "test-bucket": {
      "server": "http://127.0.0.1:8091",
      "bucket": "test-bucket",
      "username": "Administrator",
      "password": "password",
      "import_docs": true,
      "enable_shared_bucket_access": true,
      "num_index_replicas": 0,
      "users": {
        "GUEST": {"disabled": false, "admin_channels": ["*"] }
      }
    }
  }
}

Expected behavior

All documents created from Java SDK should be imported from Sync Gateway.

Actual behavior

The issue is that the "import documents function" ignore all docs that contains a field with a 0 bigdecimal with scale set to 7 or higher.
Let me to explain better.

I have:

• 1 Couchbase Server 6.5 EE
• 1 Sync Gateway 2.7 EE
• 1 Java application using Couchbase Java Client 3.0.1. The Sync Gateway has "import docs" enabled.

Steps to reproduce

1. From the Java application create a new document with a field with 0 big decimal (scale set to 7) value.
2. Save the document in Couchbase Server with upsert function.
3. The Sync Gateway doesn't see the document and you cannot find it from mobile app.

I created a use case to reproduce the issue.

Here the Java code that create 10 docs. Each with a single field key with a 0 bigdecimal value.
The scale of the bigdecimal is incremental from 0 to 9.
All docs with scale from 0 to 6 will be imported from Sync Gateway. From 7 scale and higher the docs seems to be ignored.

    String connectionString = "localhost";
    String username = "Administrator";
    String password = "password";
    String bucketName = "test-bucket";

    Cluster cluster = Cluster.connect(connectionString, username, password);
    Bucket bucket = cluster.bucket(bucketName);
    Collection collection = bucket.defaultCollection();

    for (int scale = 0; scale < IntStream.rangeClosed(0, 10).toArray().length; scale++) {
        String id = "document-scale-" + scale;
        collection.upsert(
                id,
                JsonObject.create().put("key", new BigDecimal(0).setScale(scale))
        );
        System.out.println("Inserted doc " + id);
    }

I enabled the Import logs on Syng Gateway and as you can see the import process stop at document with scale 6.
Logs: https://gist.github.com/matteosist/bb9317a0ea342f48c31268be7bfb9707

In addition, if i try to get the document with id document-scale-7 using a HTTP GET http://localhost:4984/test-bucket/document-scale-7 the document will be imported.
Only if i force the retrieve of the document from Sync Gateway.

Logs: https://gist.github.com/matteosist/321566ff1dde9cd7d60b28579371d9bc

The issue seems be related with scientific notation.
In fact if i try to get a document with a 0 bigdecimal with scale from 0 to 6 i get the entire value.

{"_id":"document-scale-6","_rev":"1-1d682143e5c79803018d631be19b266c","key":0.000000}

But if i get a document with 0 bigdecimal with scale 7 or higher i get the result with scientific notation.

{"_id":"document-scale-7","_rev":"1-40682619625f9a144591157c85e8b761","key":0E-7}

Has anyone already encountered this problem? Can you help me solve this issue?

[bbrks]

I wrote a quick integration test with Couchbase Server to import raw docs of the form:
{"key":0.0000000000} for decimal place counts between 0 and 20. All get imported fine without an on-demand import nessesary.

Need to investigate what the raw JSON doc looks like in Couchbase Server when inserted through the Java SDK as BigDecimal(0).setScale(10) to try and repro in an integration test.

[bbrks]

It seems we have a problem doing auto import of docs with numbers in scientific notation.

e.g: 0E-3 will fail.

I suspect BigDecimal is encoding everything up to 7 decimal places using normal decimal notation (e.g: 0.000, and anything with more decimal places as scientific notation.

[bbrks]

I'm seeing that a document that looking like {"key":0E-3} is coming to Sync Gateway as a binary document over DCP, so it's not being imported.

Will have to investigate further to determine where the problem lies.

[matteosist]

Thanks a lot for the investigation.

[bbrks]

It looks like this is a bug in the way Couchbase Server is interpreting JSON numbers when the value is zero.

For non-zero numbers (e.g. 1E5), the doc is correctly interpreted as JSON. For values of zero (e.g. 0E5) the doc is coming to Sync Gateway via DCP as a binary document.

I'll follow this up with the server team here: https://issues.couchbase.com/browse/MB-38034

Thanks for filing the issue!

[bbrks]

As a workaround until Couchbase Server is patched to handle this case: Are you able to prevent your SDK-written documents from being created using scientific/exponent notation?

[matteosist]

I don't know. Maybe i can intercept all bigdecimal numbers with value 0 and call on it the method .stripTrailingZeros() but i need to try if this workaround can fit our code.
Is there a scheduled date for resolution?

[bbrks]

If you're running into this issue for a customer with a support entitlement, I'd highly encourage them contacting support about this. It will hurry the patch along. We're pretty close to a code-complete deadline for 6.5.1

[bbrks]

I'm not sure if you've been following the Server issue I filed.
This has been scheduled to go into Couchbase Server 6.5.1

https://issues.couchbase.com/browse/MB-38034

I'll go ahead and close this issue, as it's not dependent on anything from Sync Gateway, just awaiting a Server upgrade.