a b64/utf8 encoding issue

[jeremybmerrill]

Hi, AquilaDB seems really really neat and a terrific tool. Thanks for building it!

I worked through the Google USE / Python example and am now trying to adapt it to my usecase, but finding some persistent encoding issues on document retrieval. For instance, one of documents contains a right single quotation mark U+2019. This is read in correctly and written correctly to the CouchDB document store (I checked via the Couchdb interface). However, a db.getNearest query's response contains \x19 there instead, which isn't a valid character in JSON and causes a mess.

The issue is between btoa/atob (which I think assume UTF-16 strings?) and the response out of pouchdb (UTF-8) in this file.

Here's a minimal example, though you'll have to adjust the document ID and the slice indices to match your document containing the problematic character, of course.

const atob = require('atob');
const btoa = require('btoa');
var PouchDB = require('pouchdb');
var db = new PouchDB('http://localhost:5984/default_docsdb')
q = db.allDocs({include_docs: true, keys: ['3c80fca415c221bf3702e055c055c21f']}).then((a) => { return a})
let resp = null
q.then((a) => resp = a)

Here's a (portion of) a document from PouchDB that needs to get transmitted to the client, e.g. via the Python library.

> JSON.stringify(resp.rows).slice(289, 296)
'today’s'

The problem is that btoa mis-encodes it:

> atob(btoa(JSON.stringify(resp.rows).slice(289, 296)))
'today\u0019s'

One solution: js-base64.

> Base64.decode(Base64.encode(JSON.stringify(resp.rows).slice(289, 296)));
'today’s'

One solution that's apparently not a good one:

> decodeURIComponent(atob(btoa(encodeURIComponent(JSON.stringify(resp.rows).slice(289, 296)))))
'today’s'

[freakeinstein]

Hi @jeremybmerrill,
Thanks for reporting the issue with detailed description and suggested fixes. I have applied your suggestion to code and made a pull request, as you can see above.

Happy to hear your plans to use it in your usecase and we are open to help you along. Feel free to find and report bugs. Pull requests are also welcome. We follow Merge Quickly policy as mentioned by Pieter Hintjens. Thanks.

Updated: Build is failing need to investigate.

[freakeinstein]

Fixed. Closing.