Make an in-browser version
Closed this issue · 34 comments
After seeing how much faster NeDB is than TaffyDB, I'm considering porting NeDB to the browser, with persistence support using LocalStorage. Please comment if you think this would be useful.
localstorage isn't worthwhile, but indexeddb is. see https://github.com/rvagg/node-levelup, https://github.com/rvagg/node-leveldown and https://github.com/maxogden/level.js
Clearly localStorage is not ideal, considering its limitations, but at least it is supported by more browsers than IndexedDB. What I could do is to use IndexedDB when it is available and fall back to localStorage when it's not.
I think it would be worthwhile to create a browser version. Local client side storage is nice, but sometimes it's very useful to have a better in memory data structure for use as a data cache, etc (especially if you might exceed the HTML 5 storage limits, and can't rely on indexeddb support).
I just looked at your benchmarks against TaffyDB. Wow. NeDB would be an awesome alternative to TaffyDB. I know for some of the uses I was looking for, the find function alone would make NeDB the winner. If it was available in the browser.
The browser side world would benefit from having a faster datastore. With the advent of faster JS interpreters, more and more data processing happens on the client. In terms of persistence, I am with the others on focusing on localstorage. It may also be interesting to look at web workers to perform tasks such as async-durability (write data to localstorage in the background, rather than sync with the inserts).
Thanks for the comments, that confirms the need to port NeDB to browsers so I'll do it. The web workers idea is interesting, I'll do the porting first and then see how to do background persistence. I am experimenting on a similar idea for the current NeDB (I call it pipelining), but this is still experimental (queued saves can actually slow current operations).
I would highly recommend looking at the links I provided earlier. Local storage is not worth supporting, whereas indexeddb and websql are
Sent from my iPhone
On Jun 20, 2013, at 2:13 PM, Louis Chatriot notifications@github.com wrote:
Thanks for the comments, that confirms the need to port NeDB to browsers so I'll do it. The web workers idea is interesting, I'll do the porting first and then see how to do background persistence. I am experimenting on a similar idea for the current NeDB (I call it pipelining), but this is still experimental (queued saves can actually slow current operations).
—
Reply to this email directly or view it on GitHub.
@maxogden Yes as I said I'll probably use local storage only as fallback. But even as just an in-memory store, I think NeDB will be useful, when I see the number of people who use the much slower TaffyDB.
I was looking for a good way to store data locally in a (phonegap) type mobile app and then have it sync with MongoDB. There may be other solutions but if this was a way to do that it would be awesome too.
The other angle where I'd like to see this in the client is to make easier fakes for web browser based unit tests
@utunga I made this utility that you can use to transfer all your data from a NeDB database to a MongoDB one (for people who used NeDB for rapid prototyping and outgrew it).
For now you need to give it a NeDB datafile but I will adapt it when I port NeDB to the browser.
I would definitely say this feature would be very useful.
@louischatriot I'd very much like to have this library available for IE 9+, which would require LocalStorage support.
@maxogden PouchDB only uses IndexedDB, perhaps that's more for you. There is value for some of us in supporting older browsers. WebSQL is dead.
I think it would be cool to make the browser storage engine as plugin or adding events emitters to the database so you could have it sync to Dropbox or Google Drive as well.
Example
var tasks = new NeDB({localStorage: 'tasks'});
tasks.on('insert', function(task) {
dropbox.writeFile('tasks.db', tasks.localStorage, function(err) {
if (!err) { console.log('synced');
});
});This would require creation by specifying the id and access to the localStorage value. The reference for the dropbox sdk is here https://tech.dropbox.com/2012/08/some-love-for-javascript-applications-2/
@louischatriot I am excited to see Nedb come along, especially in seeing it's similarities to MongoDB and would love to see it working in the Browser.
I agree with @maxogden etc, that you should avoid localstorage, as it is too limiting for any serious use. Either WebSQL or IndexedDB are available for all browsers and there are front end wrappers that convert the IndexedDB API into WebSQL as required. This will enable you to just write a DB plugin for IndexedDB that will then run on all Browsers. See: http://nparashuram.com/IndexedDBShim/
I am mystified at why you guys are trying to persuade Louis to not implement a feature that he seems to want to do, and that I absolutely require, when doing so has zero effect on anyone who prefers IndexedDB as that will also be supported.
@naturalethic If your localstorage meets your requirements then that's fine, but I would have thought it less work for Louis to support a single API, that being IndexedDB, which with the shim I mentioned will work on all browsers. In that case why even bother supporting the limited capabilities of localStorage. To quote: http://stackoverflow.com/questions/2989284/what-is-the-max-size-of-localstorage-values
Web storage can be viewed simplistically as an improvement on cookies,
providing much greater storage capacity (2.5 MB per origin in Google Chrome;
5 MB per origin in Mozilla Firefox, and Opera; 10 MB per storage area in Internet Explorer)
and better programmatic interfaces.
And you may want to see the results on this page: http://arty.name/localstorage.html
Further localStorage only supports strings and has no index capabilities. And the user can delete the contents of localStorage, which could be a disaster waiting to happen.
God god yes! I'd love to see this.
Does nedb having a Browser based Console to see/edit/config the data .
Sometimes we need to watch the data in Memory when developing or testing.
The best embed database in JAVA--h2, having a Browser based Console application, which is very convenient.
There is no such thing yet. What you could do however is use Mongo Edit (another project of mine, a web based Mongo GUI) and modify it to make it work with nedb. That should not be too much work since both APIs are very close. I unfortunately don't have enough time to do it.
Here is the link : https://github.com/louischatriot/mongo-edit
Will the in browser port be available as a different project, or will the existing project be modified to work in both environments?
My goal is to use the same project since most of the code will be common.
I finally found the time to do it, the browser version is ready! Check the readme out: https://github.com/louischatriot/nedb#browser-version
It's still young, for now it doesn't support persistence (working on it) and I haven't yet tested it on all browsers (working on it) but at least you can test this young version :)
Dependencies 'underscore' and 'binary-search-tree' are missing from the dependency check.
I don't understand, where are they missing from the dependency check?
Le 14 juil. 2013 à 20:52, Joshua Kifer notifications@github.com a écrit :
Dependencies 'underscore' and 'binary-search-tree' are missing from the dependency check.
—
Reply to this email directly or view it on GitHub.
I had to install them manually with npm is all. The build didn't install
them. I didn't examine the build.js code.
On Sun, Jul 14, 2013 at 1:39 PM, Louis Chatriot notifications@github.comwrote:
I don't understand, where are they missing from the dependency check?
Le 14 juil. 2013 à 20:52, Joshua Kifer notifications@github.com a écrit
:Dependencies 'underscore' and 'binary-search-tree' are missing from the
dependency check.—
Reply to this email directly or view it on GitHub.—
Reply to this email directly or view it on GitHubhttps://github.com//issues/23#issuecomment-20943354
.
The browser version is standalone, you don't need to install anything with npm, just include nedb.js in your html files. I updated the readme with a code example, tell me if that's clear
Le 14 juil. 2013 à 23:06, Joshua Kifer notifications@github.com a écrit :
I had to install them manually with npm is all. The build didn't install
them. I didn't examine the build.js code.On Sun, Jul 14, 2013 at 1:39 PM, Louis Chatriot notifications@github.comwrote:
I don't understand, where are they missing from the dependency check?
Le 14 juil. 2013 à 20:52, Joshua Kifer notifications@github.com a écrit
:Dependencies 'underscore' and 'binary-search-tree' are missing from the
dependency check.—
Reply to this email directly or view it on GitHub.—
Reply to this email directly or view it on GitHubhttps://github.com//issues/23#issuecomment-20943354
.—
Reply to this email directly or view it on GitHub.
Yes that's clear. I had just chosen to build it myself, this console output should make clear what I meant:
joshua@zeta nedb § node browser-version/build.js
Installing build dependencies
Removing contents of the src directory
Copying source files
Copying browser specific files to replace their server-specific counterparts
Browserifying the code
Error during build
[Error: module "underscore" not found from "/Users/joshua/Workspace/nedb/browser-version/src/lib/datastore.js"]
joshua@zeta nedb § ls node_modules/
async browserify fs-extra uglify-js
joshua@zeta nedb § npm install underscore
npm http GET https://registry.npmjs.org/underscore
npm http 200 https://registry.npmjs.org/underscore
npm WARN package.json uglify-js@2.2.5 'repositories' (plural) Not supported.
npm WARN package.json Please pick one as the 'repository' field
npm WARN package.json callsite@1.0.0 No repository field.
underscore@1.4.4 node_modules/underscore
joshua@zeta nedb § node browser-version/build.js
Installing build dependencies
Removing contents of the src directory
Copying source files
Copying browser specific files to replace their server-specific counterparts
Browserifying the code
Error during build
[Error: module "binary-search-tree" not found from "/Users/joshua/Workspace/nedb/browser-version/src/lib/indexes.js"]
joshua@zeta nedb § npm install binary-search-tree
npm http GET https://registry.npmjs.org/binary-search-tree/0.2.3
npm http 200 https://registry.npmjs.org/binary-search-tree/0.2.3
npm http GET https://registry.npmjs.org/binary-search-tree/-/binary-search-tree-0.2.3.tgz
npm http 200 https://registry.npmjs.org/binary-search-tree/-/binary-search-tree-0.2.3.tgz
npm WARN package.json uglify-js@2.2.5 'repositories' (plural) Not supported.
npm WARN package.json Please pick one as the 'repository' field
npm WARN package.json callsite@1.0.0 No repository field.
binary-search-tree@0.2.3 node_modules/binary-search-tree
joshua@zeta nedb § node browser-version/build.js
Installing build dependencies
Removing contents of the src directory
Copying source files
Copying browser specific files to replace their server-specific counterparts
Browserifying the code
Creating the minified version
Build finished with success
The issue was that the source dependencies such as underscore and binary-search-tree are not checked in to the Git repo so you needed to reinstall them. This is now automatic.
Is there any way to integrate NeDB with Phonegab, so we can use it for mobile Devices(android, ios etc).
That would definitely be valuable and interesting, yes. I have no experience with Phonegap, could you open a new issue with the requirements ?
As of v1.4, NeDB automatically selects the best method to persist data on the browser (IndexDB then WebSQL then localStorage), so the limit is removed. WARNING: it's not back compatible with v1.3, but that should not be a major issue as user apps can resync entirely.
NeDB for browsers and mobile platforms like Andoid, ios through Phonegap to persist data locally will be utmost useful utility database. I am sure it will be the most popular database after Mongodb. Though, I tried including version 1.8, as mentioned in package.json, of nedb.js and/or nedb.min.js in html file using script tag from folder nedb/ browsers-version/out folder, but it gives error. Further guidance for how to include nedb.js in html file for saving data on local disk will be very much appreciated...
<title>Document</title> <script src="js/material.js"></script> <style> body { margin-left: 15px; } input { width: 400px; margin-bottom: 5px; } label { width: 90px; /* text-align: right; */ float: left; } form { margin-left: 5px; } </style>Personal Details:
<form name="testForm"
action="#"
method="POST">
<!-- onSubmit="var hasSaved = saveContact(); return hasSaved;"> -->
<label for="fname">First Name:</label>
<input type="text" placeholder="First name" id="fname"><br>
<label for="mname">Middle Name:</label>
<input type="text" placeholder="Middle name" id="mname"><br>
<label for="lname">Last Name:</label>
<input type="text" placeholder="Last name" id="lname"><br>
<label for="building">Building:</label>
<input type="text" placeholder="Building" id="building"><br>
<label for="street">Street:</label>
<input type="text" placeholder="Street" id="street"><br>
<label for="locality">Locality:</label>
<input type="text" placeholder="Locality" id="locality"><br>
<label for="landmark">Landmark:</label>
<input type="text" placeholder="Landmark" id="landmark"><br>
<label for="city">City:</label>
<input type="text" placeholder="City" id="city"><br>
<label for="zip">Zip/Pin Code:</label>
<input type="text" placeholder="ZIP/PIN Code" id="zip"><br>
<label for="state">State:</label>
<input type="text" placeholder="State" id="state"><br>
<label for="country">Country:</label>
<input type="text" placeholder="Country" id="country"><br>
</form>
</div>
<!-- buttons examples -->
<div id="buttons">
<!-- Accent-colored raised button with ripple -->
<button class="mdl-button
mdl-js-button
mdl-button--raised
mdl-js-ripple-effect
mdl-button--accent"
type="submit"
onclick="saveContact();">
Save Contact
</button>
<hr>
<h4>Result:</h4>
<p id="result"></p>
<!-- Colored FAB button -->
<!-- <button class="mdl-button mdl-js-button mdl-button--fab mdl-button--colored">
<i class="material-icons">add</i>
</button> -->
</div>
@Dasanudas there is actually a db inspired by nedb but built on in many ways and updated to Es6 and has the capability to do all the things you would like. you could write an html parser to overhead this db as well. It's maintained by myself and another and has others who are watching and waiting for new ideas to build on. https://github.com/tedb-org/teDB