Future of backbone.offline

[alekseykulikov]

I'm going to use this library in my new project and want to announce new features. It will biggest step:

• WebWorkers to sync data with server in background and does not lock main UI
• The polyfill which works with IndexedDB, WebSQL, localStorage for storing data effectively in modern browsers. Best idea is launchair or store.js
• Ability to sync data using chunks similar to Evernote EDAM
• Remove strict relation from Backbone.Collection during the sync process.

The main goal is create flexible library for syncing browser's local data with server and make the process of creating offline-applications easier. If someone is solving same problems, you're welcome ;)

Welcome to comments to share your ideas and code examples. All works are going in edge branch.

[g00fy-]

for polyfill I suggest lawnchair, or own adapters, but you would have to use callbacks, as IndexedDB and WebSQL are async.
Any idea how to solve related-models changes?

assuming we are offline:

person = new Person({id:1}); person.fetch();    
friend   = new Person({id:2}); friend.fetch();
// assuming that friends relations are keeped in fkeys manner ( [10,11,12] )
person.set('friends',person.get('friends').push(friend.id));
// assuming that friends are collection
person.get('friends').add(friend);
person.save();
// now this should be true 
true == _.contains(friend.get('friends'),person.id);
// or this
true == friend.get('freinds').contains(person);

this means, that even you actualy do 1 save on person, the other model should also keep track of the changes (2 updates to local db)

another problematic situation is when the user was offline, and someone else changed relations:

// we are offline now:
person = new Person({id:1}); person.fetch();
friend = new Person({id:2}); friend.fetch();
// person & friend are not friends here
// now someone changes friends like above
// now we are online now and we synced
// since only 'person' object has changed, we will receive only person 
// from the server in our incremental sync
// but still we should keep track of changes (on the server) and relations
// so this should also be true 
true == friend.get('freinds').contains(person);

[alekseykulikov]

Thank you @g00fy- for lawnchair, looks very close that I want (like redis for browser) but api is a bit strange. List of adapters impressed.

Any idea how to solve related-models changes?

I want to generate id alway on client using guid. So we don't support relational DB, but it will posible to write adapter later. I use Mongo and it works well with auto-assigned ids.

[g00fy-]

guid on the clinet may seem a good idea, but there is little chance of guid being in conflict with some other object,
how do you propose to handle this problem?

[alekseykulikov]

hm... I think local browser storage can not contain so much data, so guid is unique forever http://stackoverflow.com/questions/7981190/how-to-generate-absolutely-unique-guids#comment13808005_7981228
Also I'm going to consider option with MongoDB ids: #11

[backspace]

Some of those new features would be of great use to me. Any predictions on when you’ll have something to share?

I’m thinking of hacking in a sync/merge that knows whether an item was updated on the server after last sync so changes on the server after the client has last synced are not trampled by offline changes on the client. Conflicts could be manually fixed and maybe even automatically merged when possible.

Seems hard.

[alekseykulikov]

Hi @backspace, I've not started yet (a lot of other work), but I hope it's around 1-2 months.
When I think about synchronisation process, the best example that I have is Evernote EDAM and it will be great to port this algorithm to javascript and have very simple api to use.

[g00fy-]

Hi there,

You may want to take a look on our approach to this problem,

https://github.com/pxlpl/Offline

There is still much work to be done, but it seems that this solution works even for RDBMS (where there are PK&FK constrains).

We made that app with few goals in mind:

1. it's transparent - plug&play (write your app just like you would without our lib)
2. it's customizable (we used patterns like synchronizer - low level implementation, strategy (per model) - the strategy of the synchronization, store - abstraction on top of persistance layer)
3. non-blocking - default store is IndexedDB or MemoryStore, but it takes 50 lines to write localStorage or few more to do websql.
4. smart - we keep track of relations, we supply the way to manipulate the data on the client - this way, you can work with your app offline - not just view it!
5. It is failure tolerant - no problem when your connection drops during the synchronization.
6. customizable way of handling conflicts - (concurrent editing the same object possible) - default strategy is: server is always right, but keep user changes.

Go check it out & tell us what you think!

[alekseykulikov]

Hi @g00fy-, I like your approach very much. All 6 goals look reasonable. I will take a deeper look to your code and let you know my detailed opinion on next week. Exactly on next week I'm going to start implement backbone.offline to my new project http://gingkoapp.com.
For now, It would be helpful to see more documentation for strategy.js and synchronizer.js and can you give a little example how to use the lib in general.
Anyway, thanks for sharing, it means a lot. Later and good luck.

[g00fy-]

Strategy - is the object that controlls the flow of your synchronization. Our current implementation uses this order:

(push & pull - makes more sense than pull & push. because you will get a consistent state)

1. Wait untill other required objects haven't finished creating objects (to be able to save FK)
2. Create new objects
3. Update objects (including some updates for just created objects that are required because of recursive relations)
4. Fetch remote changes
5. Apply changes
6. Resolve conflicts

It is actually more than that, but it is easy to customize (overwrite method like readyToCreate). We also give status object, that helps keep track of the progress (like notification status) & furthermore, you can keep track of the progress of every kind of object you are synchronizing.

Synchronizer - this is low level implementation of synchronization. methods like createRecord, updateRecords are noop - they use your standard way of synchronization - no need to add anything on the backened.
If you would like, you could overwrite those methods to use some other transport layer or mechanizm.

For instance we need to overwrite fetchRemoteChanges, in order to limit number of request sent to the server - we need to set up some kind of a queue, that would fetch multiple kinds of objects in a single request.

since almost everything is async (promise based), this gives you a lot of flexibility but becomes sometimes hard to debug.

See example how to configure it here https://github.com/pxlpl/Offline/blob/master/test/models.js

after this (you also need to do Store.connect()) you can start writing your own code just like you would without the app.

If you do model.save(), the default behavior is to save everything locally, and using Offline.sync() to synchronize changes, but you can as well change that by customizing Offline. If you are using some other synchronization mechanizm, or transport layer, overwrite Offline.remote - note that your transport must be consistent with Backbone.sync - must return Deferred object that resolves with appropriate values.

I'll try to write some examples, but it would be great if you could give us some hints

BTW, we also were considering implementing EDAM solution, but we didn't since he ad to be able to resolve the order of the objects being updated/created etc (RDBMS).

[alekseykulikov]

@g00fy- Your implementation looks quite smart, Examples like https://github.com/pxlpl/backbone.offline/blob/master/test/models.js would be helpful to understand library's logic. I still need more time to make deeper look in the source.

I have some questions for you, which helps to understood library's logic:

• How often do you run synchronisation process?
• How do you work with realtime events?
• What do you do when server has newer version of object, but user changed old version?
• How do you work with relationships when api does not return all keys, for example nested resources?
• Do you use operational logs or journals?
• How do you clone db on first sync?
• How do you support the whole DB in actual state?

P.S. I would like if you were changed your libraries's name different with backbone.offline, because it is not a fork of my library, and add unnecessary mess. The world has a lot of cool names ;)

[pxlpl]

I will try to put answer to your questions in the toturial we are planing to release.
Most of the features/questions are already in the core, some (like nesting) may need overwriting/customizing few methods or adding new concepts to solutions (like RelationType).

We will have to think of some cool name 👍

[valerianrossigneux]

what is the status of this ?

[alekseykulikov]

In free-time I'm working on it :) Today I've just finished Indexed - indexeddb/localStorage adapter. Next step is finish https://github.com/ask11/storage and start to work on offline component.