wq/wq.app

batch submission support

Closed this issue · 8 comments

sheppard commented

wq/outbox.js includes a batchService option that makes it possible to sync the entire outbox in one AJAX request. However, there is no corresponding functionality in wq.db to process a batch request. This shouldn't be terribly hard to implement, there just needs to be a view that process the batch data into calls to each of the regular views and returns the summary result.

This could be a standalone view, or (preferably) a built-in ModelRouter view like ConfigView and MultipleListView)

Edit: Let's integrate wq.app with django-batch-requests instead (see comments)

One benefit of this functionality would be to support submitting inter-related models as separate forms without breaking foreign key constraints. (Submitting one or more child records together with a parent record in a single form is already supported via the patterns APIs, though the documentation for creating a custom pattern is limited (see wq/wq#21))

tomaszn commented

I'd like to help in implementing this feature. Can we discuss this further?

tomaszn commented

I'll try a generic solution: https://github.com/tanwanirahul/django-batch-requests (forks have some interesting commits too).

tomaszn commented

I prepared a version for Django 1.11 which seems to work. Entry for requirements.txt:
git+https://github.com/tomaszn/django-batch-requests@28cd9fafffe351a42c1e54a95e563cf4b0ca0ede

Project's settings.py need to include:

BATCH_REQUESTS = {
    "HEADERS_TO_INCLUDE": ["CSRF_COOKIE"],
}
sheppard commented

Thanks for looking into this (and sorry for the delay). Did you have to make any changes to wq.db and/or wq.app to get this to work?

tomaszn commented

I use wq==1.1.0 from upstream, only wq.app is modified (see my fork / master).

sheppard commented

Ok, good to know. I think it makes the most sense to just update wq.app to work directly with django-batch-requests, and add some configuration notes in the documentation. (I tried out GitHub's new transfer issue feature to move this to the wq.app repository.)

Are you aware of (or willing to maintain) an official PyPI package that incorporates the most up-to-date fork of django-batch-requests?

tomaszn commented

Unfortunately I can't identify the best fork of django-batch-requests.
Mine seems to work with the following testing matrix:

    - { python: "2.7", env: DJANGO=1.11 }
    - { python: "3.4", env: DJANGO=2.0 }
    - { python: "nightly", env: DJANGO=2.2 }

and I just released it as "v1.0.0" to stop referring to a proper version it by a commit.

I don't plan to implement new features, but fixes are welcome.

sheppard commented

Thanks, I implemented this support in @wq/outbox. The implementation is broadly based on your wq.app fork, but rebuilt to work with the new redux-offline code. I also switched to using JSON encoding instead of x-www-form-urlencoded for the nested requests.

The other main difference is that the new implementation (still) uses /^outbox-(\d+)$/ to determine priority between records. The effect should be mostly the same as your wq:ForeignKey-based algorithm, except that the first batch will include any/all records that don't need to wait - either because they do not include a foreign key, or because the foreign key is for an object that already has a server-assigned id.

(Since the new outbox is a class, in theory you could subclass it and override the _peek method to define a different ordering.)

It seems to be working well, but let me know if you find any issues. I just released wq.app 1.2 alpha.