Change pre-batching from 1M to 1000
maximusunc opened this issue · 2 comments
Right now, we're sending basically all returned results along to the next hop (1M ids). This makes a few KPs quite angry. We should test out changing the pre-batch size to 1000.
Should be a very simple change found here
After a little digging, this is most likely a bigger change than expected. It looks like right now what we call the "mini-batches" can be limited to 1,000 curies, but we do a second round of batching that groups multiple waiting requests into one larger request. This second batching happens in the trapi_throttle and currently doesn't expose a way to limit the number of curies in each final request, only the number of requests that it will batch together.
BTE is now complaining that we're sending too large of requests. Molepro sends a 413 error.