Documentation on submitting multiple POST requests
prasenjithaty opened this issue · 16 comments
Hi,
I'm evaluating the library and so far it seems to be pretty fast. However, I'm trying to figure out how to submit multiple POST requests to the same url obviously with different entity body.
ParallecHeader headers = new ParallecHeader();
headers.addPair(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_XML);
pc.prepareHttpPost("/metadata")
.setHttpEntityBody(xmlMapper.writeValueAsString(metadataRequest))
.setHttpHeaders(headers)
.setResponseContext(responseContext)
.setTargetHostsFromString("example.com")
.execute(new ParallecResponseHandler() {
public void onCompleted(ResponseOnSingleTask res,
Map<String, Object> responseContext) {
String responseContent = res
.getResponseContent();
LOGGER.info("!!Response: {}, TargetHost: ", responseContent, res.getHost());
responseContext.put("responseContent", responseContent);
}
});
pc.releaseExternalResources();
Request body (metadataRequest) will be like:
<MetaData>
<fileURL>/example/main_file_6.jpg</fileURL>
</MetaData>
<MetaData>
<fileURL>/example/secondary_file_1.jpg</fileURL>
</MetaData>
Thanks,
Prasenjit.
@prasenjithaty can you see if this helps? http://www.parallec.io/docs/submit-task/#apis-on-variable-replacement-for-heterogeneous-requests
Hi @jeffpeiyt,
The example you shared is for heterogeneous requests to different hosts. How about doing the same with a single host? I'm not really able to figure out a way of doing this. Also javadoc on ParallelClient#setReplace* methods would be pretty useful for users.
Thanks.
but put the $VAR_NAME inside of the string in setHttpEntityBody
if you scroll down a bit you can see:
Different requests to the same target host
Here is the example of hitting 2 different APIs to the same target host. $ZIP is the variable being replaced. setReplaceVarMapToSingleTargetSingleVar(String variable, List replaceList, String uniformTargetHost) is the API to use. There are more complex replacements APIs available in javadoc. Complete sample code is here.
http://www.parallec.io/userdata/sample_weather_48824.txt
http://www.parallec.io/userdata/sample_weather_95037.txt
pc.prepareHttpGet("/userdata/sample_weather_$ZIP.txt")
.setReplaceVarMapToSingleTargetSingleVar("ZIP",
Arrays.asList("95037","48824"), "www.parallec.io")
.setResponseContext(responseContext)
.execute(new ParallecResponseHandler() {...}...
but put the $VAR_NAME inside of the string in setHttpEntityBody
^^ was the right pointer.
Thanks. Got it working now. Now, how can I collect all the responses?
Great! Can you check examples from https://github.com/eBay/parallec-samples/tree/master/sample-apps/src/main/java/io/parallec/sample/app/http ?
I already checked those. But, I want to set the key in the map dynamically. Maybe fileURL from my first post in the request body
@prasenjithaty is your question on aggregating/collecting responses or setting the variables in different requests?
My question is to collect responses with the key being a field from my request body.
I thought I figured out with the below code, but it breaks during unmarshalling
.execute(new ParallecResponseHandler() {
public void onCompleted(ResponseOnSingleTask res,
Map<String, Object> responseContext) {
String responseContent = res
.getResponseContent();
TaskRequest taskRequest = res.getRequest();
LOGGER.info("post data = {}", taskRequest.getPostData());
LOGGER.info("post body = {}", taskRequest.getRequestContent());
MetadataRequest metadataRequest = JAXB.unmarshal(taskRequest.getRequestContent(), MetadataRequest.class);
LOGGER.info("!!Response: {}, TargetHost: ", responseContent, res.getHost());
responseContext.put(metadataRequest.getFileUrl(), responseContent);
}
});
However if, I write responseContext.put(UUID.randomUUID().toString(), responseContent); I'm able to collect the responses.
Will take a look later today
@prasenjithaty Can you log/print the response content rather unmarshal the request? https://github.com/eBay/parallec-samples/blob/master/sample-apps/src/main/java/io/parallec/sample/app/http/HttpBasicMinimumApp.java#L43
For this question My question is to collect responses with the key being a field from my request body.
I suggest you pass the response context into the handler with a hashmap. In the response handler to update the hashmap with key as the field, value is the response ( or the part you need from the response).
Then when you finish ( outside of the handler) to check on this hashmap from the context again to parse the aggregated data
here is an example: https://github.com/eBay/parallec/blob/master/src/test/java/io/parallec/core/main/http/ParallelClientHttpBasicTest.java#L105
@jeffpeiyt log/printing the response content prints the responses perfectly fine. It breaks only when I try to unmarshal the request.
And I've passed response context as a hashmap into the handler as given in the example. if you see my first post on Ln.6 that is what I've done.
Figured out the issue. Problem was while unmarshalling. Got it working finally. Thanks for your help @jeffpeiyt
@prasenjithaty great! Thanks for trying parallec.io!
@jeffpeiyt does the library support an easy way of retrying requests? Like the interceptors in OkHttp? https://github.com/square/okhttp/wiki/Interceptors
@prasenjithaty you can use your customized async http client with the config to set retry for requests. We have not supported other low level retry such as the interceptors but we can possibly enhance it to make it work like that by enhancing the task worker and the requesting worker
Another way to set general retry is to only retry on those failed requests with another parallel task only targeting. (we can start the 2nd parallel task in another thread by polling a hashmap storing only those failed the 1st attempt)