encoding support in options
ldd opened this issue · 11 comments
The latest request has an option 'encoding' which allows getting files as buffers or at least, bypassing the conversion to unicode. This option is not present in browser-request not does it warn the user about it.
🆙 for this issue
Hi...
I ran into this problem as well. If you are looking to receive binary data as an ArrayBuffer or Buffer, you can set the xhr.responseType property to arraybuffer before sending the request. On return the xhr.response will contain the response as a ArrayBuffer object.
You can convert the ArrayBuffer to a Buffer thus
var buff = new Buffer( new Uint8Array(xhr.response));
Keep in mind that browserify creates the Buffer type for use in the browser. So if you are not using browserify, user the buffer library directly from npm
See pull request #29
how do you set the responseType in this browser-request module? there do you put this?
Hi .. it seems a lot of pull requests haven't been added in here.
I have extensively modified browser-request to add in a bunch of features such as binary buffers, callbacks, binary file uploads in multiple formats etc.
You can download it from https://github.com/yegodz/browser-request
I haven't updated the documentation from the original fork, but this code is working in a fairly complex web application.
@yegodz ok, a few questions then: 1. does "npm install browser-request" install your version? or should I download it manually?
2. is there any other besides reading the sources to find out how to send/receive binary data?
(at least some test/example/...)
I haven’t uploaded to npm yet (meaning to do so sometime :-)
For now, just download and use it from Github.
I will be updating the documentation on github shortly, but meanwhile, you can do two things:
- See some code snippets below
- See the changes I have put in by looking at the last commit on my repo. That will tell you exactly what has changed.
Some extra options are available to be added to the ‘options’ object that is passed to request
e.g.
progressCallback points to a function that is called everytime some block of data is loaded by the system. This helps you to display a progress bar etc for file upload/downloads
responseType: ‘buffer’ indicates that the expected data returned in the ‘body’ argument of the callback should be in a binary buffer
var options = {
url: res.downloadUrl,
headers: {
//'User-Agent': 'xo',
Authorization: 'Bearer ' + thisObj.tokens.access_token
},
progressCallback: function (progressEvt){
thisObj.downloads[fileid].total = progressEvt.total;
thisObj.downloads[fileid].loaded = progressEvt.loaded;
if (progressEvt.loaded >= chunkDoc.size)
console.log('Completed Google download')
},
encoding: null,
responseType: 'buffer' // required in the browser version of request
};
request.get(options,
function (err, response, body) {
if (err) {
...
}
…
Support for Multipart/related uploads
var fileMeta = {
'Content-Type': 'application/json',
body: JSON.stringify({
modifiedDate: chunkDoc.mtime
})
}
var fileData = {
'Content-Type': 'application/octet-stream',
body: chunkBuff
}
var options = {
url: 'https://www.googleapis.com/upload/drive/v2/files/'
headers: {
'Authorization': 'Bearer ' + thisObj.tokens.access_token
},
qs: {uploadType: 'multipart'},
progressCallback: function (progressEvt){
thisObj.uploads[filename].total = progressEvt.total;
thisObj.uploads[filename].loaded = progressEvt.loaded;
if (progressEvt.loaded >= chunkDoc.size)
console.log('Completed Google upload')
},
multipart: [fileMeta, fileData]
}
Support for multipart/formData file upload
var formData = {
attributes: JSON.stringify({
name: chunkDoc.uri,
parent: {
id: thisObj.rootFolderId
}
}),
file: {
value: chunkBuff,
options: {
filename: 'file',
contentType: 'application/octet-stream'
}
}
}
var options = {
url: 'https://upload.box.com/api/2.0/files/content',
headers: {
//'User-Agent': 'xo',
Authorization: 'Bearer ' + thisObj.tokens.access_token
},
formData: formData,
progressCallback: function (progressEvt){
thisObj.uploads[fileDoc_id].total = progressEvt.total;
thisObj.uploads[fileDoc_id].loaded = progressEvt.loaded;
if (progressEvt.loaded >= chunkDoc.size)
console.log('Completed Box upload')
}
}
Ruchir Godura
ruchir@cerient.com
Founder, Cerient Technology, LLC.
Keep your cloud data private at http://www.xooui.com http://www.xooui.com/
On Jul 31, 2015, at 10:47 AM, Tomas Novella notifications@github.com wrote:
@yegodz https://github.com/yegodz ok, a few questions then: 1. does "npm install browser-request" install your version? or should I download it manually?
2. is there any other besides reading the sources to find out how to send/receive binary data?
(at least some test/example/...)—
Reply to this email directly or view it on GitHub #16 (comment).
Hey. Thank you for the examples!
I tried it, but still I'm having the same problem - in node(iojs) environment the script runs like clockwork, whereas in the browser I get "Failed to load resource: the server responded with a status of 400 (OK)".
Has it ever happened to you?
let syncRequest = new Bufferr(new Uint8Array(BuildSyncRequest(db)));
let url = /*'http://localhost/chrome-sync/command';//*/'https://clients4.google.com/chrome-sync/command';
console.log('syncRequest instanceof Uint8Array?', syncRequest instanceof Uint8Array,
'len:', syncRequest.length, 'TYPE:', syncRequest.toString());
request.post({
url: url,
//qs: {
// 'client': 'Google Chrome',
// 'client_id': config.clientId
//},
headers: {
// todo just test, send the same headers from node as from browser...
'Accept': '*/*',
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.15 Safari/537.36',
'Accept-Encoding': 'gzip, deflate',
'Accept-Language': 'en-US,en;q=0.8,cs;q=0.6,sk;q=0.4,de;q=0.2',
'Content-Type': 'application/octet-stream',
'Authorization': 'Bearer '+ accessToken
},
encoding: null, // if you expect binary data
responseType: 'buffer',
body: syncRequest
}, (error, response, body) => {
console.log('error', error,'body', body);
if (!error) {
readSyncRequest(body, cb);
}
});
Is this module supposed to handle also SENDING binary data?
Not really … if you use Chrome, you can look at the dev tools and actually capture every HTTP request as it goes out.
There you can check if the outgoing HTTP request has been constructed properly.
It seems like you are creating a database query that you are sending to a remote server that executes it and returns the results in a binary buffer. However, the remote server does not seem to be getting the body of your post properly formatted, hence the 400 response.
- Try not adding the ‘User-Agent’ header… you are not allowed to add those when sending http req from a browser, and in general should be ignored, but it may causing some error.
- Try not adding the new Buffer( ) around the new UIn8Array in the browser
- Are you sure the syncRequest that you are building is expected by the servers as a binary file? Try just setting the ‘body’ in the post request to the string returned by the BuildSyncRequest
Ruchir Godura
ruchir@cerient.com
Founder, Cerient Technology, LLC.
Keep your cloud data private at http://www.xooui.com http://www.xooui.com/
On Aug 3, 2015, at 5:34 AM, Tomas Novella notifications@github.com wrote:
Hey. Thank you for the examples!
I tried it, but still I'm having the same problem - in node(iojs) environment the script runs like clockwork, whereas in the browser I get "Failed to load resource: the server responded with a status of 400 (OK)".
Has it ever happened to you?let syncRequest = new Bufferr(new Uint8Array(BuildSyncRequest(db)));
let url = /_'http://localhost/chrome-sync/command';//_/'https://clients4.google.com/chrome-sync/command';console.log('syncRequest instanceof Uint8Array?', syncRequest instanceof Uint8Array,
'len:', syncRequest.length, 'TYPE:', syncRequest.toString());request.post({
url: url,
//qs: {
// 'client': 'Google Chrome',
// 'client_id': config.clientId
//},
headers: {
// todo just test, send the same headers from node as from browser...
'Accept': '/',
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.15 Safari/537.36',
'Accept-Encoding': 'gzip, deflate',
'Accept-Language': 'en-US,en;q=0.8,cs;q=0.6,sk;q=0.4,de;q=0.2','Content-Type': 'application/octet-stream', 'Authorization': 'Bearer '+ accessToken }, encoding: null, // if you expect binary data responseType: 'buffer', body: syncRequest}, (error, response, body) => {
console.log('error', error,'body', body);
if (!error) {
readSyncRequest(body, cb);
}
});
—
Reply to this email directly or view it on GitHub #16 (comment).
Thanks for the advice!. Tried all the things, didnt work. Now I got one hypothesis and this seems to be the case: google sends 400 if I send any cookies in that request(chrome adds automatically cookies relevant for the domain).
yes.. that is one more difference between the browser and node.js is that the browser will automatically send any relevant cookies whereas node will only send those that you explicitly add to the request.
Ruchir Godura
ruchir@cerient.com
Founder, Cerient Technology, LLC.
Keep your cloud data private at http://www.xooui.com http://www.xooui.com/
On Aug 3, 2015, at 9:55 AM, Tomas Novella notifications@github.com wrote:
Tried all the things, didnt work. Now I got one hypothesis and this seems to be the case: google sends 400 if I send any cookies. And it seems to be sending cookies (that belong to google).
—
Reply to this email directly or view it on GitHub #16 (comment).
Yeah, in the end the problem was with the cookie. Thanks for the examples:)