Memory Problems
Closed this issue · 15 comments
After the proxy runs for a while, the baseline memory usage is definitely going in an upward trend. The usage also fluctuates a lot. For example, if I play 2 youtube HD video together, the memory consumption can go more than 125 MB (25% on a 512MB VPS).
I am playing with all the profiling methods that I found in google. If you have any quick hunches or insights, it will be very helpful. Thanks.
Hmm, thanks for reporting this. Quick question: are you streaming YT videos or HTTPS or HTTP?
Guessing, HTTP...
I think it is http. There seems to be similar problems with ftp downloading, as well.
Which version of node are you on? It looks like there have been a number of memory leak fixes between 0.8.0 and 0.8.6 (current). Having said that, it does look like we have a leak in the proxy - the connection is not properly terminated when client aborts -- looking into it..
Thanks very much for spending time on it. I am using 0.8.4. I will upgrade to 0.8.6.
The behavior of the huge fluctuation differs on the hardware. It seems that the memory usage spike is much larger on a faster hardware. I suspect the stream's pipe didn't cut off the incoming traffic quick enough.
Hmm, this is bizzarre.. Either I'm missing something completely obvious here, or there is a bug somewhere up in the stack. Steps to reproduce the problem:
- launch spdyproxy + Chrome
- open a YT video.. switch to HD format for best demo :)
- start playing video, kill Chrome
At this point, spdyproxy continues to accumulate data from YT.. you can just see it grow in size as its fetching data from YT. For whatever reason, I'm not seeing the error (I would expect EPIPE) fire on spdyproxy.. Well, almost.. It does, but very rarely, and I can't figure out what triggers it to fire.
@indutny any thoughts or ideas?
I'll try to figure it out in next days
Hey Peter. Confused... did you reply with the below on the ticket, or this
a private message? I can't see your response on the actual thread... or I'm
missing something obvious here. (confused by new UI / changes).
ig
On Mon, Aug 20, 2012 at 11:10 AM, peteroconnor notifications@github.comwrote:
So I tried to insert the union/buffered-stream into the pipe, and applied
an explicit limit on the buffer size.peteroconnor@7460c9fhttps://github.com/peteroconnor/node-spdyproxy/commit/7460c9f169
However, the following code in buffered-stream did nothing stopping the
incoming traffic filling up the memory, when spdyproxy is deployed on a *
real* server.BufferedStream.prototype.write = function (chunk) {
if (!this.chunks || this.piped) {
this.emit('data', chunk);
return;
}this.chunks.push(chunk);
this.size += chunk.length;
if (this.limit < this.size) {
this.pause();
}};Any pointers are really appreciated.
@indexzero https://github.com/indexzero @flatironhttps://github.com/flatiron
—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3#issuecomment-7878005.
Sorry for confusion. I did post something, but realized there are a few tests that I should run. However, I was running out of time.
I will post again, once I finishes these tests.
I am struggling to create a stream buffer in the pipe, sp that I can throttle and limit the memory usage, as I am just picking up nodejs.
Just wonder if anyone has got a chance to look at it.
I believe the behavior you originally described was due to a bug in node-spdy.. Not sure that there is much you can do on your end to address this.
@indutny have you had any luck with this one?
Your time is deeply appreciated.
Chrome's https-over-spdy support is the first time that it is possible for webvpn to be popularized, barring several existing proprietary solutions.
node-spdyproxy is the only forward spdy proxy server implementation that I could find is close to production-ready. Most spdy developments have been focusing on reverse proxies.
@igrigorik sorry, I was busy with other stuff, will look on it today.
Looks like it did the trick. Thank you both very much.
You're welcome!
Sorry for taking too much time to process it.
0.2.3 should be up now.