Memory leak with getImageData()

[Nesciosquid]

Hey! Thanks for the awesome library, I'm using it for a graduate computer vision course at Boston University.

I've been playing around with your demos, and noticed a memory leak that occurs when you try to run them on large (HD) video sources. Thankfully, it's not a problem with your code, but instead with the way you're grabbing image data from the 2D rendering context.

In your debug and debug-posit examples, you have the following function updating a 2d canvas context:

    function snapshot(){
      context.drawImage(video, 0, 0, camera.width, camera.height);
      imageData = context.getImageData(0, 0, camera.width, camera.height);
    }

With HD video sources, calling context.getImageData() multiple times quickly causes memory usage to skyrocket (100's of MB per second).

For now, it seems like I'll have to hack something together where I downsample the HD video stream before writing to/reading from the canvas in order to get the image into the correct format for this library.

How tough would it be to add support for images in formats other than ImateData from a canvas object?

[Nesciosquid]

I was able to hack together a temporary solution using webcam.js to create a JPEG data URI using the snap() function, which I then feed into (pixel-util)[https://github.com/59naga/pixel-util] to convert it to an ImageData object. This is pretty clunky, and it seems to choke itself to death after a few minutes, but it's better than the rapid memory explosion that I get with context.getImageData() being called on every frame.

function snapshot() {
   Webcam.snap( function(data_uri){
    pixelUtil.fetchImageData(data_uri).then(function(data){
      imageData = data;
    });
  });
};

Will post more if I can figure out why pixel-util is crashing and stop it from doing that.

[jcmellado]

Which browser are you using?

[Nesciosquid]

I'm using Chrome. I'll see if I can't whip up an example real quick showing the leak...

[Nesciosquid]

Here, I've hosted an example where I changed the video stream from your debug-posit example to request a 720p webcam stream.

https://nesciosquidsecure.github.io/CS585_Project/leak/

A better solution to the one I mentioned above (using pixel-util) is to simply leave the canvas as a small size and adjust the image drawn to the canvas to downscale the HD webcam stream. Pose detection seems to still work well at the smaller resolution.

I then use the original, HD webcam image to pass into Three.js as a texture, and render on top of it.

[jcmellado]

Is your solution (with the small canvas) working fine then?

I'm closing this because it isn't really a js-aruco issue, but thanks for reporting it anyway.

[Nesciosquid]

Yep, small canvas seems to be working OK.