BodyPix error in p5.js instance mode

[nickmcintyre]

→ Description 📝

• found a bug 🐛

Hi! Not sure if I should raise this issue here or in the library repo. The "BodyPix with Webcam" example doesn't work in p5.js instance mode–it throws ReferenceError: loadImage is not defined.

→ Screenshots 🖼

→ Helpful Information 🦄

• Web browser and version Firefox version 67 and Chrome 74.
• the ml5-examples branch you're using - release
• Operating System macOS
• Steps to reproduce the issue - add <div id="p5sketch"></div> to the html and run the modified sketch below.
• ml5 version you're using - 0.3.1

const s = (sketch) => {
    let bodypix;
    let video;
    let segmentation;
    let img;

    const options = {
        outputStride: 8, // 8, 16, or 32, default is 16
        segmentationThreshold: 0.3 // 0 - 1, defaults to 0.5
    }

    sketch.setup = function () {
        sketch.createCanvas(320, 240);

        // load up your video
        video = sketch.createCapture(sketch.VIDEO);
        video.size(sketch.width, sketch.height);
        // video.hide(); // Hide the video element, and just show the canvas
        bodypix = ml5.bodyPix(video, modelReady)
    }

    function modelReady() {
        console.log('ready!')
        bodypix.segment(gotResults, options)
    }

    function gotResults(err, result) {
        if (err) {
            console.log(err)
            return
        }
        // console.log(result);
        segmentation = result;

        sketch.background(0);
        sketch.image(video, 0, 0, sketch.width, sketch.height)
        sketch.image(segmentation.maskBackground, 0, 0, sketch.width, sketch.height)

        bodypix.segment(gotResults, options)
    }
}

let myp5 = new p5(s, document.getElementById('p5sketch'));

[joeyklee]

@nickmcintyre Hello! (I should also say, thank you for the really nice bug report - you added all the helpful details! 🙏)

Oh! Interesting. Thanks for flagging this. I haven't tested with p5 instance mode, but will try to give this a look soon. Let me know if you discover anything in the meantime. Thanks!

@WenheLI - I feel like you might have a good idea about why this is occurring. Is it because in our p5Utils the loadImage() function would need to be prepended with the p5 object? In this case, maybe we need to add window.p5.prototype.loadImage() to our p5 functions in ml5 to ensure that our functions also work in instance mode?

[joeyklee]

Hi @nickmcintyre - I just did a quick test and yes, my above hunch was correct. It seems that in our handy p5Utils of ml5 we need to directly target the browser's window object and the p5 instance living there.

in ml5-library/src/utils/p5Utils.js

// Load Image Async
const loadAsync = (url) => {
    return new Promise((resolve) => {
        window.p5.prototype.loadImage(url, (img) => {
            resolve(img);
        });
    });
};

I managed to get a quick version working with your helpful code.

The issue we will need to resolve is this error that is thrown by p5.

[joeyklee]

@nickmcintyre - I've made a PR to start the discussion here: ml5js/ml5-library#481

We'll do our best to resolve this soon! Thanks again!

[nickmcintyre]

You rock @joeyklee! Thanks so much–can't wait to play around with the model further.

[joeyklee]

@nickmcintyre - Thanks! Looking forward to seeing your explorations!

[joeyklee]

@nickmcintyre - just circling back around on this. We currently support the instance mode now in the development build of ml5. If you are still interested to test it out, you're welcome to give it a whirl!

You can see a working example here: https://github.com/ml5js/ml5-examples/blob/development/p5js/BodyPix/BodyPix_p5Instance/sketch.js

[nickmcintyre]

This is fantastic @joeyklee! I've tinkered a bit with the development build in an existing project and it works well. A print dialogue occasionally opens on macOS but I'm not sure if it's due to ml5 or the surrounding components (React + Meteor)... I'll look into it.

Thanks man!

[joeyklee]

@nickmcintyre - Closing this up now. Thanks for reporting on this and testing!

@all-contributors please add @nickmcintyre for bug

[allcontributors]

@joeyklee

I've put up a pull request to add @nickmcintyre! 🎉