ml5js/ml5-next-gen

Making `tfjs` runtime the default

Opened this issue · 2 comments

After discussing with @shiffman, I am creating an issue to start a discussion about potentially switching our default runtime to tfjs instead of mediapipe.

The MediaPipe runtime has been causing some issues:

  • Unable to load two models at the same time: #69
  • Memory leak: #129

The TFJS runtime, although slightly worse in performance, seems to be more stable and without the above issues. I think it might be a good idea to set the tfjs runtime as the default for all models and have mediapipe as a more "experimental" option.

Please share your thoughts on this @gohai @MOQN!

I too would pick stability over performance, if the better-performing avenue is having those other drawbacks that impact the experience of folks using it.

Just to share some thoughts on the issues @ziyuan-linn mentioned (in case we want to explore addressing those still):

  • Unable to load two models at the same time: I am wondering if we could work around this with a synchronization mechanism, where the parent ml5 object keeps a promise (or an array of promises) that get fulfilled whenever a model finishes loading? And a helper function that waits for no outstanding promises before returning? This function could be sprinkled in to the classes that need it?

  • Memory leak: those seem to be notoriously hard to get to the bottom of... I'd do the same approach as you already did - checking if this also occurs in tfjs example codes, and filing bugs against them if yes, and seeing if there is anything we're doing differently if not

Thank you for the thoughts @gohai! @ziyuan-linn let's go ahead and switch over to tfjs for the upcoming soft launch! We can keep issues open and revisit in the future.