ibmtjbot/tjbot

Remove support for camera due to Visual Recognition API not being available anymore

robertoetcheverryr opened this issue · 12 comments

robertoetcheverryr commented

As the title says, May I make a PR removing the config.js option for the camera or at least adding a comment that it's not available anymore? And the same with the ibm-credentials.env file? Maybe delete the switch case regarding the camera?

jweisz commented

@robertoetcheverryr I originally decided to leave it in just in case someone had an already-provisioned visual recognition service, but the likelihood of this keeps going down over time. That said, I'm not fully convinced we should remove the camera support, as it's a huge part of what makes TJBot fun! But, it's not clear to me what the other options might be. Are there any vision models we might be able to run locally on the Pi?

robertoetcheverryr commented

I haven't used it yet, but Google has a Vision API with 1000 monthly queries. I'm not sure if that works with the free account or with the "you must put your CC and only then you get the free tier"...

jweisz commented

We would not able to officially support the use of Google APIs with TJBot.

andycitron commented

I do have an already-provisioned visual recognition service provisioned, but it seems to have stopped working. I'm having trouble figuring out why. The reply tjbot returns says: Error: <TITLE>Error</TITLE>
An error occurred while processing your request.


Reference #30.713a2f17.1639419281.1a1a60f6

Do I have any hope of working around this?
jweisz commented

Likely not. There's no support for the Watson Visual Recognition service anymore as it's been discontinued, and it seems I really should go ahead with removing it from the TJBot library. That said, I haven't yet found a viable replacement, since I'd hate for TJBot to lose the ability to see. 😢

I'm definitely open to someone submitting a PR to replace Visual Recognition with something else. Preferably something on-device, though that might up the hardware requirements...

andycitron commented

Justin, I reimplemented my tjbot code using Microsoft Azure visual services. I uploaded the code fragments to github in case you were interested in incorporating it into your tjbot node js implementation. You can find it here:
https://github.com/andycitron/tjBotFragmentThatUsesAzureVisualServices

Note that it does introduce a dependency that the user has a Microsoft Azure account.

Also, if you want to incorporate it into tj.see(), you'd want to structure it a bit differently. tj.see() takes a photo. I did not want my Microsoft functions to have a dependency on tj.takePhoto. So I put the 'take a photo' part into the intent processing for 'see' and passed the photo into the code that uses Microsoft functions.

The code I put out there includes additional methods that invoke Microsoft facial recognition. That is not part of the 'tj.see()' functionality. That code requires pre-training of the facial recognition models. I included that because it might be useful to someone.

jweisz commented

Hey @andycitron, happy new year. :)

Thanks for the effort you put into TJBot, this is a really great contribution. Unfortunately, I won't be able to make this part of the official library because it uses a competitor's cloud service. But, I will put this on our Featured Recipes page to showcase your work.

andycitron commented

Cool. Yesterday I posted a video to Youtube illustrating how TJ works with Azure. Perhaps you want to include the 4 minute video along with the featured recipe: https://youtu.be/B92efwFqXSs

Could you give me the link to the Featured Recipes page where you referenced my code? I'd like to include a link to that on my home page. thx.

andycitron commented
jweisz commented

Hi @andycitron -- take a look at tjbot.js:792, where listen() is defined:
https://github.com/ibmtjbot/tjbotlib/blob/4fe0263bd0050f910752ae589d3b33cdb9cb93ae/src/tjbot.js#L792

The audio isn't stored locally, the data is streamed through a pipe between the microphone and a web socket. So it would need some modification to save the output to a file first, before uploading to the Microsoft service. Maybe check to see if their API supports WebSockets?

jweisz commented

Closing as this is now an issue in the tjbotlib repo: ibmtjbot/node-tjbotlib#73