brianmtully/flutter_google_ml_vision

No text detected with a physical Android device

Opened this issue · 9 comments

cswkim commented

Using the latest version 0.0.5 and a physical Android phone on Android 11 API 30 I can't seem to get the example to detect text. This util method returns a VisionText instance that should have a list of blocks/lines/elements but it always returns an empty [] for it's blocks field. I have no issues using iOS. I'm not sure how to debug further, does anyone else have this issue?

flutter doctor -v 

[✓] Flutter (Channel stable, 2.2.1, on Mac OS X 10.15.7 19H1030 darwin-x64, locale en-US)
    • Flutter version 2.2.1 at /Users/cswkim/Dev/flutter
    • Framework revision 02c026b03c (4 days ago), 2021-05-27 12:24:44 -0700
    • Engine revision 0fdb562ac8
    • Dart version 2.13.1

[✓] Android toolchain - develop for Android devices (Android SDK version 29.0.2)
    • Android SDK at /Users/cswkim/Library/Android/sdk
    • Platform android-29, build-tools 29.0.2
    • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6915495)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 12.4, Build version 12D4e
    • CocoaPods version 1.10.1

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 4.1)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6915495)

[✓] VS Code (version 1.56.2)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.22.0

Soumajit2004 commented

i have same problem

brianmtully commented

are you detecting from camera or from an image?

brianmtully commented

Running the example application on Android 11, I get text results for both camera and picture scanning.

cswkim commented

are you detecting from camera or from an image?

I'm using the official camera plugin ^0.8.1 and the stream listener startImageStream((CameraImage image) {}). There is no additional set-up/config required for Android right? Setting any sort of gradle variable, etc.

brianmtully commented

Have you tried the example application in the repository?

brianmtully commented

Also, what ResolutionPreset are you using for the camera?

cswkim commented

I generally use ResolutionPreset.medium. Just as a test I tried it out on the Android Emulator for MacOS using a Pixel 3 device with API 29 and it seems to work fine. I'll fiddle around with it on another physical Android phone and report back.

cswkim commented

Ah strange. So on the physical Android device (Samsung Galaxy S10+), ResolutionPreset.medium produces no results for text detection, but the other values work fine: (low, high, veryHigh). Default behavior for the camera plugin is to select the next available lower resolution if the chosen is not available. I'm guessing that is not what is going on here.

cswkim commented

I decided to come back to this to try and figure out the issue and I might have stumbled upon the problem. The root of the issue is way over my head and involves knowledge of image processing, but I did find this article in case anyone is interested in the topic: https://www.programmersought.com/article/94353620149/

A more relevant link can be found in a prior open discussion on the now deprecated firebase_ml_vision library: firebase/flutterfire#5575 (comment)

This issue only seems to effect Android devices (and which ones specifically is a mixed bag). Here are some logging outputs for this util method:

Samsung Galaxy

ResolutionPreset.low
image.planes[0].bytesPerRow.toDouble(): 320.0
image.width.toDouble(): 320.0

ResolutionPreset.medium
image.planes[0].bytesPerRow.toDouble(): 768.0
image.width.toDouble(): 720.0

ResolutionPreset.high
image.planes[0].bytesPerRow.toDouble(): 1280.0
image.width.toDouble(): 1280.0

As you can see, for medium, the bytesPerRow has an extra 48.0. If you set the width here to:
size: Size(image.planes[0].bytesPerRow.toDouble(), image.height.toDouble()),
text detection now works with ResolutionPreset.medium. You DO NOT want to do this for iOS, this is an Android-specific hack.

I've only tried this out on 2 separate Samsung Galaxy physical devices and am not sure what other Android phones need to address the issue. More importantly, this problem seems to be a deeper Android-related image processing quirk and not a bug of this package. I'm not sure how I will be attempting to apply this hack, especially since I'd rather avoid having to place some ugly Platform check conditional.

Anyways, I should have put a TL;DR disclaimer. One last note, I tried to set the imageFormatGroup parameter for the camera controller to ImageFormatGroup.jpeg to see if I could avoid this issue as Android defaults to ImageFormatGroup.yuv420 but it made things worse, text detection did not work at all, for any resolution preset.