Library for accessing VisionKit and visual applications of CoreML from React Native. iOS Only
Incredibly super-alpha, and endeavors to provide a relatively thin wrapper between the underlying vision functionality and RN. Higher-level abstractions are @TODO and will be in a separate library.
yarn add react-native-vision react-native-swift
react-native link
Note react-native-swift is a peer dependency of react-native-vision.
The package exports a number of components to facilitate the vision process. Note that the <RNVisionProvider /> needs to be ancestors to any others in the tree. So a simple single-classifier using dominant image would look something like:
<RNVisionProvider isStarted={true}>
<RNVDefaultRegion classifiers={[url: this.state.FileUrlOfClassifier, max: 5]}>
{({classifications})=>{
return (
<Text>
{classifications[this.state.FileUrlOfClassifier][0].label}
</Text>
}}
</RNVDefaultRegion>
</RNVisionProvider>Context provider for information captured from the camera. Allows the use of regional detection methods to initialize identification of objects in the frame.
isStarted: Whether the camera should be activated for vision capture. BooleanisCameraFront: Facing of the camera. False for the back camera, true to use the front. Note only one camera facing can be used at a time. As of now, this is a hardware limitation.regions: Specified regions on the camera capture frame articulated as{x,y,width,height}that should always be returned by the consumertrackedObjects: Specified regions that should be tracked as objects, so that the regions returned match these object IDs and show current position.onRegionsChanged: Fires when the list of regions has been alteredonDetectedFaces: Fires when the number of detected faces has changed
detectFaces: Triggers one call to detect faces based on current active frame. Directly returns locations.
Consumer partner of RNVisionProvider. Must be its descendant in the node tree.
imageDimensions: Object representing size of the camera frame in{width, height}isCameraFront: Relaying whether camera is currently in selfie mode. This is important if you plan on displaying camera output, because in selfie mode a preview will be mirrored.regions: The list of detected rectangles in the most recently captured frame, where detection is driven by theRNVisionProviderprops
region: ID of the region (Note the default region, which is the whole frame, has an id of""- blank.)classifiers: CoreML classifiers passed as file URLs to the classifier mlmodelc itself. Arraygenerators: CoreML image generators passed as file URLs to the classifier mlmodelc itself. Arraygenerators: CoreML models that generate a collection of output values passed as file URLs to the classifier mlmodelc itself.bottlenecks: A collection of CoreML models that take other CoreML model outputs as their inputs. Keys are the file URLs of the original models (that take an image as their input) and values are arrays of mdoels that generate the output passed via render props.onFrameCaptured: Callback to fire when a new image of the current frame in this region has been captured. Making non-null activates frame capture, setting to null turns it off. The callback passes a URL of the saved frame image file.
key: ID of the regionframe: Collection containing{x, y, width, height}of the frame containing the region. All values expressed as percentages of the overall frame size, so a 50x100 frame at origin 5,10 in a 500x500 frame would come across as{x: 0.01, y: 0.02, width: .1, height: .2}. Changes in this value are often what drives the re-render of the component (and therefore re-run of the render prop)confidence: If set, the confidence that the object identified askeyis actually at this location. Used by tracked objects API of iOS Vision. Sometimes null.classifications: Collection, keyed by the file URL of the classifier passed in props, of collections of labels and probabilities. (e.g.{"file:///path/to/myclassifier.mlmodelc": {"label1": 0.84, "label2": 0.84}})genericResults: Collection of generic results returned from generic models passed in via props to the region
Convenience region that references the full frame. Same props as RNVRegion, except region is always set to "" - the full frame. Useful for simple style transfers or "dominant image" classifiers.
Same as RNVRegion, with the exception that region is forced to ""
Same as RNVRegion, with the note that key will always be ""
Preview of the camera captured by the RNVisionProvider.
Note that the preview is flipped in selfie mode (e.g. when isCameraFront is true)
The properties of a View plus:
gravity: how to scale the captured camera frame in the view. String. Valid values:fill: Fills the rectangle much like the "cover" in an Imageresize: Leaves transparent (or style:{backgroundColor}) the parts of the rectangle that are left over from a resized version of the image.
Render prop consumer for delivering additional context that regions will find helpful, mostly for rendering rectangles that map to the regions identified.
viewPortDimensions: A collection of{width, height}of the view rectangle.viewPortGravity: A pass-through of thegravityprop to help decide how to manage the math converting coordinates.
A compound consumer that blends the render prop members of RNVRegion and RNVCameraConsumer and adds a style prop that can position the region on a specified camera preview
Same as RNVRegion
Includes members from RNVRegion and RNVCameraConsumer and adds:
style: A pre-built colleciton of style prop members{position, width, height, left, top}that are designed to act in the context of theRNVCameraViewrectangle. Spread-prop with your other style preferences (border? backgroundColor?) for easy on-screen representation.
View for displaying output of image generators. Link it to , and the resulting image will display in this view. Useful for style transfer models. More performant because there is no round trip to JavaScript notifying of each frame update.
id: the ID of an image generator model attached to a region. Usually is thefile:///URL of the .mlmodelc.
Otherwise conforms to Image and View API.