Face Detection with Vision Framework

Previously, in iOS 10, to detect faces in a picture, you can use CIDetector (Apple) or Mobile Vision (Google)

In iOS11, Apple introduces CoreML. With the Vision Framework, it's much easier to detect faces in real time 😃

Try it out with real time face detection on your iPhone! 📱

You can find out the differences between CIDetector and Vison Framework down below.

Moving From Voila-Jones to Deep Learning

Details

Specify the VNRequest for face recognition, either VNDetectFaceRectanglesRequest or VNDetectFaceLandmarksRequest.

private var requests = [VNRequest]() // you can do mutiple requests at the same time

var faceDetectionRequest: VNRequest!
@IBAction func UpdateDetectionType(_ sender: UISegmentedControl) {
    // use segmentedControl to switch over VNRequest
    faceDetectionRequest = sender.selectedSegmentIndex == 0 ? VNDetectFaceRectanglesRequest(completionHandler: handleFaces) : VNDetectFaceLandmarksRequest(completionHandler: handleFaceLandmarks) 
}

Perform the requests every single frame. The image comes from camera via captureOutput(_:didOutput:from:), see AVCaptureVideoDataOutputSampleBufferDelegate

func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
    guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer),
        let exifOrientation = CGImagePropertyOrientation(rawValue: exifOrientationFromDeviceOrientation()) else { return }
    var requestOptions: [VNImageOption : Any] = [:]

    if let cameraIntrinsicData = CMGetAttachment(sampleBuffer, kCMSampleBufferAttachmentKey_CameraIntrinsicMatrix, nil) {
      requestOptions = [.cameraIntrinsics : cameraIntrinsicData]
    }
    
    // perform image request for face recognition
    let imageRequestHandler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer, orientation: exifOrientation, options: requestOptions)

    do {
      try imageRequestHandler.perform(self.requests)
    }

    catch {
      print(error)
    }

}

Handle the return of your request, VNRequestCompletionHandler.

handleFaces for VNDetectFaceRectanglesRequest
handleFaceLandmarks for VNDetectFaceLandmarksRequest

then you will get the result from the request, which are VNFaceObservations. That's all you got from the Vision API

func handleFaces(request: VNRequest, error: Error?) {
    DispatchQueue.main.async {
        //perform all the UI updates on the main queue
        guard let results = request.results as? [VNFaceObservation] else { return }
        print("face count = \(results.count) ")
        self.previewView.removeMask()

        for face in results {
            self.previewView.drawFaceboundingBox(face: face)
        }
    }
}
    
func handleFaceLandmarks(request: VNRequest, error: Error?) {
    DispatchQueue.main.async {
        //perform all the UI updates on the main queue
        guard let results = request.results as? [VNFaceObservation] else { return }
        self.previewView.removeMask()
        for face in results {
            self.previewView.drawFaceWithLandmarks(face: face)
        }
    }
}

Lastly, DRAW corresponding location on the screen! <Hint: UIBezierPath to draw line for landmarks>

func drawFaceboundingBox(face : VNFaceObservation) {
    // The coordinates are normalized to the dimensions of the processed image, with the origin at the image's lower-left corner.

    let transform = CGAffineTransform(scaleX: 1, y: -1).translatedBy(x: 0, y: -frame.height)

    let scale = CGAffineTransform.identity.scaledBy(x: frame.width, y: frame.height)

    let facebounds = face.boundingBox.applying(scale).applying(transform)

    _ = createLayer(in: facebounds)

}

// Create a new layer drawing the bounding box
private func createLayer(in rect: CGRect) -> CAShapeLayer {

    let mask = CAShapeLayer()
    mask.frame = rect
    mask.cornerRadius = 10
    mask.opacity = 0.75
    mask.borderColor = UIColor.yellow.cgColor
    mask.borderWidth = 2.0

    maskLayer.append(mask)
    layer.insertSublayer(mask, at: 1)

    return mask
}

ArianZargaran/AppleFaceDetection

Face Detection with Vision Framework

Details