MetalPetal

An image processing framework based on Metal.

Design Overview
Builtin Filters
Example Code
Quick Look Debug Support
Best Practices
Build Custom Filter
Install
Contribute
License

Design Overview

MetalPetal is an image processing framework based on Metal designed to provide real-time processing for still image and video with easy to use programming interfaces.

This chapter covers the key concepts of MetalPetal, and will help you to get a better understanding of its design, implementation, performance implications and best practices.

Goals

MetalPetal is designed with the following goals in mind.

Easy to use API

Provides convenience APIs and avoids common pitfalls.
Performance

Use CPU, GPU and memory efficiently.
Extensibility

Easy to create custom filters as well as plugin your custom image processing unit.
Swifty

Provides a fluid experience for Swift programmers.

Core Components

Some of the core concepts of MetalPetal are very similar to those in Apple's Core Image framework.

MTIContext

Provides an evaluation context for rendering MTIImages. It also stores a lot of caches and state information, so it's more efficient to reuse a context whenever possible.

MTIImage

A MTIImage object is a representation of an image to be processed or produced. It does directly represent image bitmap data instead it has all the information necessary to produce an image or more precisely a MTLTexture. It consists of two parts, a recipe of how to produce the texture (MTIImagePromise) and other information such as how a context caches the image (cachePolicy), and how the texture should be sampled (samplerDescriptor).

MTIFilter

A MTIFilter represents an image processing effect and any parameters that control that effect. It produces a MTIImage object as output. To use a filter, you create a filter object, set its input images and parameters, and then access its output image. Typically, a filter class owns a static kernel (MTIKernel), when you access its outputImage property, it asks the kernel with the input images and parameters to produce a output MTIImage.

MTIKernel

A MTIKernel represents an image processing routine. MTIKernel is responsible for creating the cooresponding render or compute pipeline state for the filter, as well as building the MTIImagePromise for a MTIImage.

Alpha Type Handling

If an alpha channel is used in an image, there are two common representations that are available: unpremultiplied (straight/unassociated) alpha, and premultiplied (associated) alpha.

With unpremultiplied alpha, the RGB components represent the color of the pixel, disregarding its opacity.

With premultiplied alpha, the RGB components represent the color of the pixel, adjusted for its opacity by multiplication.

Most of the filters in MetalPetal accept unpremultiplied alpha and opaque images and output unpremultiplied alpha images. Some filters, such as MTIMultilayerCompositingFilter accepts both unpremultiplied/premultiplied alpha images.

MetalPetal handles alpha type explicitly. You are responsible for providing the correct alpha type during image creation.

There are three alpha types in MetalPetal.

MTIAlphaType.nonPremultiplied: the alpha value in the image is not premultiplied.

MTIAlphaType.premultiplied: the alpha value in the image is premultiplied.

MTIAlphaType.alphaIsOne: there's no alpha channel in the image or the image is opaque.

Typically, CGImage, CVPixelBuffer, CIImage objects have premultiplied alpha channel. MTIAlphaType.alphaIsOne is strongly recommanded if the image is opaque, e.g. a CVPixelBuffer from camera feed, or a CGImage loaded from a jpg file.

You can call unpremultiplyingAlpha() or premultiplyingAlpha() on a MTIImage to convert the alpha type of the image.

For performance reasons, alpha type validation only happens in debug build.

Optimizations

MetalPetal does a lot of optimizations for you under the hood.

It automatically caches functions, kernel states, samplers, etc.

Before rendering, MetalPetal can look into your image render graph and figure out the minimal number of intermedinate textures needed to do the rendering, saving memory, energy and time.

It can also re-organize the image render graph if multiple “recipes” can be concatenated to eliminate redundant render passes. (MTIContext.isRenderGraphOptimizationEnabled)

Concurrency Considerations

MTIImage objects are immutable, which means they can be shared safely among threads.

However, MTIFilter objects are mutable and thus cannot be shared safely among threads.

A MTIContext contains a lot of states and caches. There's a thread-safe mechanism for MTIContext objects, making it safe to share a MTIContext object among threads.

Advantages over Core Image

Fully customizable vertex and fragment functions.
MRT (Multiple Render Targets) support.
Generally better performance. (Detailed benchmark data needed)

Builtin Filters

Color Matrix
Color Lookup

Uses an color lookup table to remap the colors in an image.
Opacity
Exposure
Saturation
Brightness
Contrast
Color Invert
Vibrance

Adjusts the saturation of an image while keeping pleasing skin tones.
RGB Tone Curve
Blend Modes
- Normal
- Multiply
- Overlay
- Screen
- HardLight
- SoftLight
- Darken
- Lighten
- ColorDodge
- Difference
- Exclusion
- Hue
- Saturation
- Color
- Luminosity
- ColorLookup512x512
Blend with Mask
Transform
Crop
Pixellate
Multilayer Composite
MPS Convolution
MPS Gaussian Blur
High Pass Skin Smoothing
CLAHE (Contrast-Limited Adaptive Histogram Equalization)
Lens Blur (Hexagonal Bokeh Blur)

Example Code

Create a `MTIImage`

let imageFromCGImage = MTIImage(cgImage: cgImage)

let imageFromCIImage = MTIImage(ciImage: ciImage)

let imageFromCoreVideoPixelBuffer = MTIImage(cvPixelBuffer: pixelBuffer)

let imageFromContentsOfURL = MTIImage(contentsOf: url)

Create a filtered image

let inputImage = ...

let filter = MTISaturationFilter()
filter.saturation = 0
filter.inputImage = inputImage

let outputImage = filter.outputImage

Render a `MTIImage`

let context = MTIContext()

let image: MTIImage = ...

do {
    try context.render(image, to: pixelBuffer) 
    
    //context.makeCIImage(from: image)
    
    //context.makeCGImage(from: image)
} catch {
    print(error)
}

Quick Look Debug Support

If you do a Quick Look on a MTIImage, it'll show you the image graph that you constructed to produce that image.

Best Practices

Reuse a MTIContext whenever possible.

Contexts are heavyweight objects, so if you do create one, do so as early as possible, and reuse it each time you need to render an image.
Use MTIImage.cachePolicy wisely.

Use MTIImageCachePolicyTransient when you do not want to preserve the render result of a image, for example when the image is just an intermediate result in a filter chain, so the underlying texture of the render result can be reused. It is the most memory efficient option. However, when you ask the context to render a previously rendered image, it may re-render that image since its underlying texture has been reused.

By default, a filter's output image has the transient policy.

Use MTIImageCachePolicyPersistent when you want to prevent the underlying texture from being reused.

By default, images created from external sources have the persistent policy.

Understand that MTIFilter.outputImage is a compute property.

Each time you ask a filter for its output image, the filter may give you a new output image object even if the inputs are identical with the previous call. So reuse output images whenever possible.

For example,

//          ╭→ filterB
// filterA ─┤
//          ╰→ filterC
// 
// filterB and filterC use filterA's output as their input.

In this situation, the following solution:

let filterOutputImage = filterA.outputImage
filterB.inputImage = filterOutputImage
filterC.inputImage = filterOutputImage

is better than:

filterB.inputImage = filterA.outputImage
filterC.inputImage = filterA.outputImage

Build Custom Filter

If you want to include the MTIShaderLib.h in your .metal file, you need to add ${PODS_CONFIGURATION_BUILD_DIR}/MetalPetal/MetalPetal.framework/Headers to the Metal Compiler - Header Search Paths (MTL_HEADER_SEARCH_PATHS).

Simple single input/output filters

To build a custom unary filter, you can subclass MTIUnaryImageRenderingFilter and override the methods in the SubclassingHooks category. Examples: MTIPixellateFilter, MTIVibranceFilter, MTIUnpremultiplyAlphaFilter, MTIPremultiplyAlphaFilter, etc.

@interface MTIPixellateFilter : MTIUnaryImageRenderingFilter

@property (nonatomic) float fractionalWidthOfAPixel;

@end

@implementation MTIPixellateFilter

- (instancetype)init {
    if (self = [super init]) {
        _fractionalWidthOfAPixel = 0.05;
    }
    return self;
}

+ (MTIFunctionDescriptor *)fragmentFunctionDescriptor {
    return [[MTIFunctionDescriptor alloc] initWithName:@"pixellateEffect" libraryURL:[bundle URLForResource:@"default" withExtension:@"metallib"]];
}

- (NSDictionary<NSString *,id> *)parameters {
    return @{@"fractionalWidthOfAPixel": @(self.fractionalWidthOfAPixel)};
}

@end

Fully custom filters

To build more complex filters, all you need to do is create a kernel (MTIRenderPipelineKernel/MTIComputePipelineKernel/MTIMPSKernel), then apply the kernel to the input image(s). Examples: MTIChromaKeyBlendFilter, MTIBlendWithMaskFilter, MTIColorLookupFilter, etc.

@interface MTIChromaKeyBlendFilter : NSObject <MTIFilter>

@property (nonatomic, strong, nullable) MTIImage *inputImage;

@property (nonatomic, strong, nullable) MTIImage *inputBackgroundImage;

@property (nonatomic) float thresholdSensitivity;

@property (nonatomic) float smoothing;

@property (nonatomic) MTIColor color;

@end

@implementation MTIChromaKeyBlendFilter

@synthesize outputPixelFormat = _outputPixelFormat;

+ (MTIRenderPipelineKernel *)kernel {
    static MTIRenderPipelineKernel *kernel;
    static dispatch_once_t onceToken;
    dispatch_once(&onceToken, ^{
        kernel = [[MTIRenderPipelineKernel alloc] initWithVertexFunctionDescriptor:[[MTIFunctionDescriptor alloc] initWithName:MTIFilterPassthroughVertexFunctionName] fragmentFunctionDescriptor:[[MTIFunctionDescriptor alloc] initWithName:@"chromaKeyBlend"]];
    });
    return kernel;
}

- (instancetype)init {
    if (self = [super init]) {
        _thresholdSensitivity = 0.4;
        _smoothing = 0.1;
        _color = MTIColorMake(0.0, 1.0, 0.0, 1.0);
    }
    return self;
}

- (MTIImage *)outputImage {
    if (!self.inputImage || !self.inputBackgroundImage) {
        return nil;
    }
    return [self.class.kernel applyToInputImages:@[self.inputImage, self.inputBackgroundImage]
                                      parameters:@{@"color": [[MTIVector alloc] initWithFloat4:(simd_float4){self.color.red, self.color.green, self.color.blue,self.color.alpha}],
                                    @"thresholdSensitivity": @(self.thresholdSensitivity),
                                               @"smoothing": @(self.smoothing)}
                         outputTextureDimensions:MTITextureDimensionsMake2DFromCGSize(self.inputImage.size)
                               outputPixelFormat:self.outputPixelFormat];
}

@end

Multiple draw calls in one render pass

You can use MTIRenderCommand to issue multiple draw calls in one render pass.

Install

You can use CocoaPods to install the lastest version.

use_frameworks!

pod 'MetalPetal'

# Swift extensions (optional).
pod 'MetalPetal/Swift'

Contribute

Thank you for considering contributing to MetalPetal. Please read our Contributing Guidelines.

License

MetalPetal is MIT-licensed. LICENSE

The files in the /MetalPetalDemo directory are licensed under a separate license. LICENSE.md

Documentation is licensed CC-BY-4.0.

jeslyvarghese/MetalPetal