Export Green Screen in Swift

Question

Export Green Screen in Swift

Closed this issue 5 years ago · 27 comments

how it's possible to export Green Screen to asset or video, I use the example here
Green Screen in Swift & PixelKit
http://blog.hexagons.se/blog/green-screen-in-swift-pixelkit/

Answer 1 · 2020-04-05T21:11:50.000Z

Hi, you can record a video with the RecordPIX

Answer 2 · 2020-04-06T09:50:20.000Z

I mean this is my code

let content = VideoPIX()
content.load(fileNamed: "superman", withExtension: "mov")

let image = ImagePIX()
image.image = UIImage(named: "city")

let key = ChromaKeyPIX()
key.input = content
key.keyColor = .green

let blend = BlendPIX()
blend.blendingMode = .over
blend.inputA = image
blend.inputB = key

let final: PIX = blend
final.view.frame = view.bounds
view.addSubview(final.view)

how it's possible to render the video and save it with blending?

Answer 3 · 2020-04-06T10:28:11.000Z

So PixelKit is a realtime based framework. It dose not have a timeline.
Tho it's still possible to export a video!
Depending on the quality you are looking for it can be done in different ways.
All ways make use of the RecordPIX.

The quick and dirty way:

Start recording: recordPix.startRec(name: "video-name")
Start the video: content.play()
Wait with a timer. Get the duration in seconds: content.duration
Stop the recoding: recordPix.stopRec({ url in })

Let me know if you want a more frame accurate method.

Answer 4 · 2020-04-06T10:35:01.000Z

Wait with a timer. Get the duration in seconds: content.duration

you mean that need to check video duration with the playing status so I decied to stop recording

my code

        let content = VideoPIX()
        let record = RecordPIX()
        content.load(fileNamed: "bo", withExtension: "mp4")
        let image = ImagePIX()
        image.image = UIImage(named: "bg")
        let key = ChromaKeyPIX()
        key.input = content
        key.keyColor = .green
        let blend = BlendPIX()
        blend.blendMode = .over
        blend.inputA = image
        blend.inputB = key
        let final: PIX = blend
        final.view.frame = view.bounds
        view.addSubview(final.view)
        do {
            try? record.startRec(name: "testMe")
            content.play()
            record.stopRec({ url in
                print("Finish", url)
            }, didError: { error in
                print("Error Record", error)
            })
        } catch {
            print("Not Working")
        }

Answer 5 · 2020-04-06T10:39:45.000Z

Also, I got an error

PixelKit #413 ERROR [2] RecordPIX "record" >>> Rec Stop A. Writer or Input not found.
Error Record render("Rec Stop A. Writer or Input not found.")
my code

        let content = VideoPIX()
        let record = RecordPIX()
        content.load(fileNamed: "bo", withExtension: "mp4")
        let image = ImagePIX()
        image.image = UIImage(named: "bg")
        let key = ChromaKeyPIX()
        key.input = content
        key.keyColor = .green
        let blend = BlendPIX()
        blend.blendMode = .over
        blend.inputA = image
        blend.inputB = key
        let final: PIX = blend
        final.view.frame = view.bounds
        view.addSubview(final.view)
        
        try? record.startRec(name: "testMe")
        content.play()
        content.listenToDone {
            record.stopRec({ url in
                print("Finish", url)
            }, didError: { error in
                print("Error Record", error)
            })
        }

Answer 6 · 2020-04-06T10:43:08.000Z

Don't forget to connect the final to the record:

record.input = final

Then setup the timer something like this:

do {
    try record.startRec(name: "testMe")
    content.play()
    RunLoop.current.add(Timer(timeInterval: content.duration!, repeats: false, block: { _ in
        record.stopRec({ url in
            print("Finish", url)
        }, didError: { error in
            print("Error Record", error)
        })
    }), forMode: .common)
} catch {
    print("Not Working")
}

Answer 7 · 2020-04-06T10:47:28.000Z

The listenToDone way should work too.
Forgot I added that method. Nice find!
Let me know if it works!

Answer 8 · 2020-04-06T10:48:37.000Z

Thank you for helping but I got the same error even after doing the input thing

PixelKit #413 ERROR [2] RecordPIX "record" >>> Rec Stop A. Writer or Input not found.
Error Record render("Rec Stop A. Writer or Input not found.")

Code

let content = VideoPIX()
        let record = RecordPIX()
        content.load(fileNamed: "bo", withExtension: "mp4")
        let image = ImagePIX()
        image.image = UIImage(named: "bg")
        let key = ChromaKeyPIX()
        key.input = content
        key.keyColor = .green
        let blend = BlendPIX()
        blend.blendMode = .over
        blend.inputA = image
        blend.inputB = key
        let final: PIX = blend
        final.view.frame = view.bounds
        view.addSubview(final.view)
        
        record.input = blend

        try? record.startRec(name: "testMe")
        content.play()
        content.listenToDone {
            record.stopRec({ url in
                print("Finish", url)
            }, didError: { error in
                print("Error Record", error)
            })
        }

Answer 9 · 2020-04-06T10:53:37.000Z

Ok, so first make sure startRec(name:) worked by either try! or a do catch block.
Then double check listenToDone is not being called to early. That might cause the crash.
If the listenToDone method dose not work (if so I'm sorry), then try the timer method.

Answer 10 · 2020-04-06T10:57:58.000Z

maybe try to start the record and playback in the video load callback, like this:

content.load(fileNamed: "bo", withExtension: "mp4", done: { resolution in
    try! record.startRec(name: "testMe")
    content.play()
    content.listenToDone {
        record.stopRec({ url in
            print("Finish", url)
        }, didError: { error in
            print("Error Record", error)
        })
    }
})

Answer 11 · 2020-04-06T11:03:36.000Z

it export but the problem now with the video it's very strange

writer Optional(<AVAssetWriter: 0x600002130770, outputURL = file:///Users/ahmedsafadi/Library/Developer/CoreSimulator/Devices/837DB0F3-5156-4254-8E54-F94029FB9B3E/data/Containers/Data/Application/CBB87496-B419-472E-98B2-775E8FCD208C/Documents/pixelKit/renders/8624EE74-1295-463C-9D69-0910E38F3DCD/testMe.mov, outputFileType = com.apple.quicktime-movie>)
PixelKit #315 ERROR [2] RecordPIX "record" >>> Exported frame failed, writer status: 1.
PixelKit #323 WARNING [1] VideoPIX "video" Resource >>> Texture genration failed, using backup method. Error: makeTexture("Pixel Buffer to Metal Texture Converion Faied with result: -6660") LD: "The operation couldn’t be completed. (RenderKit.Texture.TextureError error 4.)"
Finish file:///Users/ahmedsafadi/Library/Developer/CoreSimulator/Devices/837DB0F3-5156-4254-8E54-F94029FB9B3E/data/Containers/Data/Application/CBB87496-B419-472E-98B2-775E8FCD208C/Documents/pixelKit/renders/8624EE74-1295-463C-9D69-0910E38F3DCD/testMe.mov

Preview looks like

Output looks like

so do you know what the problem could be?

I just want to say that this is amazing efforts from this library

Answer 12 · 2020-04-06T11:10:40.000Z

I keep getting this error when the recording is start

ixelKit #367 WARNING [1] VideoPIX "video" Resource >>> Texture genration failed, using backup method. Error: makeTexture("Pixel Buffer to Metal Texture Converion Faied with result: -6660") LD: "The operation couldn’t be completed. (RenderKit.Texture.TextureError error 4.)"
PixelKit #370 WARNING [1] VideoPIX "video" Resource >>> Texture genration failed, using backup method. Error: makeTexture("Pixel Buffer to Metal Texture Converion Faied with result: -6660") LD: "The operation couldn’t be completed. (RenderKit.Texture.TextureError error 4.)"
PixelKit #372 WARNING [1] VideoPIX "video" Resource >>> Texture genration failed, using backup method. Error: makeTexture("Pixel Buffer to Metal Texture Converion Faied with result: -6660") LD: "The operation couldn’t be completed. (RenderKit.Texture.TextureError error 4.)"
PixelKit #376 WARNING [1] VideoPIX "video" Resource >>> Texture genration failed, using backup method. Error: makeTexture("Pixel Buffer to Metal Texture Converion Faied with result: -6660") LD: "The operation couldn’t be completed. (RenderKit.Texture.TextureError error 4.)"
PixelKit #380 WARNING [1] VideoPIX "video" Resource >>> Texture genration failed, using backup method. Error: makeTexture("Pixel Buffer to Metal Texture Converion Faied with result: -6660") LD: "The operation couldn’t be completed. (RenderKit.Texture.TextureError error 4.)"
PixelKit #384 WARNING [1] VideoPIX "video" Resource >>> Texture genration failed, using backup method. Error: makeTexture("Pixel Buffer to Metal Texture Converion Faied with result: -6660") LD: "The operation couldn’t be completed. (RenderKit.Texture.TextureError error 4.)"
PixelKit #387 WARNING [1] VideoPIX "video" Resource >>> Texture genration failed, using backup method. Error: makeTexture("Pixel Buffer to Metal Texture Converion Faied with result: -6660") LD: "The operation couldn’t be completed. (RenderKit.Texture.TextureError error 4.)"
PixelKit #392 WARNING [1] VideoPIX "video" Resource >>> Texture genration failed, using backup method. Error: makeTexture("Pixel Buffer to Metal Texture Converion Faied with result: -6660") LD: "The operation couldn’t be completed. (RenderKit.Texture.TextureError error 4.)"
PixelKit #396 WARNING [1] VideoPIX "video" Resource >>> Texture genration failed, using backup method. Error: makeTexture("Pixel Buffer to Metal Texture Converion Faied with result: -6660") LD: "The operation couldn’t be completed. (RenderKit.Texture.TextureError error 4.)"
PixelKit #402 WARNING [1] VideoPIX "video" Resource >>> Texture genr

Answer 13 · 2020-04-06T11:12:42.000Z

Wow that dose not look good.
So I think you can disregard some of the errors (-6660). I'll check them.
So print double check the resolution of your blend pix with .renderResolution.
The resolution is always derived from inputA.
You can swap inputA and inputB and use .under as the blending mode.
Thanks for the kudos.

Answer 14 · 2020-04-06T11:15:03.000Z

Let me know if swapping the blend inputs help.

blend.blendMode = .under
blend.inputA = key
blend.inputB = image

Answer 15 · 2020-04-06T11:19:09.000Z

If you need to flip the video (some known bugs with vertical flipping are not fixed yet), use ._flipY()

Answer 16 · 2020-04-06T11:20:00.000Z

@Hexagons I did the swap and I guess it's work but the frame per second looks too slow

Answer 17 · 2020-04-06T11:21:09.000Z

also, I guess the green background for the video didn't go

Answer 18 · 2020-04-06T11:21:28.000Z

try blend.placement = .aspectFill

Answer 19 · 2020-04-06T11:23:10.000Z

@Hexagons yes the green hide but the record looks like the fps is soo slow after export the video is very slow show up it like 10 frame per second

Answer 20 · 2020-04-06T11:23:39.000Z

ah ok, try recordPix.timeSync = false

Answer 21 · 2020-04-06T11:23:57.000Z

or recordPix.realtime = true

Answer 22 · 2020-04-06T11:25:50.000Z

recordPix.timeSync = false ( make the video too fast)
recordPix.realtime = true ( make the video slow like it was )

Answer 23 · 2020-04-06T11:26:42.000Z

So this might be hard to fix with this method. PixelKit is a realtime software, so if your device can't playback in realtime it can't record in realtime. Tho there is a manual override mode. It's a bit more complex to setup. This will render frame accurate video. Let me know if you want try learn the method and I can write a post about it.

Answer 24 · 2020-04-06T11:31:45.000Z

It involves turning off realtime mode in the engine, seeking the video to each frame and rendering manually by saving the .renderedImage of the PIX.

PixelKit.main.render.engine.renderMode = .manual
PixelKit.main.render.engine.manuallyRender { print("frame rendered") }

Answer 25 · 2020-04-06T11:31:53.000Z

First, thank you very much for this kind of fast and helping

what I discover that I guess the export setting need to be controlled by the user since you export .mov with a very good setting for good quality, that what makes it slow my guessing, I try on a real device it was faster but not smooth, I recommend creating documentation for export since this will make this library very famous.

and again thank you

Output
test.mov.zip

Answer 26 · 2020-04-06T11:37:43.000Z

I will work on writing more documentation.
I will also look at adding a quality parameter.
Thank you!

Answer 27 · 2020-04-06T11:39:00.000Z

I will be watching it more and more, keep this amazing effort, and I will look to see it In future <3