A few ideas, FrameRate additions.
Closed this issue · 17 comments
Hi, this is a great project. Thanks for doing this! I've started integrating the framework into my project and everything I believe is working as intended. I did have a few things which I needed that I didn't see which I'll put below as some ideas. I'm not sure if I somehow overlooked these options, but I don't think so.
In particular I think a common format list is useful for displaying in a UI. Not very common to list 120 DF for example.
I have the situation where I don't know a frame rate until I open a video and all I've got from that is a floatValue. I'm not actually sure if there is a way to tell if it's drop frame or not? In any case, the floatValue init seems like I think I need in that case.
• Common Format List
• floatValue for FrameRate
• init FrameRate from a floatValue
If you're interested in these, I'd be happy to PR them in context, but this is what I'm using at the moment, Thanks!
Ryan
extension Timecode.FrameRate {
public static var commonFormats: [Timecode.FrameRate] {
[._23_976,
._24,
._25,
._29_97,
._29_97_drop,
._30,
._30_drop,
._50,
._59_94,
._59_94_drop,
._60,
._60_drop]
}
public var floatValue: Float {
switch self {
case ._23_976:
return 24.0 / 1.001
case ._24:
return 24.0
case ._24_98:
return 25.0 / 1.001
case ._25:
return 25.0
case ._29_97, ._29_97_drop, ._30_drop:
return 30.0 / 1.001
case ._30:
return 30.0
case ._47_952:
return 48.0 / 1.001
case ._48:
return 48.0
case ._50:
return 50.0
case ._59_94, ._59_94_drop, ._60_drop:
return 60.0 / 1.001
case ._60:
return 60.0
case ._100:
return 100.0
case ._120:
return 120.0
case ._120_drop, ._119_88, ._119_88_drop:
return 120.0 / 1.001
}
}
public var truncatedFloatValue: Float {
floatValue.truncated(decimalPlaces: 3)
}
public init?(floatValue: Float, restrictToCommonFormats: Bool = true, favorDropFrame: Bool = false) {
let collection = restrictToCommonFormats ? Self.commonFormats : Self.allCases
let findMatches = collection.filter {
$0.truncatedFloatValue == floatValue.truncated(decimalPlaces: 3)
}
// in cases where it's not clear which frame rate it is based off the floatValue
// we will have more than one match
if favorDropFrame, findMatches.count > 1 {
if let firstDrop = findMatches.first(where: {
$0.isDrop
}) {
self = firstDrop
return
}
}
if let firstMatch = findMatches.first {
self = firstMatch
} else {
return nil
}
}
}
Thanks for the interest! It took a few weeks of research and testing to put the library together and ensure its accuracy.
In particular I think a common format list is useful for displaying in a UI. Not very common to list 120 DF for example.
public static var commonFormats: [Timecode.FrameRate] { [ ... ] }
I feel a commonFormats property falls under subjective features. The developer adopting the library can determine what frame rates they want to use or not use, since what constitutes a "common format" is subjective, AFAIK. So you'd be free to have this property as an extension on Timecode.FrameRate as you've illustrated, but in your app.
Also, Avid's Pro Tools shows 119.88/119.88d/120/120d in its list of project frame rates now, so they may become a bit more common as other software suites follow their lead.
That said, I would alter your method to be:
public init?(floatValue: Float, restrictTo: [FrameRate]?, favorDropFrame: Bool = false)and then just pass in your own commonFormats array to the method whenever you call it:
FrameRate(floatValue: x, restrictTo: FrameRate.commonFormats, favorDropFrame: true)and if restrictTo is nil, just have the method internally use .allCases as default.
I have the situation where I don't know a frame rate until I open a video and all I've got from that is a floatValue. I'm not actually sure if there is a way to tell if it's drop frame or not?
I'm not sure I fully understand. Whatever API you're using to read/open video files is only supplying float value of the frame rate? The question about drop or non-drop is tricky - is it designed to be deterministic? The way you've implemented definition and discovery of float value seems heuristic in nature. For example, would 29.97 vs 29.97d produce different float values? At the very least I'd check the documentation of the video API you're using, and I'd want to externally generate/acquire video files in all frame rates (known frame rate) then see what your API produces as float values to compare.
I also found that 30df/60df/120df are not treated as actual drop-frame rates in most contexts, so on cursory glance I'm not sure your code is correct.
Is float value a standard in video libraries? Or is it just endemic to the one you're using? That may gauge the viability of including a float value in TimecodeKit. I just want to understand it first and avoid including novel features where they may not be appropriate for most contexts.
But now you've got me curious so I'd love to see this figured out!
oh i didn't notice ProTools added the higher frame rates. Disregard my comment in that regard then!
if i use common formats, then yes that makes sense to make that an application level thing and doesn't need to be in your API.
About the Float float frame rate:
I'm using Apple's AVPlayer for video and that will report a video's frame rate as a Float, yeah. In my particular application, this value isn't super telling given something like this. These FrameRate's of yours all have the same numerical value:
case ._29_97, ._29_97_drop, ._30_drop:
return 30.0 / 1.001
case ._59_94, ._59_94_drop, ._60_drop:
return 60.0 / 1.001
case ._120_drop, ._119_88, ._119_88_drop:
return 120.0 / 1.001
So in that case, how would you know which should be the specific Timecode.FrameRate - or can you? Or, did I miss something? As far as I can tell, all you can get from the AVPlayer is the AVAssetTrack.nominalFrameRate, which is a Float.
/**
@property nominalFrameRate
@abstract For tracks that carry a full frame per media sample, indicates the frame rate of the track in units of frames per second.
@discussion For field-based video tracks that carry one field per media sample, the value of this property is the field rate, not the frame rate.
*/
open var nominalFrameRate: Float { get }
So what I'm doing at the moment is limiting the frame rate selection to the ones I'm sure have the approximated base floatValue, and the user could select drop or non-drop. I'm unsure if it's possible to really know more than that in that case. Here's a screenshot of what that looks like:
For the particular video, encoded in Premiere, I encoded as 29.97 - premiere doesn't offer any info on export beyond that, but it does export as 29.97 DF. That's the only export preset that exports DF.
So, opening that video in AVPlayer says the frame rate is "29.970032".
Any advice?
If curious, here's some frame rate selectors I had a look at to see how others are labeling these.
Premiere 2020
Logic
Final Cut Pro
ProTools
I'm using Apple's AVPlayer for video
Ok, I figured it was 1st-party API. I wouldn't mind playing around with AVPlayer and seeing what it's doing.
These FrameRate's of yours all have the same numerical value
case ._29_97, ._29_97_drop, ._30_drop: return 30.0 / 1.001 case ._59_94, ._59_94_drop, ._60_drop: return 60.0 / 1.001 case ._120_drop, ._119_88, ._119_88_drop: return 120.0 / 1.001
This is your code, not the library. Internally in the library, these are calculated differently and are not quite parallelized or reduced in this way. 30d/60d/120d are calculated differently even though they are labelled as a "drop" frame rate. You'll notice in Logic's frame rate selection list, these rates are in bold and italics to show they are not traditional frame rates but actually modern adaptation rates.
Here's the issue as I see it: The reason why boiling video encoding frame rates down to a float value is obscure is that drop rates technically have fewer "frames" but as you likely know, only certain frames are dropped. What you would end up with would be an average, if it were a float value. But it does not mean that there are that many frames per second in any given second of time. My guess is that AVPlayer may produce a float value that averages the per-second frame count across an entire hour or something, and then flatten it out to a "per-second" frame rate float value. Which would be kind of stupid, but possible I suppose.
Since I'm not familiar with AVPlayer, from a quick Google search I'm guessing nominalFrameRate has to do with the overall video/GPU playback rate which may be misleading. I did read that AVAssetReader can possibly determine media's encoding frame rate with better precision. Have you looked into that?
As an aside, in retrospect I think I will update the Timecode.FrameRate.stringValue property and add a verbose version so it's more flexible for use in human-readable UI.
I'll update stringValue to produce short Logic-style strings. Then add another property called stringValueVerbose with longer style strings like Pro Tools. Although ultimately this is in the library as a convenience and this nomenclature is up to the developer if it doesn't precisely suit their needs; they can set up their own strings in their app code.
ie:
/// Returns human-readable frame rate string.
public var stringValue: String {
switch self {
...
case ._29_97: return "29.97"
case ._29_97_drop: return "29.97d"
...
}
}
/// Returns human-readable frame rate string in long form.
public var stringValueVerbose: String {
switch self {
...
case ._29_97: return "29.97 fps"
case ._29_97_drop: return "29.97 fps drop"
...
}I played around with AVFoundation a bit and the only mechanism I could find for reading frame rate was the nominalFrameRate property as you mentioned.
Also keep in mind that this value can be an arbitrary value that doesn't correspond to a concrete frame rate, if a video file is encoded as Variable frame rate. For example, I did a screen recording video and it was encoded as AVC Variable rate. When read by AVAssetTrack, it returns 55.473682 as the nominalFrameRate value. Not sure how you want to handle that, but just an FYI in case you haven't run into that yet.
I'll keep digging and see if there's a reasonable solution to the drop/nondrop float value issue.
I have a video file known to be 29.97d and I ran a few tests.
File meta data's frame rate is shown as "29.970 FPS" as read in Invisor.app
Also, nominalFrameRate is read as exactly float value 29.97.
So from that, you cannot know whether timecode display should be drop or non-drop. Essentially, the user has to specify it.
So from that, you cannot know whether timecode display should be drop or non-drop. Essentially, the user has to specify it.
Right yeah. that was what I had decided as well. Which isn't really such a big deal, but good to know you came to the same conclusion.
As an aside, in retrospect I think I will update the
Timecode.FrameRate.stringValueproperty and add a verbose version so it's more flexible for use in human-readable UI.
oh that's good. Actually I added this same idea as well like this:
/// Returns human-readable frame rate string.
public var stringValueUILabel: String {
switch self {
case ._23_976: return "23.976"
case ._24: return "24"
case ._24_98: return "24.98"
case ._25: return "25"
case ._29_97: return "29.97"
case ._29_97_drop: return "29.97 Drop"
case ._30: return "30"
case ._30_drop: return "30 Drop"
case ._47_952: return "47.952"
case ._48: return "48"
case ._50: return "50"
case ._59_94: return "59.94"
case ._59_94_drop: return "59.94 Drop"
case ._60: return "60"
case ._60_drop: return "60 Drop"
case ._100: return "100"
case ._119_88: return "119.88"
case ._119_88_drop: return "119.88 Drop"
case ._120: return "120"
case ._120_drop: return "120 Drop"
}
}
Definitely seeing all those DF and NDF's isn't the best look in a selector.
I think the conclusion is that float value is not a standard value for timecode expression frame rate, so adding a floatValue property or init to the library will be ambiguous given what we know here.
This is your code, not the library. Internally in the library, these are calculated differently and are not quite parallelized or reduced in this way. 30d/60d/120d are calculated differently even though they are labelled as a "drop" frame rate. You'll notice in Logic's frame rate selection list, these rates are in bold and italics to show they are not traditional frame rates but actually modern adaptation rates.
oh ok. actually i was looking at :
frameRateForRealTimeCalculation
In that switch the values are as I have them. I am doing a lot of realtime conversion myself as I'm deriving the timecode from a seconds value. Is there another way to do that?
I think the conclusion is that float value is not a standard value for timecode expression frame rate, so adding a floatValue property or init to the library will be ambiguous given what we know here.
that's fine. though, regardless of that, I still have to do it. But it doesn't need to be in your API, I just wanted to bring it up as something I'm looking at. In all videos I've tested the rates are close enough - but that is why in my init for floatValue I'm just taking a truncated version and setting the selector to the possible rates. Given that, it doesn't have to be a 100% confident result, but limiting the frame rate in this way gets it at least to the rates that matter.
I hope.
I am doing a lot of realtime conversion myself as I'm deriving the timecode from a seconds value. Is there another way to do that?
Assuming you are supplied a seconds float for current video playback position, this is the most obvious way:
// x == position in seconds
// fr == FrameRate
Timecode(TimeValue(seconds: x), at: fr, limit: ._24hours)Timecode(TimeValue(seconds: x), at: fr, limit: ._24hours)
cool, that's essentially what I did though am using the toTimecode convenience.
I'll push the stringValue / stringValueVerbose update shortly.
Also while I'm at it, I will flesh out the README.md with some basic code examples since I've been meaning to get around to it.
And thanks again for bringing up the float fps issue - something I hadn't dug into before and it was a good consideration to explore.
If you have any other ideas for improvements or additions, feel free to post more Issue tickets.
cool, thanks!
@ryanfrancesconi pushed release version 1.0.3 to main with updates.