[Feature Request] Support Programmatic Single Frame Capture

[dcyoung]

After exploring other options, I really like the implementation provided here.

However, the project seems constrained to sequence based (video like) workflows. Is there anyway to expose some of the internals for programmatic use?

Having simple library calls to provide a camera and trigger the capture of a single frame would open up a lot of use cases outside of sequence. For example, I'm looking to programmatically reset/perturb the same scene countless times over a range of parameters and capture an individual frame for each "run".

[NikolaJov96]

Hi @dcyoung,

Thanks for reaching out, here is why we went with the approach we did:

• We actually work with video-like image sequences, and the LevelSequence editor enables defining the camera trajectory in an intuitive and efficient way.
• LevelSequence enables defining the movement of other actors during rendering.
• LevelSequence is exact and deterministic with the timing and position of each rendered frame.
• We tried the approach using SceneCapture and individual capture triggering but were unable to achieve high FPS for rendering long sequences. This was due to the inexact timing of the SceneCapture movement and capture actions, requiring considerable pauses between frames.

Still, your suggestion sounds very useful, I'll look into adding that.
How would you interface with that functionality, through your own editor plugin?

[dcyoung]

@NikolaJov96 that makes total sense, and really streamlines the process building on top of the powerful tooling unreal provides.

Just hoping there might be an opportunity to support other use cases by exposing some of the internal mechanisms you likely built along the way. This is of course entirely dependent on how the src is architected.. perhaps there is a useful class or abstraction that could be made available, even if that’s just exposing certain utilities as a static library.

If the hypothetical exposed utilities could be orchestrated at runtime, then a simple level blueprint/scene component or managing actor could carry out the logic of establishing a scene, and resetting the scene between runs… triggering the capture for every run using the exposed utility. If it can’t be managed at runtime, then maybe this requires an additional editor plugin?

[4ndr3aR]

Thanks for the great plugin, I'm also very interested in a feature like this, especially to be able to avoid the editor altogether and be able to simply make captures in standalone applications by walking around with the camera. Ideally, what I would like to do is to build my level with the editor, close the editor (not to reopen it again), edit a csv file to have the regex<->color mappings of the segmentation masks, run my executable in standalone mode walking/flying around as if I were playing a video game, possibly logging the x,y,z coordinates in case in the future I wanted to exactly replicate the path but varying the parameters of my simulation (for example the atmospheric ones).

The regexes in the csv file would be used to match the names of the objects so that I don't have to select them all manually in the editor (maybe that's how the plugin is already working now and I might not have understood).

Oh, yes, I was forgetting the important part. This involves stopping the simulation after the first screenshot, changing all textures to semantic segmentation masks, re-rendering the same frame, re-replacing all textures with the originals, and unpausing the simulation. It might not be an easy thing.