This is the repository for the tools and network source codes for our paper [ExtraNet: Real-time Extrapolated Rendering for Low-latency Temporal Supersampling][ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia 2021)].
We provide a custom built Unreal Engine for generating dataset. Any Unreal Engine scenes requiring engine version <= 4.25.3 can be opened by our build. The link of our build: https://drive.google.com/drive/folders/1ZUbct6Z2T3gGhbxjQU0-u2PPflwYv-UZ?usp=sharing
Make sure to enable DX12 RHI and Ray tracing in Unreal settings, and enable stencil in project setting: project setting -> Rendering::Postprocessing -> Enable with Stencil.
For dynamic objects, make sure to enable Render Custom depth pass and set Custom Depth Stencil Value(any value larger than 0) in object properties.
This custom build is modified by Hengjun Ma.
We use Unreal Engine's sequence to generate data. More information about sequence and Sequencer, please refer to Unreal Engine documentation. For a recorded sequence, settings shown below are required before capturing frames:
Make sure to store frames from different sequences in different folders.
Raw data needs to be preprocessed before doing training or testing. We provide a script "preprocess.py" to generate training/testing dataset from raw data. The usage of script is:
python preprocess.py \some_path\raw_data_folder
For better efficiency of data loading, we also provide a script compressData.py for compressing data into .npz file. Usage: comming soon...
Currently our script has only been tested on Windows and accepts only "\" in file path. In addition, there are some settings (like thread number) can only be editted in script. We might commit some modification later to provide better user experience.
The training script is: Model/train.py.
python train.py
The testing script is: Model/inference.py.
python inference.py
https://github.com/fuxihao66/UnrealEngine/tree/UE425ExtraNet_active (still under development..)
(before accessing this repo, please refer to https://www.unrealengine.com/ue4-on-github to associate your GitHub account with your Unreal Engine account)
If you find the tools or codes useful in your research, please cite:
@article{10.1145/3478513.3480531,
author = {Guo, Jie and Fu, Xihao and Lin, Liqiang and Ma, Hengjun and Guo, Yanwen and Liu, Shiqiu and Yan, Ling-Qi},
title = {ExtraNet: Real-Time Extrapolated Rendering for Low-Latency Temporal Supersampling},
year = {2021},
issue_date = {December 2021},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
volume = {40},
number = {6},
doi = {10.1145/3478513.3480531},
journal = {ACM Trans. Graph.},
month = {dec},
articleno = {278},
numpages = {16}
}