WinSyn is a research project to provide a matched pair of synthetic and real images for machine learning tasks such as segmentation. You can learn more on the project page, reading the paper or watching a video showing how to run the procedural model.
This repository contains code for running the synthetic procedural model. We also took photos of 75k real windows.
We use Blender 3.3. Below we introduct options to run from a gui (a blender add-on; gui.py
), python script withing blender (go.py
), headless, & on a cluster.
- Open Blender 3.3.
- add
gui.py
as an add-on:- Edit > Preferences > Add-ons > install... then select
gui.py
. - Enable the WinSyn addon: ensure "community" is highlighted (blue) and search for "WinSyn". Click the checkbox next to "Research: WinSyn". Close the preferences window.
- The WinSyn tab should now be visible at the top-right of the 3D area. You may have to press the
n
key with your mouse over the 3D view to show the options (below the item and tool tabs). - You should now see the interface as above.
- Edit > Preferences > Add-ons > install... then select
- You have the options to
- Generate from a seed. Each different seed creates a different window.
- Generate from a file. The parameters from the file will be read to generate a building.
- Specify a parameter file.
- Edit the parameter file - edit and save (alt-s) the parameters in json format. Requires a text-editor pane be open (shift-f11).
- Set the resource path. By default this is the stripped down resource set in this repository, see below for instructions on how to download the full resource set.
- Set the styles/variation. These are used in the paper to explore different graphics techniques.
- Run the slow physics simulator or not
- The "generate" button will create your scene & write out the parameters to the given location
- Open the
winsyn.blend
file in Blender 3.3. - Open a text pane in blender with the
go.py
script - Optional: Set the
resource_path
in config.py to where you downloaded the resource files (see below) - Run the script! Generation time varies from 20 sections to a few minutes. Blender hangs during generation. Some generation may take a very long time depending on the parameters selected.
- Debugging requires a more challenging setup, I use Pycharm with something like this combined with the commented out
pydevd_pycharm.settrace
lines ingo.py
. The workflow goes something like - edit code in pycharm, switch to blender to run, switch back to pycharm to set breakpoints/inspect elements.
- as well as
resource_path
as above... - set the
render_path
in config.py to the location where renders should be written - set the number of renders you want in
render_number
. - set
interactive
to False in config.py. - optional: set the
style
(variations) inconfig.py
- run with something like (the CUDA bit says to use an Nvidia GPU to accelerate rendering):
blender -b /path/to/winsyn/winsyn.blend --python /path/to/winsyn/src/go.py -- --cycles-device CUDA
I deploy on our ibex/slurm cluster to render large datasets. I use the nytimes Blender docker image built as singularity container (singularity definition file) and a job script similar to the below. On ibex I rendered on the p100 and v100 nodes, and run about 10 machines to render 2k images overnight.
with a run.sh
script:
echo "Welcome to Windows"
outdir="/ibex/wherever/you/want/output_dataset"
mkdir -p $outdir
style="rgb;labels"
while : # "robustness"
do
SINGULARITYENV_WINDOWZ_STYLE="$1" singularity exec --nv --bind $outdir:/container/winsyn/output --bind /ibex/winsyn/resources:/container/winsyn/resources --bind /ibex/winsyn:/container/winsyn /ibex/wherever/you/put//blender_3_3.sif blender -b /container/winsyn/winsyn.blend --python /container/winsyn/src/go.py -- --cycles-device OPTIX
echo "blender crashed. let's try that again..."
done
with config.py
lines render_path=/container/winsyn/output
and resource_path=/container/winsyn/resources
.
The model requires a resources files with various textures and meshes from different sources. We include a single example resource of each type - these are enough to run the code, but do not have much diversity. Running the model with only these resources will not match our results... The config.py
file defines resource_path
which should be the location of the resources folder.
-
The 3D clutter scenes can be downloaded from the KAUST datastore. They should be added added to the
exterior_clutter
folder. -
The script
import_google_panos.py
can be used to download the panoramas used for the published dataset. It takes a single argument: your resource folder, and downloads images here in the subfolderoutside
.- Alternately, download a different set of panoramas from google directly.
-
Signs can be downloaded from the Kaust datastore. They should be in the
signs
folder of your resource folder. The downloaded and unzipped files can be split into folderslarge
,medium
, andsmall
using the scriptsplit_signs.py
. It takes two arguments - the root folder of the unzipped signs dataset and the resource folder. -
The interior textures are from matterport.
- You can download them by following the instructions (involving sending them a form and getting a download script)
- Run the script thusly to download all the interior panoramas:
download_mp.py /a/location --type matterport_skybox_images
-
extract and convert the downloaded skyboxes into the panoramic image format using the script
import_matterport.py
. It takes two arguments: the root of the downloaded panoramas and your resource folder. -
If you wish to generate the variant with many textures (
texture_rot
), download and unzip the dtd dataset into thedtd
folder inside your resource folder.
These are known as 'styles' in the code and change the behavior of the model (e.g., all-grey walls, or all-nighttime lighting - see the end of the paper's appendix for examples). They are defined in the config.py
file or using the WINDOWZ_STYLE
env variable. The sequences below render the variations for various sequences of paramters and create the labels where required.
rgb;labels
the default baseline model (and also render the labels).rgb;128nwall;64nwall;32nwall;16nwall;8nwall;4nwall;2nwall;1nwall;labels
changes the number of wall materials0monomat;0.33monomat;0.66monomat;1monomat;2monomat;4monomat;0multimat;0.33multimat;0.66multimat;1multimat;2multimat;4multimat;labels;all_brick
changes the parameterization of the procedural materials. monomat is a single proc material for each object class. multi-mat is the baseline number of materials. The numbers are multipliers on the deviations for parameter generation.nosplitz;nosplitz_labels;mono_profile;mono_profile_labels;only_rectangles;only_rectangles_labels;no_rectangles;no_rectangles_labels;only_squares;only_squares_labels;single_window;single_windows_labels;wide_windows;wide_windows_labels
the window-shape parameterization variation.lvl9;lvl8;lvl7;lvl6;lvl5;lvl4;lvl3;lvl2;lvl1;lvl9_labels;lvl8_labels;lvl7_labels;lvl6_labels;lvl5_labels;lvl4_labels;lvl3_labels;lvl2_labels;lvl1_labels
these are the number of modeled labels (i.e., just starting will thewall
label withlvl1
.0cen;3cen;6cen;12cen;24cen;48cen;96cen;labels;0cenlab;3cenlab;6cenlab;12cenlab;24cenlab;48cenlab;96cenlab
these are the camera positions (over a circle).1spp;2spp;4spp;8spp;16spp;32spp;64spp;128spp;256spp;512spp;nightonly;dayonly;notransmission;0cen;3cen;6cen;12cen;24cen;48cen;nosun;nobounce;fixedsun;monomat;labels;0cenlab;3cenlab;6cenlab;12cenlab;24cenlab;48cenlab
these are the rendering samples per pixel.canonical;64ms;128ms;256ms;512ms;1024ms;2048ms;labels;edges;diffuse;normals;col_per_obj;texture_rot;voronoi_chaos,phong_diffuse
these are the many varied materials experiments.
There are a variable number of parameters (sometimes thousands) depending on the code-path, and not all have human-friendly names. The parameters are described in the code samples which samples them from the RantomCache
class in rantom.py
:
r2.uniform(0.1, 0.22, "stucco_crack_size", "Size of stucco cracks")
The model writes out an attribute file to the attribs
directory containing all the parameters used to generate a given scene. The file also contains assorted metadata including the random seed and render times. You can vary the model's output by changing the parameters. By default a random seed is created and used to generate the remainder of the parameters.
After a parameter name has been assigned ("stucco_crack_size"
), asking for it again in the code will return the same value (even if it lies outside of the given distribution - invalid ranges ).
If you generate using the same random seed, it should always generate the same scene. However, changes in the code will change this, so consider creating a parameter list as below.
There is a mechanism to render a dataset using fixed seeds/parameters in headless/non-interactive mode. If there is a todo.txt
file in the config.render_path
, the system will try to render for each random seed (e.g., a number) in the file. One seed (a number) per line. There is a robustness mechanism for multi-node rendering, but I have observed occasional failures and had to re-run some seeds manually.
In addition, if there is an existing parameter (attribs) file for that seed (i.e., the file render_path/attribs/random_seed.txt
exists), those parameters will override the random values that would otherwise be used. Attributes that are required but not specified in this file are sampled as usual. This can be used to manually set the parameters for a derivation.
go.py
start reading here - the main loop (for step in range (config.render_number):
) runs until all renders have completed.config.py
contains the per-setup (for me this is laptop/desktop/cluster) configurationrantom.py
contains the paramter-sampling and storage codecgb.py
my CGAShape implementation. Very different extensions and limitations to other implementations :/cgb_building.py
the main entry point for actual geometry generation. Uses CGA to create a basic buildingcgb_*.py
the other major components which use CGA-list constructionsmaterials.py
responsible for adding materials to the scene's geometry, as well as all variations.pre_render
andpost_render
are the most interesting, withgo
as the entrypoint for the main texturing routine.shape.py
andsubframe.py
create bezier shaped windows and then add geometry to them
We thank the blender procedural artists Gabriel de Laubier for the UCP Wood material and Simon Thommes for the fantastic Br'cks material. Both were modified and used in our procedural model.
@inproceedings{winsyn,
title={WinSyn: A High Resolution Testbed for Synthetic Data},
author={Tom Kelly and John Femiani and Peter Wonka},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
month = {June},
year={2024}
}