FILTER_VALUE>1 produce issues in make_dataset.py
CameronBodine opened this issue · 6 comments
CameronBodine commented
Describe the bug
When I specify a FILTER_VALUE != 1 (i.e. FILTER_VALUE=2) while running make_dataset.py (using current version), the examples saved in Output/train_data/train_npzs/noaug_sample only show one class when there should be up to 8, as shown below.
To Reproduce
Steps to reproduce the behavior:
- Specify a value
FILTER_VALUE=2inconfig.json
Click me to see complete config
{
"TARGET_SIZE": [
512,
512
],
"MODEL": "segformer",
"NCLASSES": 8,
"BATCH_SIZE": 30,
"N_DATA_BANDS": 1,
"DO_TRAIN": true,
"PATIENCE": 10,
"MAX_EPOCHS": 10,
"VALIDATION_SPLIT": 0.6,
"FILTERS": 6,
"KERNEL": 7,
"STRIDE": 2,
"LOSS": "dice",
"DROPOUT": 0.1,
"DROPOUT_CHANGE_PER_LAYER": 0.0,
"DROPOUT_TYPE": "standard",
"USE_DROPOUT_ON_UPSAMPLING": false,
"ROOT_STRING": "EGN_Substrate_inclShadow",
"FILTER_VALUE": 2,
"DOPLOT": true,
"USEMASK": false,
"RAMPUP_EPOCHS": 10,
"SUSTAIN_EPOCHS": 0.0,
"EXP_DECAY": 0.9,
"START_LR": 1e-07,
"MIN_LR": 1e-07,
"MAX_LR": 0.0001,
"AUG_ROT": 0,
"AUG_ZOOM": 0.05,
"AUG_WIDTHSHIFT": 0.05,
"AUG_HEIGHTSHIFT": 0.05,
"AUG_HFLIP": true,
"AUG_VFLIP": false,
"AUG_LOOPS": 3,
"AUG_COPIES": 3,
"TESTTIMEAUG": false,
"SET_GPU": "0",
"DO_CRF": false,
"SET_PCI_BUS_ID": true,
"WRITE_MODELMETADATA": true,
"OTSU_THRESHOLD": true,
"REMAP_CLASSES": {
"0": 0,
"1": 1,
"2": 2,
"3": 3,
"4": 4,
"5": 5,
"6": 6,
"7": 7,
"8": 0
}
}
Expected behavior
I expect to see all the classes, as in when I specify FILTER_VALUE=1 as shown below.
Desktop (please complete the following information):
- OS: Ubuntu
Click me to see `conda list`
(gym) cbodine@filfy-Thelio-Massive:~/PythonRepos/segmentation_gym$ conda list
# packages in environment at /home/cbodine/miniconda3/envs/gym:
#
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
absl-py 1.4.0 pypi_0 pypi
alsa-lib 1.2.8 h166bdaf_0 conda-forge
anyio 3.7.0 pyhd8ed1ab_0 conda-forge
aom 3.5.0 h27087fc_0 conda-forge
argon2-cffi 21.3.0 pyhd8ed1ab_0 conda-forge
argon2-cffi-bindings 21.2.0 py311hd4cff14_3 conda-forge
asttokens 2.2.1 pyhd8ed1ab_0 conda-forge
astunparse 1.6.3 pypi_0 pypi
attr 2.5.1 h166bdaf_1 conda-forge
attrs 23.1.0 pyh71513ae_1 conda-forge
backcall 0.2.0 pyh9f0ad1d_0 conda-forge
backports 1.0 pyhd8ed1ab_3 conda-forge
backports.functools_lru_cache 1.6.4 pyhd8ed1ab_0 conda-forge
beautifulsoup4 4.12.2 pyha770c72_0 conda-forge
bleach 6.0.0 pyhd8ed1ab_0 conda-forge
blosc 1.21.4 h0f2a231_0 conda-forge
brotli 1.0.9 h166bdaf_8 conda-forge
brotli-bin 1.0.9 h166bdaf_8 conda-forge
brunsli 0.1 h9c3ff4c_0 conda-forge
bzip2 1.0.8 h7f98852_4 conda-forge
c-ares 1.19.1 hd590300_0 conda-forge
c-blosc2 2.9.2 hb4ffafa_0 conda-forge
ca-certificates 2023.5.7 hbcca054_0 conda-forge
cachetools 5.3.1 pypi_0 pypi
cairo 1.16.0 hbbf8b49_1016 conda-forge
certifi 2023.5.7 pyhd8ed1ab_0 conda-forge
cffi 1.15.1 py311h409f033_3 conda-forge
cfitsio 4.2.0 hd9d235c_0 conda-forge
charls 2.4.2 h59595ed_0 conda-forge
charset-normalizer 3.1.0 pyhd8ed1ab_0 conda-forge
click 8.1.3 unix_pyhd8ed1ab_2 conda-forge
cloudpickle 2.2.1 pyhd8ed1ab_0 conda-forge
colorama 0.4.6 pyhd8ed1ab_0 conda-forge
comm 0.1.3 pyhd8ed1ab_0 conda-forge
contourpy 1.0.7 py311ha3edf6b_0 conda-forge
cuda 12.1.1 0 nvidia
cuda-cccl 12.1.109 0 nvidia
cuda-command-line-tools 12.1.1 0 nvidia
cuda-compiler 12.1.1 0 nvidia
cuda-cudart 12.1.105 0 nvidia
cuda-cudart-dev 12.1.105 0 nvidia
cuda-cudart-static 12.1.105 0 nvidia
cuda-cuobjdump 12.1.111 0 nvidia
cuda-cupti 12.1.105 0 nvidia
cuda-cupti-static 12.1.105 0 nvidia
cuda-cuxxfilt 12.1.105 0 nvidia
cuda-demo-suite 12.1.105 0 nvidia
cuda-documentation 12.1.105 0 nvidia
cuda-driver-dev 12.1.105 0 nvidia
cuda-gdb 12.1.105 0 nvidia
cuda-libraries 12.1.1 0 nvidia
cuda-libraries-dev 12.1.1 0 nvidia
cuda-libraries-static 12.1.1 0 nvidia
cuda-nsight 12.1.105 0 nvidia
cuda-nsight-compute 12.1.1 0 nvidia
cuda-nvcc 12.1.105 0 nvidia
cuda-nvdisasm 12.1.105 0 nvidia
cuda-nvml-dev 12.1.105 0 nvidia
cuda-nvprof 12.1.105 0 nvidia
cuda-nvprune 12.1.105 0 nvidia
cuda-nvrtc 12.1.105 0 nvidia
cuda-nvrtc-dev 12.1.105 0 nvidia
cuda-nvrtc-static 12.1.105 0 nvidia
cuda-nvtx 12.1.105 0 nvidia
cuda-nvvp 12.1.105 0 nvidia
cuda-opencl 12.1.105 0 nvidia
cuda-opencl-dev 12.1.105 0 nvidia
cuda-profiler-api 12.1.105 0 nvidia
cuda-runtime 12.1.1 0 nvidia
cuda-sanitizer-api 12.1.105 0 nvidia
cuda-toolkit 12.1.1 0 nvidia
cuda-tools 12.1.1 0 nvidia
cuda-visual-tools 12.1.1 0 nvidia
cudatoolkit 11.8.0 h37601d7_11 conda-forge
cycler 0.11.0 pyhd8ed1ab_0 conda-forge
cython 0.29.35 py311hb755f60_0 conda-forge
cytoolz 0.12.0 py311hd4cff14_1 conda-forge
dask-core 2023.5.1 pyhd8ed1ab_0 conda-forge
dav1d 1.2.0 hd590300_0 conda-forge
dbus 1.13.6 h5008d03_3 conda-forge
debugpy 1.6.7 py311hcafe171_0 conda-forge
decorator 5.1.1 pyhd8ed1ab_0 conda-forge
defusedxml 0.7.1 pyhd8ed1ab_0 conda-forge
doodleverse-utils 0.0.30 pypi_0 pypi
entrypoints 0.4 pyhd8ed1ab_0 conda-forge
executing 1.2.0 pyhd8ed1ab_0 conda-forge
expat 2.5.0 hcb278e6_1 conda-forge
filelock 3.12.0 pypi_0 pypi
flatbuffers 23.5.26 pypi_0 pypi
flit-core 3.9.0 pyhd8ed1ab_0 conda-forge
font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge
font-ttf-inconsolata 3.000 h77eed37_0 conda-forge
font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge
font-ttf-ubuntu 0.83 hab24e00_0 conda-forge
fontconfig 2.14.2 h14ed4e7_0 conda-forge
fonts-conda-ecosystem 1 0 conda-forge
fonts-conda-forge 1 0 conda-forge
fonttools 4.39.4 py311h459d7ec_0 conda-forge
freetype 2.12.1 hca18f0e_1 conda-forge
fsspec 2023.5.0 pyh1a96a4e_0 conda-forge
gast 0.4.0 pypi_0 pypi
gds-tools 1.6.1.9 0 nvidia
gettext 0.21.1 h27087fc_0 conda-forge
giflib 5.2.1 h0b41bf4_3 conda-forge
glib 2.76.3 hfc55251_0 conda-forge
glib-tools 2.76.3 hfc55251_0 conda-forge
google-auth 2.19.0 pypi_0 pypi
google-auth-oauthlib 1.0.0 pypi_0 pypi
google-pasta 0.2.0 pypi_0 pypi
graphite2 1.3.13 h58526e2_1001 conda-forge
grpcio 1.54.2 pypi_0 pypi
gst-plugins-base 1.22.3 h938bd60_1 conda-forge
gstreamer 1.22.3 h977cf35_1 conda-forge
h5py 3.8.0 pypi_0 pypi
harfbuzz 7.3.0 hdb3a94d_0 conda-forge
huggingface-hub 0.14.1 pypi_0 pypi
icu 72.1 hcb278e6_0 conda-forge
idna 3.4 pyhd8ed1ab_0 conda-forge
imagecodecs 2023.1.23 py311hd374d05_2 conda-forge
imageio 2.28.1 pyh24c5eb1_0 conda-forge
importlib-metadata 6.6.0 pyha770c72_0 conda-forge
importlib_metadata 6.6.0 hd8ed1ab_0 conda-forge
importlib_resources 5.12.0 pyhd8ed1ab_0 conda-forge
ipykernel 6.23.1 pyh210e3f2_0 conda-forge
ipython 8.13.2 pyh41d4057_0 conda-forge
ipython_genutils 0.2.0 py_1 conda-forge
ipywidgets 8.0.6 pyhd8ed1ab_0 conda-forge
jax 0.4.10 pypi_0 pypi
jedi 0.18.2 pyhd8ed1ab_0 conda-forge
jinja2 3.1.2 pyhd8ed1ab_1 conda-forge
joblib 1.2.0 pyhd8ed1ab_0 conda-forge
jsonschema 4.17.3 pyhd8ed1ab_0 conda-forge
jupyter 1.0.0 py311h38be061_8 conda-forge
jupyter_client 8.2.0 pyhd8ed1ab_0 conda-forge
jupyter_console 6.6.3 pyhd8ed1ab_0 conda-forge
jupyter_core 5.3.0 py311h38be061_0 conda-forge
jupyter_events 0.6.3 pyhd8ed1ab_0 conda-forge
jupyter_server 2.6.0 pyhd8ed1ab_0 conda-forge
jupyter_server_terminals 0.4.4 pyhd8ed1ab_1 conda-forge
jupyterlab_pygments 0.2.2 pyhd8ed1ab_0 conda-forge
jupyterlab_widgets 3.0.7 pyhd8ed1ab_1 conda-forge
jxrlib 1.1 h7f98852_2 conda-forge
keras 2.12.0 pypi_0 pypi
keyutils 1.6.1 h166bdaf_0 conda-forge
kiwisolver 1.4.4 py311h4dd048b_1 conda-forge
krb5 1.20.1 h81ceb04_0 conda-forge
lame 3.100 h166bdaf_1003 conda-forge
lazy_loader 0.2 pyhd8ed1ab_0 conda-forge
lcms2 2.15 haa2dc70_1 conda-forge
ld_impl_linux-64 2.40 h41732ed_0 conda-forge
lerc 4.0.0 h27087fc_0 conda-forge
libaec 1.0.6 hcb278e6_1 conda-forge
libavif 0.11.1 hff004cb_1 conda-forge
libblas 3.9.0 16_linux64_openblas conda-forge
libbrotlicommon 1.0.9 h166bdaf_8 conda-forge
libbrotlidec 1.0.9 h166bdaf_8 conda-forge
libbrotlienc 1.0.9 h166bdaf_8 conda-forge
libcap 2.67 he9d0100_0 conda-forge
libcblas 3.9.0 16_linux64_openblas conda-forge
libclang 16.0.0 pypi_0 pypi
libclang13 15.0.7 default_h9986a30_2 conda-forge
libcublas 12.1.3.1 0 nvidia
libcublas-dev 12.1.3.1 0 nvidia
libcublas-static 12.1.3.1 0 nvidia
libcufft 11.0.2.54 0 nvidia
libcufft-dev 11.0.2.54 0 nvidia
libcufft-static 11.0.2.54 0 nvidia
libcufile 1.6.1.9 0 nvidia
libcufile-dev 1.6.1.9 0 nvidia
libcufile-static 1.6.1.9 0 nvidia
libcups 2.3.3 h36d4200_3 conda-forge
libcurand 10.3.2.106 0 nvidia
libcurand-dev 10.3.2.106 0 nvidia
libcurand-static 10.3.2.106 0 nvidia
libcurl 8.1.2 h409715c_0 conda-forge
libcusolver 11.4.5.107 0 nvidia
libcusolver-dev 11.4.5.107 0 nvidia
libcusolver-static 11.4.5.107 0 nvidia
libcusparse 12.1.0.106 0 nvidia
libcusparse-dev 12.1.0.106 0 nvidia
libcusparse-static 12.1.0.106 0 nvidia
libdeflate 1.18 h0b41bf4_0 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 h516909a_1 conda-forge
libevent 2.1.12 h3358134_0 conda-forge
libexpat 2.5.0 hcb278e6_1 conda-forge
libffi 3.4.2 h7f98852_5 conda-forge
libflac 1.4.2 h27087fc_0 conda-forge
libgcc-ng 12.2.0 h65d4601_19 conda-forge
libgcrypt 1.10.1 h166bdaf_0 conda-forge
libgfortran-ng 12.2.0 h69a702a_19 conda-forge
libgfortran5 12.2.0 h337968e_19 conda-forge
libglib 2.76.3 hebfc3b9_0 conda-forge
libgomp 12.2.0 h65d4601_19 conda-forge
libgpg-error 1.46 h620e276_0 conda-forge
libiconv 1.17 h166bdaf_0 conda-forge
libjpeg-turbo 2.1.5.1 h0b41bf4_0 conda-forge
liblapack 3.9.0 16_linux64_openblas conda-forge
libllvm15 15.0.7 h5cf9203_2 conda-forge
libnghttp2 1.52.0 h61bc06f_0 conda-forge
libnpp 12.1.0.40 0 nvidia
libnpp-dev 12.1.0.40 0 nvidia
libnpp-static 12.1.0.40 0 nvidia
libnsl 2.0.0 h7f98852_0 conda-forge
libnvjitlink 12.1.105 0 nvidia
libnvjitlink-dev 12.1.105 0 nvidia
libnvjpeg 12.2.0.2 0 nvidia
libnvjpeg-dev 12.2.0.2 0 nvidia
libnvjpeg-static 12.2.0.2 0 nvidia
libnvvm-samples 12.1.105 0 nvidia
libogg 1.3.4 h7f98852_1 conda-forge
libopenblas 0.3.21 pthreads_h78a6416_3 conda-forge
libopus 1.3.1 h7f98852_1 conda-forge
libpng 1.6.39 h753d276_0 conda-forge
libpq 15.3 hbcd7760_1 conda-forge
libsndfile 1.2.0 hb75c966_0 conda-forge
libsodium 1.0.18 h36c2ea0_1 conda-forge
libsqlite 3.42.0 h2797004_0 conda-forge
libssh2 1.10.0 hf14f497_3 conda-forge
libstdcxx-ng 12.2.0 h46fd767_19 conda-forge
libsystemd0 253 h8c4010b_1 conda-forge
libtiff 4.5.0 ha587672_6 conda-forge
libuuid 2.38.1 h0b41bf4_0 conda-forge
libvorbis 1.3.7 h9c3ff4c_0 conda-forge
libwebp-base 1.3.0 h0b41bf4_0 conda-forge
libxcb 1.15 h0b41bf4_0 conda-forge
libxkbcommon 1.5.0 h5d7e998_3 conda-forge
libxml2 2.11.4 h0d562d8_0 conda-forge
libzlib 1.2.13 h166bdaf_4 conda-forge
libzopfli 1.0.3 h9c3ff4c_0 conda-forge
locket 1.0.0 pyhd8ed1ab_0 conda-forge
lz4-c 1.9.4 hcb278e6_0 conda-forge
markdown 3.4.3 pypi_0 pypi
markupsafe 2.1.2 py311h2582759_0 conda-forge
matplotlib 3.7.1 py311h38be061_0 conda-forge
matplotlib-base 3.7.1 py311h8597a09_0 conda-forge
matplotlib-inline 0.1.6 pyhd8ed1ab_0 conda-forge
mistune 2.0.5 pyhd8ed1ab_0 conda-forge
ml-dtypes 0.1.0 pypi_0 pypi
mpg123 1.31.3 hcb278e6_0 conda-forge
munkres 1.1.4 pyh9f0ad1d_0 conda-forge
mysql-common 8.0.32 hf1915f5_2 conda-forge
mysql-libs 8.0.32 hca2cd23_2 conda-forge
natsort 8.3.1 pyhd8ed1ab_0 conda-forge
nbclassic 1.0.0 pyhb4ecaf3_1 conda-forge
nbclient 0.8.0 pyhd8ed1ab_0 conda-forge
nbconvert 7.4.0 pyhd8ed1ab_0 conda-forge
nbconvert-core 7.4.0 pyhd8ed1ab_0 conda-forge
nbconvert-pandoc 7.4.0 pyhd8ed1ab_0 conda-forge
nbformat 5.8.0 pyhd8ed1ab_0 conda-forge
ncurses 6.3 h27087fc_1 conda-forge
nest-asyncio 1.5.6 pyhd8ed1ab_0 conda-forge
networkx 3.1 pyhd8ed1ab_0 conda-forge
notebook 6.5.4 pyha770c72_0 conda-forge
notebook-shim 0.2.3 pyhd8ed1ab_0 conda-forge
nsight-compute 2023.1.1.4 0 nvidia
nspr 4.35 h27087fc_0 conda-forge
nss 3.89 he45b914_0 conda-forge
numpy 1.23.5 pypi_0 pypi
nvidia-cublas-cu11 11.11.3.6 pypi_0 pypi
nvidia-cudnn-cu11 8.6.0.163 pypi_0 pypi
oauthlib 3.2.2 pypi_0 pypi
openjpeg 2.5.0 hfec8fc6_2 conda-forge
openssl 3.1.0 hd590300_3 conda-forge
opt-einsum 3.3.0 pypi_0 pypi
overrides 7.3.1 pyhd8ed1ab_0 conda-forge
packaging 23.1 pyhd8ed1ab_0 conda-forge
pandas 2.0.2 py311h320fe9a_0 conda-forge
pandoc 2.19.2 h32600fe_2 conda-forge
pandocfilters 1.5.0 pyhd8ed1ab_0 conda-forge
parso 0.8.3 pyhd8ed1ab_0 conda-forge
partd 1.4.0 pyhd8ed1ab_0 conda-forge
pcre2 10.40 hc3806b6_0 conda-forge
pexpect 4.8.0 pyh1a96a4e_2 conda-forge
pickleshare 0.7.5 py_1003 conda-forge
pillow 9.5.0 py311h0b84326_1 conda-forge
pip 23.1.2 pyhd8ed1ab_0 conda-forge
pixman 0.40.0 h36c2ea0_0 conda-forge
pkgutil-resolve-name 1.3.10 pyhd8ed1ab_0 conda-forge
platformdirs 3.5.1 pyhd8ed1ab_0 conda-forge
plotly 5.14.1 pyhd8ed1ab_0 conda-forge
ply 3.11 py_1 conda-forge
pooch 1.7.0 pyha770c72_3 conda-forge
prometheus_client 0.17.0 pyhd8ed1ab_0 conda-forge
prompt-toolkit 3.0.38 pyha770c72_0 conda-forge
prompt_toolkit 3.0.38 hd8ed1ab_0 conda-forge
protobuf 4.23.2 pypi_0 pypi
psutil 5.9.5 py311h2582759_0 conda-forge
pthread-stubs 0.4 h36c2ea0_1001 conda-forge
ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge
pulseaudio-client 16.1 hb77b528_4 conda-forge
pure_eval 0.2.2 pyhd8ed1ab_0 conda-forge
pyasn1 0.5.0 pypi_0 pypi
pyasn1-modules 0.3.0 pypi_0 pypi
pycparser 2.21 pyhd8ed1ab_0 conda-forge
pygments 2.15.1 pyhd8ed1ab_0 conda-forge
pyparsing 3.0.9 pyhd8ed1ab_0 conda-forge
pyqt 5.15.7 py311ha74522f_3 conda-forge
pyqt5-sip 12.11.0 py311hcafe171_3 conda-forge
pyrsistent 0.19.3 py311h2582759_0 conda-forge
pysocks 1.7.1 pyha2e5f31_6 conda-forge
python 3.11.3 h2755cc3_0_cpython conda-forge
python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge
python-fastjsonschema 2.17.1 pyhd8ed1ab_0 conda-forge
python-json-logger 2.0.7 pyhd8ed1ab_0 conda-forge
python-tzdata 2023.3 pyhd8ed1ab_0 conda-forge
python_abi 3.11 3_cp311 conda-forge
pytz 2023.3 pyhd8ed1ab_0 conda-forge
pywavelets 1.4.1 py311hcb2cf0a_0 conda-forge
pyyaml 6.0 py311hd4cff14_5 conda-forge
pyzmq 25.0.2 py311hd6ccaeb_0 conda-forge
qt-main 5.15.8 h01ceb2d_13 conda-forge
qtconsole 5.4.3 pyhd8ed1ab_0 conda-forge
qtconsole-base 5.4.3 pyha770c72_0 conda-forge
qtpy 2.3.1 pyhd8ed1ab_0 conda-forge
readline 8.2 h8228510_1 conda-forge
regex 2023.5.5 pypi_0 pypi
requests 2.31.0 pyhd8ed1ab_0 conda-forge
requests-oauthlib 1.3.1 pypi_0 pypi
rfc3339-validator 0.1.4 pyhd8ed1ab_0 conda-forge
rfc3986-validator 0.1.1 pyh9f0ad1d_0 conda-forge
rsa 4.9 pypi_0 pypi
scikit-image 0.20.0 py311h2872171_1 conda-forge
scipy 1.10.1 py311h64a7726_3 conda-forge
send2trash 1.8.2 pyh41d4057_0 conda-forge
setuptools 67.7.2 pyhd8ed1ab_0 conda-forge
sip 6.7.9 py311hb755f60_0 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
snappy 1.1.10 h9fff704_0 conda-forge
sniffio 1.3.0 pyhd8ed1ab_0 conda-forge
soupsieve 2.3.2.post1 pyhd8ed1ab_0 conda-forge
stack_data 0.6.2 pyhd8ed1ab_0 conda-forge
tenacity 8.2.2 pyhd8ed1ab_0 conda-forge
tensorboard 2.12.3 pypi_0 pypi
tensorboard-data-server 0.7.0 pypi_0 pypi
tensorflow 2.12.0 pypi_0 pypi
tensorflow-estimator 2.12.0 pypi_0 pypi
tensorflow-io-gcs-filesystem 0.32.0 pypi_0 pypi
termcolor 2.3.0 pypi_0 pypi
terminado 0.17.1 pyh41d4057_0 conda-forge
tifffile 2023.4.12 pyhd8ed1ab_0 conda-forge
tinycss2 1.2.1 pyhd8ed1ab_0 conda-forge
tk 8.6.12 h27826a3_0 conda-forge
tokenizers 0.13.3 pypi_0 pypi
toml 0.10.2 pyhd8ed1ab_0 conda-forge
tomli 2.0.1 pyhd8ed1ab_0 conda-forge
toolz 0.12.0 pyhd8ed1ab_0 conda-forge
tornado 6.3.2 py311h459d7ec_0 conda-forge
tqdm 4.65.0 pyhd8ed1ab_1 conda-forge
traitlets 5.9.0 pyhd8ed1ab_0 conda-forge
transformers 4.29.2 pypi_0 pypi
typing-extensions 4.6.2 hd8ed1ab_0 conda-forge
typing_extensions 4.6.2 pyha770c72_0 conda-forge
typing_utils 0.1.0 pyhd8ed1ab_0 conda-forge
tzdata 2023c h71feb2d_0 conda-forge
urllib3 1.26.16 pypi_0 pypi
versioneer 0.28 pypi_0 pypi
wcwidth 0.2.6 pyhd8ed1ab_0 conda-forge
webencodings 0.5.1 py_1 conda-forge
websocket-client 1.5.2 pyhd8ed1ab_0 conda-forge
werkzeug 2.3.4 pypi_0 pypi
wheel 0.40.0 pyhd8ed1ab_0 conda-forge
widgetsnbextension 4.0.7 pyhd8ed1ab_0 conda-forge
wrapt 1.14.1 pypi_0 pypi
xcb-util 0.4.0 hd590300_1 conda-forge
xcb-util-image 0.4.0 h8ee46fc_1 conda-forge
xcb-util-keysyms 0.4.0 h8ee46fc_1 conda-forge
xcb-util-renderutil 0.3.9 hd590300_1 conda-forge
xcb-util-wm 0.4.1 h8ee46fc_1 conda-forge
xkeyboard-config 2.38 h0b41bf4_0 conda-forge
xorg-kbproto 1.0.7 h7f98852_1002 conda-forge
xorg-libice 1.1.1 hd590300_0 conda-forge
xorg-libsm 1.2.4 h7391055_0 conda-forge
xorg-libx11 1.8.4 h8ee46fc_1 conda-forge
xorg-libxau 1.0.11 hd590300_0 conda-forge
xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge
xorg-libxext 1.3.4 h0b41bf4_2 conda-forge
xorg-libxrender 0.9.10 h7f98852_1003 conda-forge
xorg-renderproto 0.11.1 h7f98852_1002 conda-forge
xorg-xextproto 7.3.0 h0b41bf4_1003 conda-forge
xorg-xf86vidmodeproto 2.3.1 h7f98852_1002 conda-forge
xorg-xproto 7.0.31 h7f98852_1007 conda-forge
xz 5.2.6 h166bdaf_0 conda-forge
yaml 0.2.5 h7f98852_2 conda-forge
zeromq 4.3.4 h9c3ff4c_1 conda-forge
zfp 1.0.0 h27087fc_3 conda-forge
zipp 3.15.0 pyhd8ed1ab_0 conda-forge
zlib 1.2.13 h166bdaf_4 conda-forge
zlib-ng 2.0.7 h0b41bf4_0 conda-forge
zstd 1.5.2 h3eb15da_6 conda-forge
I will investigate this further this morning.
CameronBodine commented
I identified the source of the issue:
segmentation_gym/make_dataset.py
Lines 455 to 465 in 43edaaf
In my case, if I change:
segmentation_gym/make_dataset.py
Line 552 in 43edaaf
to:
final_sum = initial_sum
in order to bypass if final_sum < initial_sum: ### this ambiguity can happen in 0/1 masks (NCLASSES=2), then the files export as expected.
Potential Fix (just a recommendation)
Change if statement to:
if (final_sum < initial_sum) and (NCLASSES==2):
dbuscombe-usgs commented
Cam, thanks for looking into this function, which is designed for NCLASSES=2. I recently updated it and I don't think the docs have changed for quite a while (apologies).
Please trial your proposed extension to NCLASSES >2 and submit a PR, thanks!
dbuscombe-usgs commented
Thanks for the PR @CameronBodine . the new version does not work for me on a 5-band dataset, so I will need to spend some time fixing it.
dbuscombe-usgs commented
Reopening because
a) I did not update make_datasets for N_DATA_BANDS>3 on my latest revision
b) Cam's PR only applies to N_DATA_BANDS <=3, and the changes break the other cases. For example, [f[0] for f in files] breaks everything because it gets rid of all the additional sets of images.
Therefore, with hindsight, I probably should not have asked Cam to modify. Sorry, Cam.
This really is a very tricky script to modify, and hopefully not too many more changes (!), but let's all get better at testing the N_DATA_BANDS>3 case before making suggestions for improving and using make_datasets. (also, if anyone wants to just rewrite this whole insane workflow so it doesn;t use keras' augmentation options, which are super hard to use, please feel free!!!)
Also, while I'm here, it is NO LONGER the case that label files have to have _label suffix. Ideally, your images and labels have IDENTICAL file names. Otherwise, ensure they natsort the same ... that's the only requirement, other than there being equal numbers of labels and images.
I'm now modifying the script so it works with all cases.
dbuscombe-usgs commented
I my revision, I have attempted to simplify things by moving some repurposed codes into functions. So, we're still under 1000 lines of code! cf47e63
I also managed to simplify the workflow, by offloading any move commands to the resize functions. These changes have required changes to doodleverse-utils (the much-maligned 1-star repo at the heart of the doodleverse)
https://pypi.org/project/doodleverse-utils/0.0.31/
pip install -U doodleverse-utils
CameronBodine commented
Thanks Dan, will give this a test. doodleverse-utils now has 2 stars ;-)