/AIAproject-FittingRoom

Virtual Try-On scheme base on cycleGAN

Primary LanguagePython

AIAproject-FittingRoom

Fig.1 Testing result for different types of Try-On task.

Dataset

Discussion

  • We replicated CP-VTON's Try-On Module by the implementation details, provided by their paper, to make comparison with our cycleTryOn module in this section.

Geometric Matching Limitation

  • Although Geometric Matching Module(GMM), proposed by CP-VTON, had proven its efficiency in aligning in-shop clothes with human images, GMM dose not have the ability to differentiate the inside and outside of clothes, To be more precisely, GMM tends to force the WHOLE in-shop cloth into the original clothing shape on person if the deformation grid is dense enough.

  • Cycle Consistency loss and Adversarial loss are introduced to calibrate this kind of unrealistic problems in this work.

  • As shown in Fig.2, the lining and tag of target clothes still exist in results of CP-VTON. In comparison to CP-VTON, our cycleTryOn module has learned how to hide the inner part of clothes and successfully generate the appropriate skin color of people.

Fig.2 In contrary to the lining showing problem with CP-VTON's, our cycleTryOn module has successfully adjusted this problem.

Arms Missing Problem

  • For some rare poses in dataset, pose with folded arms for example, failure rate become extremely high in CP-VTON. Substantially, there has NO success testing result been found in our facsimile of CP-VTON module.

  • From our perspective, networks without inverse mapping is hard to learn in which case people's body information should be reserved. The best strategy of generator is to paste warped clothes on the right position when the data of folded arms pose is short.

  • Cycle Consistency takes a huge advantage to minimize this problem. It's simply because if people's body information is missing in final results, the generator will become heavily loaded when it mapping back to original images. Therefore, the best strategy will be reserving those information at the first time.

Fig.3 In comparison of arms missing problem with CP-VTON's results, our cycleTryOn module has successfully reserved arms information. Even though our model haven't optimize to its best state, the tendency is still obvious.

Joined Limbs Into Body Information

  • After resolving arms missing problem we combine limbs information into body information, aiming to refine the details.

  • Providing limbs information to the network do result great limbs details as shown in Fig.4. However, the presence of limbs information in input seems to strongly limit the clothing shape to the original one on model(see Fig.4).

  • Although providing limbs information restricted the usage to same clothing type only, we still suggest training model by case for real word usage considering the prestigious refinement of its details.

Fig.4 Comparison of limbs information been joined.

Fig.5 Cases of the clothing type changed.

Human Parser Influence

  • Since GMM takes the original clothing on person as ground truth by using human parser to crop the clothing out from person image, we premeditated the higher mIoU human parsing network can achieve the better GMM result would be.

Fig.6 Higher IoU dose improve GMM result .

User Guide

Training

python train_cycleTryOn.py --name 'gmm_train' --stage 'GMM' --save_count 5000 --shuffle
python train_cycleTryOn.py --name 'cycleTryOn_train' --stage 'cycleTryOn' --uselimbs

Testing

python test_cycleTryOn.py --name 'gmm_test' --stage 'GMM' --datamode test --data_list 'test_pairs.txt' --checkpoint checkpoints/gmm_train/gmm_final.pth
python test_cycleTryOn.py --name 'cycleTryOn_test' --stage 'cycleTryOn' --uselimbs --datamode test --data_list 'test_pairs.txt' --checkpoint checkpoints/cycleTryOn_train/cycleTryOn_final.pth

Reference

  • Wang, Bochao, et al. "Toward characteristic-preserving image-based virtual try-on network." Proceedings of the European Conference on Computer Vision (ECCV). 2018.
  • Zheng, Na, et al. "Virtually trying on new clothing with arbitrary poses." Proceedings of the 27th ACM International Conference on Multimedia. 2019.