Details
ke-s opened this issue · 12 comments
I have some doubts about the selection of parameters when running the code, if so, can you please tell me the specific role of the parameters, or please give a specific example when running the code.
“path/to/activation” this type of statement to explain its role or the need to establish a path called path/to/activation in the program.
When I run： python write_act.py path/to/dataset --act_path path/to/activation --pca_rate pca_rate --pca_path path/to/pca --gpu gpu_id " line of code, I downloaded fid relevant data by default. Is it the default download or is my running wrong?
I would like to test the quality of other GAN models generated images with your pre-trained model, so when running the Extract features, do path/to/dataset need to point to the dataset you gave or my own? How do you operate if you need to point to the data set you gave?
Thank you very much for your help, as a new person, I hope I will not bother you with your questions, thank you very much.
[我在运行代码时对于参数的选择存在一些疑惑，如果可以，能不能请您详细告知参数的具体作用，或者请您给出一个运行代码时的具体的例子。
path/to/activation这种类型的语句是解释其作用还是说需要在程序中建立名为path/to/activation的路径。
我在运行python write_act.py path/to/dataset --act_path path/to/activation --pca_rate pca_rate --pca_path path/to/pca --gpu gpu_id这行代码时默认下载了fid相关数据，请问是默认下载还是我的运行有错？
我想用您预训练好的模型测试其他GAN模型生成图像的质量，那么在运行Extract features时，path/to/dataset是需要指向您给出的数据集还是我自己的数据集？若需要指向您给出的数据集的话应该如何操作？]
- path/to/activation means you need to give a path that denotes the activation file.
- It's correct to download the InceptionV3 model used in FID, because we also use it to extract features.
- For testing, the path/to/dataset denotes the test data folder. (Since you have already built the model.)
Hello, I used your method to operate the vg dataset, because the image size is different,l adjusted the image to 500 x400 size and got the relevant knn data.
When I choose pca=0.95 and activate operation, its output value is 825, when pca=0.9, its output value is 525, when pca=0.85, the output value is 386. What does this value represent?
When I run knn_score.py measurement gan generate images,pca=0.95 & k= 7000,the score of sample_2557.jpg is:23.120098128922876;pca=0.90 & k= 7000,the score of sample_2557.jpg is:28.22968146838341; pca=0.85 & k=7000,the score of sample_2557.jpg is:33.52534136281287.And when k=3500, score wasn't as high as k=7000,score distribution around 12.
What does this score means? I don't understand it.How did you get your table9? And how can I evaluate my data with your method?
Besides, my behaviors are
“python write_act.py VGomg --act_path activation/vg54.pkl --pca_rate 0.95 --pca_path pca/ pca95-vg54.pkl --gpu 1”
“python knn_score.py data/lostvg --act_path activation/vg54.pkl --pca_path pca/pca95-vg54.pkl --K 3500 --output_file out/knn98/35005495vg.txt --gpu 1”
I will continue to operate your gmm experiment, thank you very much for your help, thank you
- the output value denotes the remaining dimension from PCA, please refer to our paper for more details.
- the score is a “relative” score, compare the number among different models is meaningless, the correct way to use the score is to compare two image's scores from the same model(KNN or GMM), and the higher score denotes the image has better quality.
- For table9 in our paper, first these are evaluated on our LGIQA dataset, you may download it from "readme". LGIQA dataset contains pairs of images which has annotated which has the higher quality, you can compare it with the score you get, than you may get the accuracy. I really suggest you read our paper more carefully, especially Sec3.2 and Sec5.0, it may improve your understanding of our work.
- Besides, I think VisualGenorm dataset is kind of complicated, I haven't tried on this dataset, I think for this kind of complicated dataset, it's better to use all the images to get the activation, for KNN_score, I am not sure what's the best K number, a bigger K cost more time than small K, it's worth trying a smaller K, besides, it's also worth trying the GMM_score, it needs an additional time for building GMM, but the inference is much faster compared with KNN.
@cientgu Hello, I used my own data and the GMM model, but got a very large negative score value like this, is this normal?
@cientgu Hello, I used my own data and the GMM model, but got a very large negative score value like this, is this normal?
Only analysis of the value of the score is meaningless, but you can use the score to compare which is better and which is worse on the same dataset, I think the number is normal. Notice one more thing, there should be enough data in your own dataset, otherwise, it may be inaccurate.
@cientgu Hello, I used my own data and the GMM model, but got a very large negative score value like this, is this normal?
Only analysis of the value of the score is meaningless, but you can use the score to compare which is better and which is worse on the same dataset, I think the number is normal. Notice one more thing, there should be enough data in your own dataset, otherwise, it may be inaccurate.
Thank you for your reply, just to confirm one more thing, when the score is negative, the image quality with a score of -8202388 is higher than that with a score of -8929917, right?
In other words, I should not compare the absolute value but the real value of the score, right?
Hello, I am sorry to bother you again could you provide your evaluation database to me ? I downloaded your data set through the link, but it doesn't give a corresponding score.
I used my own datasets to train the GMM model to score, but the score was too low, I wanted to test whether the model I got by training my own data set was consistent with your manual evaluation criteria. To see if there is a problem with my own model.
Thank you for your help.
This is the fraction of the partial vg dataset I got, where the m component is 50:
score of sample_2557.jpg is:
-331.1316756187852
score of sample_2283.jpg is:
-633.6856968862959
score of sample_2106.jpg is:
-274.1916243507116
score of sample_540.jpg is:
-333.962998405286
score of sample_66.jpg is:
-334.99846765170224
score of sample_434.jpg is:
-538.2433860777709
score of sample_2387.jpg is:
-509.22751200925626
score of sample_3122.jpg is:
-717.7854063733386
score of sample_710.jpg is:
-518.359456909934
score of sample_1136.jpg is:
-843.3530016434424
score of sample_1158.jpg is:
-533.8602747861478
score of sample_662.jpg is:
-651.9051215940399
score of sample_697.jpg is:
-796.5913233245227
score of sample_3185.jpg is:
-651.4458333069384
score of sample_748.jpg is:
-483.165836370943
score of sample_751.jpg is:
-443.9178540298226
what's more，I get the mu and sigma are both 0， so ,I feel like I might be wrong
@cientgu Hello, I used my own data and the GMM model, but got a very large negative score value like this, is this normal?
Only analysis of the value of the score is meaningless, but you can use the score to compare which is better and which is worse on the same dataset, I think the number is normal. Notice one more thing, there should be enough data in your own dataset, otherwise, it may be inaccurate.
Thank you for your reply, just to confirm one more thing, when the score is negative, the image quality with a score of -8202388 is higher than that with a score of -8929917, right?
In other words, I should not compare the absolute value but the real value of the score, right?
Yes, you are right.
Hello, I am sorry to bother you again could you provide your evaluation database to me ? I downloaded your data set through the link, but it doesn't give a corresponding score.
I used my own datasets to train the GMM model to score, but the score was too low, I wanted to test whether the model I got by training my own data set was consistent with your manual evaluation criteria. To see if there is a problem with my own model.
Thank you for your help.
This is the fraction of the partial vg dataset I got, where the m component is 50:
score of sample_2557.jpg is:
-331.1316756187852
score of sample_2283.jpg is:
-633.6856968862959
score of sample_2106.jpg is:
-274.1916243507116
score of sample_540.jpg is:
-333.962998405286
score of sample_66.jpg is:
-334.99846765170224
score of sample_434.jpg is:
-538.2433860777709
score of sample_2387.jpg is:
-509.22751200925626
score of sample_3122.jpg is:
-717.7854063733386
score of sample_710.jpg is:
-518.359456909934
score of sample_1136.jpg is:
-843.3530016434424
score of sample_1158.jpg is:
-533.8602747861478
score of sample_662.jpg is:
-651.9051215940399
score of sample_697.jpg is:
-796.5913233245227
score of sample_3185.jpg is:
-651.4458333069384
score of sample_748.jpg is:
-483.165836370943
score of sample_751.jpg is:
-443.9178540298226
what's more，I get the mu and sigma are both 0， so ,I feel like I might be wrong
I will update it in the next two days.
Hello, I am sorry to bother you again could you provide your evaluation database to me ? I downloaded your data set through the link, but it doesn't give a corresponding score.
I used my own datasets to train the GMM model to score, but the score was too low, I wanted to test whether the model I got by training my own data set was consistent with your manual evaluation criteria. To see if there is a problem with my own model.
Thank you for your help.
This is the fraction of the partial vg dataset I got, where the m component is 50:
score of sample_2557.jpg is:
-331.1316756187852
score of sample_2283.jpg is:
-633.6856968862959
score of sample_2106.jpg is:
-274.1916243507116
score of sample_540.jpg is:
-333.962998405286
score of sample_66.jpg is:
-334.99846765170224
score of sample_434.jpg is:
-538.2433860777709
score of sample_2387.jpg is:
-509.22751200925626
score of sample_3122.jpg is:
-717.7854063733386
score of sample_710.jpg is:
-518.359456909934
score of sample_1136.jpg is:
-843.3530016434424
score of sample_1158.jpg is:
-533.8602747861478
score of sample_662.jpg is:
-651.9051215940399
score of sample_697.jpg is:
-796.5913233245227
score of sample_3185.jpg is:
-651.4458333069384
score of sample_748.jpg is:
-483.165836370943
score of sample_751.jpg is:
-443.9178540298226
what's more，I get the mu and sigma are both 0， so ,I feel like I might be wrong
I have updated some test images and results, please refer to readme.
Hello, I input the testset , run get_gmm, get the mu and sigma in the “calculate_activation_statistics “are both 0, such as : mu：[0.0.0....0.0.0.]. What’s the reason?
Is this the mean and sigma of my test image orthe training set? I'm confused about that. Thank you very much
Hello, I input the testset , run get_gmm, get the mu and sigma in the “calculate_activation_statistics “are both 0, such as : mu：[0.0.0....0.0.0.]. What’s the reason?
Is this the mean and sigma of my test image orthe training set? I'm confused about that. Thank you very much
The test set only contains 50 test images (not the training dataset!). Use it to test the gmm_score.py file with the pre-built GMM model on the LSUN-cat dataset, which I have put here(https://drive.google.com/drive/folders/17fAzhyQGXwgSJYO1PhmbnSl72FAE4VCJ). We did not provide code to test "extract features" or "get gmm". If you have more questions, please email me at gsy777@mail.ustc.edu.cn.