xvecs_size seq4_model_00000000.fvec: weird file size 415130 for vectors of dimen sion 128

[liaoyu1992]

I am very interested in this method. I tried to run the program according to your thoughts. When I made the stadium prediction, I made this mistake：

I found this error for a long time. What is the cause of this? This is part of the log when I was training：

The code is tested on Visual Studio 2015 on a Windows7 system with MATLAB R2016a.
thank you.

[lood339]

The cause of the problem is the file size of the .fvec file is incorrect. The .fvec file stores (SIFT) feature descriptors in the leaf node . The size should be N x 128 x 4, where N is the number of leaf node, 128 is the SIFT feature dimension and 4 is each float requires 4 bytes.
Not very sure, but one of the reason is the .fvec was damaged or replaced improperly after training. The relevant code is bool BTDTRegressor::saveModel(const char *file_name) const , in two_point_calib/src/pan_tilt_forest/bt_dtr/bt_dt_regressor.cpp).
Here are things you may try:

1. manually check the size of .fvec, is the size dividable by 128 * 4?
2. print the size before white the .fvec file. You can do that in line 167 (before YaelIO::write_fvecs_file(leaf_node_files[i].c_str(), data);) of bt_dt_regressor.cpp.
   Hope that helps.

[liaoyu1992]

I have done many trainings and printed the size before white the .fvec file.
This is one of the training results:

ths size of (SIFT) feature descriptors in the leaf node is not dividable by 128 * 4 .
Why is this happening? I studied the code and found no reason.
In many trainings,only once the result the size of .fvec, is the size dividable by 128 * 4 .

There is another question,The code static long xvecs_fsize(long unitsize, const char * fname, int *d_out, int *n_out) , in two_point_calib-master/src/pan_tilt_forest/dt_util/yael_io.cpp.

Why is the rule of judgment is nbytes % (unitsize * d + 4) != 0,Instead of nbytes % (unitsize * d * 4) != 0.

[lood339]

You dive deeper than I do to the yael. My guess of unitsize*d+4, the 4 is an extra data for the number of features. If it is the case, my first suggestion is not correct. Just for your reference.

What do you print for "tree index 0 size is xxxx", it looks the number of leaf nodes in the tree or the sizeof(data) in which data is a Eigen::Matrix.

Is the code sometime works, but some other times do not work? Or it never works?
You may try this way to test if the code works a simple case? Set the number of tree to 1, so that there is only a .fvec file. Then, manually check this name in the model.txt.

[liaoyu1992]

"tree index 0 size is xxxx",it the sizeof(data) in which data is a Eigen::Matrix.

I tested the code many times,it never works.

[lood339]

What you get is the size of a Eigen::Matrix. It is N x 128, 4 should be removed as its type is "float". These numbers are good.
How do you test the code? The problem is in .fvec file read/write. So you'd better to isolate the problem and write a unit test to test the YaelIO::read_fvecs_file and YaelIO::write_fvecs_file. For example, using a randomly generated 1x 128 matrix.

[liaoyu1992]

I prepared data by two_point_calib/data/prepare_train_data_ptz.m,Trained the model by two_point_calib/src/pan_tilt_forest/btdtr_ptz_train_soccer.cpp,then tested the soccer by two_point_calib/src/pan_tilt_forest/btdtr_ptz_test_soccer.cpp,

I studied the code modifed from: https://gforge.inria.fr/projects/yael,then I will test the read_fvecs_file and write_fvecs_file.

thank you.

[liaoyu1992]

The problem is in YaelIO::fvecs_read(file_name, d, n, v) and YaelIO::fvecs_fwrite(FILE *fo, int d, int n, const float *vf) ,This code not work on vc++2015 on a Windows7 system.I modified the code like this:
`int d = (int)data.cols();
int n = (int)data.rows();
const float * v = data.data();
std::ofstream os(file_name, std::ios::binary);
for (int i = 0; i < n; i++)
{
int ret;
printf("fvec_fwrite: row %d\n", i);
os.write((const char*)&d,4);
Eigen::VectorXf vx = data.row(i);
int size = vx.size();

	for (int j = 0; j < size; j++)
	{
		printf("fvec_fwrite: row %d col %d\n", i,j);
		float result = vx[j];
		os.write((const char*)&result, 4);
	}
}
os.close();`

and
`std::ifstream is(fname, std::ios::binary);

long i;
for (i = 0; i < n; i++) {
	int new_d;
	is.read((char*)&new_d, 4);
	if (new_d != d) {
		fprintf(stderr, "fvecs_read error 2: unexpected vector dimension\n");
		is.close();
		return -1;
	}
	float *loc = (a + d * (long)i);
	is.read((char*)loc, d * 4);
}
is.close();`

the code can work normally.