Changing the name of the .txt file

Question

Changing the name of the .txt file

Opened this issue 5 years ago · 14 comments

Hello @sowson, when I was trying to train a yolov3-tiny model, I ran into an error. The program couldn't open the file labels_19886_r270.txt. When I checked the file, I found that I don't have a file by that name, instead I have one by the name images_19886_r270.txt, the same as the name of the jpg file.

Since I am not familiar with the code of darknet, can you tell what I can change to fix this??

Here's the image of the output, just in case -

Thanks for the help

Answer 1 · 2020-02-14T06:19:48.000Z

Hi, training should be like that:

./darknet detector train yolov3-tiny.data yolov3-tiny.cfg yolov3-tiny.conv.15 -I 1

where missing in your case yolo3-tiny.data should be similar to this:

classes= 20
train = /home/pjreddie/data/voc/train.txt
valid = /home/pjreddie/data/voc/2007_test.txt
names = data/voc.names
backup = backup

Answer 2 · 2020-02-14T07:14:27.000Z

Hi, I'm using a different .data file for my model, the peoplecount.data file.

Here are the contents :-

classes = 1
train = /Users/Prakhar/Image_Processing_and_CV/Adappt/peoplecount_neural_network_2020-02-09_19-54-59_result_test_copy/peoplecount_train.txt
valid = /Users/Prakhar/Image_Processing_and_CV/Adappt/peoplecount_neural_network_2020-02-09_19-54-59_result_test_copy/peoplecount_valid.txt
names = /Users/Prakhar/Image_Processing_and_CV/Adappt/peoplecount_neural_network_2020-02-09_19-54-59_result_test_copy/peoplecount.names
backup = /Users/Prakhar/Image_Processing_and_CV/Adappt/peoplecount_neural_network_2020-02-09_19-54-59_result_test_copy/backup/

Also, I made a mistake in uploading the image, I cut off a bit off the command, which included the peoplecount.data file, I'll reupload it here :-

Answer 3 · 2020-02-14T07:25:32.000Z

One more thing pls make sure you have Linux End of Line (EOL) in your text files and not Windows EOL, that is instead of "\n\r" you should have only "\n" at the end of txt files. One of the tool that can automatically change that is Notepad++ but you may use different. That is the only guest I have :-).

Answer 4 · 2020-02-14T07:30:39.000Z

I'm using a MacBook, so I don't think I would be using Windows EOL. Could you point me to the function which reads the training.txt file, so I can check which function it calls??

I can go through the functions and check where the change is being made, hopefully. I double checked my image dataset, I have no filename starting with labels, so I think there is a function making the change.

Thanks for the help

Answer 5 · 2020-02-14T07:33:18.000Z

I think all files are read by the code from src/data.c where is the function on the top named get_paths... is that help you?

Answer 6 · 2020-02-14T07:41:20.000Z

Also if you are in darknet directory you may run:

grep -Rn "\"labels\"" ./

or

grep -Rn "labels" ./

to find that part in code ;-).

Answer 7 · 2020-02-14T07:45:47.000Z

The error is on Line 222, 260 and 452 of src/data.c in the function fill_truth_swag, fill_truth_region and fill_truth_detection. Line 222 & 223 replace "images" and "JPEGImages" with "labels". Same with Lines 260 & 261 and 452 & 453.

Relevant part of code for function fill_truth_region
void fill_truth_swag(char *path, float *truth, int classes, int flip, float dx, float dy, float sx, float sy) { char labelpath[4096]; find_replace(path, "images", "labels", labelpath); find_replace(labelpath, "JPEGImages", "labels", labelpath); find_replace(labelpath, ".jpg", ".txt", labelpath); find_replace(labelpath, ".JPG", ".txt", labelpath); find_replace(labelpath, ".JPEG", ".txt", labelpath);

Relevant part of code for function fill_truth_detection
`void fill_truth_region(char *path, float *truth, int classes, int num_boxes, int flip, float dx, float dy, float sx, float sy)
{
char labelpath[4096];
find_replace(path, "images", "labels", labelpath);
find_replace(labelpath, "JPEGImages", "labels", labelpath);

find_replace(labelpath, ".jpg", ".txt", labelpath);
find_replace(labelpath, ".png", ".txt", labelpath);
find_replace(labelpath, ".JPG", ".txt", labelpath);
find_replace(labelpath, ".JPEG", ".txt", labelpath);

`

Relevant part of code for function fill_truth_swag
`void fill_truth_detection(char *path, int num_boxes, float *truth, int classes, int flip, float dx, float dy, float sx, float sy)
{
char labelpath[4096];
find_replace(path, "images", "labels", labelpath);
find_replace(labelpath, "JPEGImages", "labels", labelpath);

find_replace(labelpath, "raw", "labels", labelpath);
find_replace(labelpath, ".jpg", ".txt", labelpath);
find_replace(labelpath, ".png", ".txt", labelpath);
find_replace(labelpath, ".JPG", ".txt", labelpath);
find_replace(labelpath, ".JPEG", ".txt", labelpath);

`

I'll comment out the lines changing "images" and "JPEGImages" and test it. If there are no errors, I'll comment over here

Answer 8 · 2020-02-14T07:47:42.000Z

or you may make a directory labels and:

mv *.txt labels/

what do you think?

Answer 9 · 2020-02-14T07:50:14.000Z

The function is finding and replacing them, right. So the file name would still be wrong, and the code would not open the file
If I am not wrong,

mv *.txt labels/

would move all .txt files to a new directory labels, right??

Answer 10 · 2020-02-14T07:55:31.000Z

right it looks like in images folder you need to have only images and inside folder labels with all txt labels... so to make sure I checked on VOC example by:

find ./ -type d

It shows me:

.//VOCdevkit
.//VOCdevkit/VOC2007
.//VOCdevkit/VOC2007/ImageSets
.//VOCdevkit/VOC2007/ImageSets/Segmentation
.//VOCdevkit/VOC2007/ImageSets/Layout
.//VOCdevkit/VOC2007/ImageSets/Main
.//VOCdevkit/VOC2007/SegmentationClass
.//VOCdevkit/VOC2007/labels
.//VOCdevkit/VOC2007/SegmentationObject
.//VOCdevkit/VOC2007/Annotations
.//VOCdevkit/VOC2007/JPEGImages
.//VOCdevkit/VOC2012
.//VOCdevkit/VOC2012/ImageSets
.//VOCdevkit/VOC2012/ImageSets/Segmentation
.//VOCdevkit/VOC2012/ImageSets/Layout
.//VOCdevkit/VOC2012/ImageSets/Action
.//VOCdevkit/VOC2012/ImageSets/Main
.//VOCdevkit/VOC2012/SegmentationClass
.//VOCdevkit/VOC2012/labels
.//VOCdevkit/VOC2012/SegmentationObject
.//VOCdevkit/VOC2012/Annotations
.//VOCdevkit/VOC2012/JPEGImages

So, looks like I am right and labels sub-directory is needed with all txt labels inside.

Answer 11 · 2020-02-14T07:58:27.000Z

The error is also on lines 598, 599, 642 and 643.

Also, I have not been using VOC Dataset for training. I'm using a custom dataset. I'll move the labels and test both methods and get back to you.

Thanks for all the help.

Answer 12 · 2020-02-14T07:59:37.000Z

ok, happy training!

Answer 13 · 2020-02-14T09:58:13.000Z

I changed the "labels" argument with the "images", "JPEGImages", "imgs" and "raw" wherever they were mentioned in the src/data.c . The find_replace() function is also used in detector.c, so I had to replace the strings over there as well. Now, the model is happily training.

It will take a while before it finishes and I can test the second method.

@sowson , should I close the issue or should I close it after testing the second method??

I think you can still comment on the issue after the issue is closed, so I'll post the end result of the method.

And again, Thanks for helping out

Answer 14 · 2020-02-14T12:55:22.000Z

@pgarhwal regarding the close or not it is up to you, if it works it will be nice to people with similar issue in future ;-) Thank you!