[question] CCTag Feature Extraction fails

Question

[question] CCTag Feature Extraction fails

Fylg1 opened this issue 3 years ago · 13 comments

Hi,

First of all thank you very much for this awesome tool! I really enjoy working with it! I hope you can help me with my problem:
I am trying to use CCTags in my SfM-Project, however when I enable the "cctag3"-describers in the "feature extraction"-node, the step fails.
I have tried it with different scenes, amounts of markers (from the markersToPrint-section), two different cameras (IPhone SE and Sony Alpha 6000), on two realeases (Meshroom 2020.1.1 and 2021.1.0) and two computers. The result is however always the same. The problem also appears when I use other describers (sift/akaze) with it. As soon as I mark cctag3, there is an error. Settings for Describer Preset and/or Verbose Level don't change anything either.

I don't know if I am doing something completely wrong, but also cannot find anything regarding the error return code. I really hope you can help me.

I am using the standard out-of-the-box graph for photogrammetry.
Here is the status-file for a project with only 8 images:

{
    "status": "ERROR",
    "execMode": "LOCAL",
    "nodeName": "FeatureExtraction_1",
    "nodeType": "FeatureExtraction",
    "packageName": "aliceVision",
    "packageVersion": null,
    "graph": "",
    "commandLine": "aliceVision_featureExtraction  --input \"D:/CCTag_Test/MeshroomCache/CameraInit/d68935f8bd4655a5b54f7ed8a87f6adaea058bab/cameraInit.sfm\" --describerTypes cctag3 --describerPreset normal --forceCpuExtraction True --maxThreads 0 --verboseLevel info --output \"D:/CCTag_Test/MeshroomCache/FeatureExtraction/18e89274cd014fe2b1cd27603b8fb7210fcbeb85\" --rangeStart 0 --rangeSize 40",
    "env": null,
    "startDateTime": "2021-11-25 10:34:02.849487",
    "endDateTime": "",
    "elapsedTime": 0,
    "hostname": "abc"
    "sessionUid": "534dcf06-4dd2-11ec-8480-186024bc5d4c",
    "returnCode": 3221226505,
    "elapsedTimeStr": "0:00:00"
}

The log is:

Program called with the following parameters:
 * describerPreset = "normal"
 * describerTypes = "cctag3"
 * forceCpuExtraction = 1
 * input = "D:/CCTag_Test/MeshroomCache/CameraInit/d68935f8bd4655a5b54f7ed8a87f6adaea058bab/cameraInit.sfm"
 * maxThreads = 0
 * output = "D:/CCTag_Test/MeshroomCache/FeatureExtraction/18e89274cd014fe2b1cd27603b8fb7210fcbeb85"
 * rangeSize = 40
 * rangeStart = 0
 * verboseLevel =  Unknown Type "enum aliceVision::system::EVerboseLevel"

[10:34:03.404036][info] CUDA-Enabled GPU.
Device information:
	- id:                      0
	- name:                    Quadro M2000M
	- compute capability:      5.0
	- total device memory:     4096 MB 
	- device memory available: 3415 MB 
	- per-block shared memory: 49152
	- warp size:               32
	- max threads per block:   1024
	- max threads per SM(X):   2048
	- max block sizes:         {1024,1024,64}
	- max grid sizes:          {2147483647,65535,65535}
	- max 2D array texture:    {65536,65536}
	- max 3D array texture:    {4096,4096,4096}
	- max 2D linear texture:   {65536,65536,1048544}
	- max 2D layered texture:  {16384,16384,2048}
	- number of SM(x)s:        5
	- registers per SM(x):     65536
	- registers per block:     65536
	- concurrent kernels:      yes
	- mapping host memory:     yes
	- unified addressing:      yes
	- texture alignment:       512 byte
	- pitch alignment:         32 byte
0
[10:34:03.525399][info] Job max memory consumption for one image: 34 MB
[10:34:03.525399][info] Memory information: 
	- Total RAM:  31.8824 GB
	- Free RAM:   14.5426 GB
	- Total swap: 131072 GB
	- Free swap:  131024 GB
[10:34:03.525399][info] Max number of threads regarding memory usage: 384
[10:34:03.525399][warning] More than half of the RAM is used by other applications. It would be more efficient to close them.
[10:34:03.525399][warning]  => 17 GB are used by other applications for a total RAM capacity of 32 GB.
[10:34:03.525399][info] # threads for extraction: 8
[10:34:04.230425][info] Extracting cctag3 features from view 'D:/CCTag_Test/20211125_092925168_iOS.jpg' [cpu]
[10:34:04.275753][info] Extracting cctag3 features from view 'D:/CCTag_Test/20211125_092922442_iOS.jpg' [cpu]
[10:34:04.282512][info] Extracting cctag3 features from view 'D:/CCTag_Test/20211125_092915038_iOS.jpg' [cpu]
[10:34:04.290745][info] Extracting cctag3 features from view 'D:/CCTag_Test/20211125_092919030_iOS.jpg' [cpu]
[10:34:04.323542][info] Extracting cctag3 features from view 'D:/CCTag_Test/20211125_092931632_iOS.jpg' [cpu]
[10:34:04.323542][info] Extracting cctag3 features from view 'D:/CCTag_Test/20211125_092935913_iOS.jpg' [cpu]
[10:34:04.344085][info] Extracting cctag3 features from view 'D:/CCTag_Test/20211125_092939145_iOS.jpg' [cpu]
[10:34:04.420078][info] Extracting cctag3 features from view 'D:/CCTag_Test/20211125_092928689_iOS.jpg' [cpu]

I have tried multiple scenes/setups, here an example image from a very "easy" scene...
Screenshots

Desktop:

OS: Win 10
CCtag version: Meshroom 2020.1.1 (also Meshroom 2021.1.0)

Answer 1 · 2021-11-25T12:16:52.000Z

can you re-run with a more detailed verbosity level, like Debug?
There is no apparent error message so it's strange that the node is set in error

Answer 2 · 2021-11-25T12:27:18.000Z

Hi, thanks for your answer. I ran it again on "Debug" and als "Trace". Here is the log output for "Trace". Status seems unchanged.

Program called with the following parameters:
 * describerPreset = "normal"
 * describerTypes = "cctag3"
 * forceCpuExtraction = 1
 * input = "D:/CCTag_Test/MeshroomCache/CameraInit/d68935f8bd4655a5b54f7ed8a87f6adaea058bab/cameraInit.sfm"
 * maxThreads = 0
 * output = "D:/CCTag_Test/MeshroomCache/FeatureExtraction/18e89274cd014fe2b1cd27603b8fb7210fcbeb85"
 * rangeSize = 40
 * rangeStart = 0
 * verboseLevel =  Unknown Type "enum aliceVision::system::EVerboseLevel"

[13:25:43.650066][info] CUDA-Enabled GPU.
Device information:
	- id:                      0
	- name:                    Quadro M2000M
	- compute capability:      5.0
	- total device memory:     4096 MB 
	- device memory available: 3415 MB 
	- per-block shared memory: 49152
	- warp size:               32
	- max threads per block:   1024
	- max threads per SM(X):   2048
	- max block sizes:         {1024,1024,64}
	- max grid sizes:          {2147483647,65535,65535}
	- max 2D array texture:    {65536,65536}
	- max 3D array texture:    {4096,4096,4096}
	- max 2D linear texture:   {65536,65536,1048544}
	- max 2D layered texture:  {16384,16384,2048}
	- number of SM(x)s:        5
	- registers per SM(x):     65536
	- registers per block:     65536
	- concurrent kernels:      yes
	- mapping host memory:     yes
	- unified addressing:      yes
	- texture alignment:       512 byte
	- pitch alignment:         32 byte
0
[13:25:43.760077][info] Job max memory consumption for one image: 34 MB
[13:25:43.760077][info] Memory information: 
	- Total RAM:  31.8824 GB
	- Free RAM:   19.1175 GB
	- Total swap: 131072 GB
	- Free swap:  131024 GB
[13:25:43.760077][info] Max number of threads regarding memory usage: 505
[13:25:43.760077][info] # threads for extraction: 8
[13:25:44.077099][trace] Read image D:/CCTag_Test/20211125_092922442_iOS.jpg (encoded in sRGB colorspace).
[13:25:44.094101][trace] Read image D:/CCTag_Test/20211125_092919030_iOS.jpg (encoded in sRGB colorspace).
[13:25:44.094101][trace] Read image D:/CCTag_Test/20211125_092915038_iOS.jpg (encoded in sRGB colorspace).
[13:25:44.313113][trace] Read image D:/CCTag_Test/20211125_092928689_iOS.jpg (encoded in sRGB colorspace).
[13:25:44.335115][trace] Read image D:/CCTag_Test/20211125_092939145_iOS.jpg (encoded in sRGB colorspace).
[13:25:44.351115][trace] Read image D:/CCTag_Test/20211125_092935913_iOS.jpg (encoded in sRGB colorspace).
[13:25:44.384117][trace] Read image D:/CCTag_Test/20211125_092931632_iOS.jpg (encoded in sRGB colorspace).
[13:25:44.431121][trace] Read image D:/CCTag_Test/20211125_092925168_iOS.jpg (encoded in sRGB colorspace).
[13:25:44.476125][info] Extracting cctag3 features from view 'D:/CCTag_Test/20211125_092922442_iOS.jpg' [cpu]
[13:25:44.508128][info] Extracting cctag3 features from view 'D:/CCTag_Test/20211125_092919030_iOS.jpg' [cpu]
[13:25:44.544129][info] Extracting cctag3 features from view 'D:/CCTag_Test/20211125_092915038_iOS.jpg' [cpu]
[13:25:44.647137][info] Extracting cctag3 features from view 'D:/CCTag_Test/20211125_092928689_iOS.jpg' [cpu]
[13:25:44.726141][info] Extracting cctag3 features from view 'D:/CCTag_Test/20211125_092939145_iOS.jpg' [cpu]
[13:25:44.773149][info] Extracting cctag3 features from view 'D:/CCTag_Test/20211125_092925168_iOS.jpg' [cpu]
[13:25:44.783145][info] Extracting cctag3 features from view 'D:/CCTag_Test/20211125_092931632_iOS.jpg' [cpu]
[13:25:44.940158][info] Extracting cctag3 features from view 'D:/CCTag_Test/20211125_092935913_iOS.jpg' [cpu]

Answer 3 · 2021-11-25T13:21:48.000Z

it's strange, it's like the threads get stuck.

can you run the command line in a terminal to see if there are some errors that pop up (you are on windows right?)

aliceVision_featureExtraction.exe  --input \"D:/CCTag_Test/MeshroomCache/CameraInit/d68935f8bd4655a5b54f7ed8a87f6adaea058bab/cameraInit.sfm\" --describerTypes cctag3 --describerPreset normal --forceCpuExtraction True --maxThreads 0 --verboseLevel trace --output \"D:/CCTag_Test/MeshroomCache/FeatureExtraction/18e89274cd014fe2b1cd27603b8fb7210fcbeb85\" --rangeStart 0 --rangeSize 40

(you may need to set the PATH to point to the folder containing all the alcievision libraries and dependencies)

Also you can quickly test with the two sample images that come with this repo
https://github.com/alicevision/CCTag/tree/develop/sample

Finally, in the advanced attributes (top right dots of the attributes panel to enable) try to set max nb Threads to 1 to only use one thread in case there is some conflict

Answer 4 · 2021-11-25T14:13:08.000Z

Hi! Thanks for the assistance!
So the command line did not work, but also did not throw any errors there (I also once changed the output dir to see if anything was happening- nothing was done). Your tip on limiting the number of threads however worked perfectly, and the FeatureExtraction succeeded. However, now it gets stuck in the ImageMatching, stating that the descriptor for the first view it tries are not found:

Program called with the following parameters:
 * featuresFolders =  = [D:/CCTag_Test2/MeshroomCache/FeatureExtraction/d004ffccfd43f55a6dfb4c0e8aad36951407df55]
 * input = "D:/CCTag_Test2/MeshroomCache/CameraInit/59cde1304f9d10229e3b7239598d3ccfe12be37a/cameraInit.sfm"
 * matchingMode = "a/a" (default)
 * maxDescriptors = 500
 * method =  Unknown Type "enum EImageMatchingMethod"
 * minNbImages = 200
 * nbMatches = 50
 * nbNeighbors = 50 (default)
 * output = "D:/CCTag_Test2/MeshroomCache/ImageMatching/6b4af77faeb259e78048f7f74a9999bf04263b5f/imageMatches.txt"
 * outputCombinedSfM = "" (default)
 * tree = "D:\Meshroom-2020.1.1-win64\Meshroom-2020.1.1\aliceVision\share\aliceVision\vlfeat_K80L3.SIFT.tree"
 * verboseLevel = "trace"
 * weights = ""

[15:03:26.023179][trace] 0 extrinsics declared and 8 extrinsics used.
[15:03:26.023179][fatal] Can't find descriptor of view 107673506 in:
	- D:/CCTag_Test2/MeshroomCache/FeatureExtraction/d004ffccfd43f55a6dfb4c0e8aad36951407df55

I can however manually find that descriptor in the folder:

Trying the same with e.g. "sift"-desribers works fine...

Answer 5 · 2021-11-25T15:10:27.000Z

try to force "Exhaustive" method in the attributes of the node.

Also can u check one of the .feat files by loading it into e.g. Notepad (it's a simple text file)

Answer 6 · 2021-11-25T15:40:41.000Z

Hi!
Forcing the exhaustive mode unfortunately did not change anything. The .feat files contain something like

1066.08 2397.35 259.721 0
2536.9 1344.63 195.555 0
481.814 1376.9 194.071 0
2565.59 2356.51 244.104 0

hope that helps...

Answer 7 · 2021-11-25T16:00:25.000Z

There is something strange because it seems like it is trying to load the SIFT* descriptors for the vocabulary tree which should not be the case for the EXHAUSTIVE method.
Do you get a line [16:46:24.431608][info] Use EXHAUSTIVE method.?

the error you receive comes from a call to getListOfDescriptorFiles() which as said above tries to look for SIFT features.
https://github.com/alicevision/AliceVision/blob/c979b13ac8c13582de5f599d8a5889a1de83b644/src/software/pipeline/main_imageMatching.cpp?_pjax=%23js-repo-pjax-container%2C%20div%5Bitemtype%3D%22http%3A%2F%2Fschema.org%2FSoftwareSourceCode%22%5D%20main%2C%20%5Bdata-pjax-container%5D#L778

  if(method != EImageMatchingMethod::EXHAUSTIVE)
  {
      // load descriptor filenames  --> should never enter here with EXHAUSTIVE
      aliceVision::voctree::getListOfDescriptorFiles(sfmDataA, featuresFolders, descriptorsFilesA);

      if(useMultiSfM)
          aliceVision::voctree::getListOfDescriptorFiles(sfmDataB, featuresFolders, descriptorsFilesB);
  }

  OrderedPairList selectedPairs;

  switch(method)
  {
    case EImageMatchingMethod::EXHAUSTIVE:
    {
      ALICEVISION_LOG_INFO("Use EXHAUSTIVE method.");
      if((matchingMode == EImageMatchingMode::A_A_AND_A_B) ||
         (matchingMode == EImageMatchingMode::A_AB) ||
         (matchingMode == EImageMatchingMode::A_A))
          generateAllMatchesInOneMap(sfmDataA.getViewsKeys(), selectedPairs);

      if((matchingMode == EImageMatchingMode::A_A_AND_A_B) ||
         (matchingMode == EImageMatchingMode::A_AB) ||
         (matchingMode == EImageMatchingMode::A_B))
          generateAllMatchesBetweenTwoMap(sfmDataA.getViewsKeys(), sfmDataB.getViewsKeys(), selectedPairs);
      break;
    }

Are u sure that you don't have other features selected?

Answer 8 · 2021-11-25T16:15:31.000Z

So I built a completely new project to avoid there being multiple cashe folders etc. I did one run with FeatureExtr. set to 1 thread and CCTag3 only. ImageMatching set to"Exhaustive" with verbose "Trace". Again stopped at ImageMatching with the following log:

Program called with the following parameters:
 * featuresFolders =  = [D:/CCTag_Test3/MeshroomCache/FeatureExtraction/fcf2f31bad18726674236ce18421536550f67a60]
 * input = "D:/CCTag_Test3/MeshroomCache/CameraInit/5e578a422022269337e96bd354e4407e3133a5c8/cameraInit.sfm"
 * matchingMode = "a/a" (default)
 * maxDescriptors = 500 (default)
 * method =  Unknown Type "enum EImageMatchingMethod"
 * minNbImages = 200 (default)
 * nbMatches = 50 (default)
 * nbNeighbors = 50 (default)
 * output = "D:/CCTag_Test3/MeshroomCache/ImageMatching/abee37127f18333113354e3966a1f6ca7b720a86/imageMatches.txt"
 * outputCombinedSfM = "" (default)
 * tree = "" (default)
 * verboseLevel = "trace"
 * weights = "" (default)

[17:06:03.828907][trace] 0 extrinsics declared and 8 extrinsics used.
[17:06:03.844532][fatal] Can't find descriptor of view 195016313 in:
	- D:/CCTag_Test3/MeshroomCache/FeatureExtraction/fcf2f31bad18726674236ce18421536550f67a60

So no "Use EXHAUSTIVE method" promt. (I am using Meshroom 2020.1.1 for this trial right now if that matters).
Is it normal that the log states:
* method = Unknown Type "enum EImageMatchingMethod"?

CCTags features were found in the images, e.g.:

Answer 9 · 2021-11-25T16:20:44.000Z

ok that seems to be a bug of 2020.1. I just replicated it.
Can you try with 2021.1 with a clean project etc? For me it works.

Answer 10 · 2021-11-25T16:32:58.000Z

Great! Worked like a charm in 2021.1.0 (on 1 thread)! I will try to run the full pipeline now!
Thanks for the great help already, really helped me out! :)

Answer 11 · 2021-11-26T07:03:45.000Z

Hi,
so the CCTag-only reconstruction failed in the SfM-step due to no valid tracks:

[17:29:24.727277][trace] Matches per image pair (before filtering):
[17:29:24.727277][trace] Matches per image pair (after filtering):
[17:29:24.727277][debug] Track building
[17:29:24.727277][debug] Track filtering
[17:29:24.727277][debug] Track export to internal structure
[17:29:24.727277][debug] Build tracks per view
[17:29:24.727277][debug] Build tracks pyramid per view
[17:29:24.727277][info] Fuse matches into tracks: 
	- # tracks: 0
	- # images in tracks: 8
[17:29:24.727277][info] TrackLength, Occurrence
[17:29:24.727277][fatal] No valid tracks.

However, by also adding sift to the describer types in all nodes (building a second FeatureExtraction node running on all threads to speed things up) I was able to get it to succeed. Probably this simple dataset with only 8 images does not suffice for CCTag-only SfM?
I am now running some trials on a much larger dataset, but that might take a while.
Here's the result from the reconstruction with sift/CCTag combined:

Thanks again for the help, I really appreciate it!

Answer 12 · 2021-11-26T10:08:53.000Z

to run it with cctag only you need more tags in the same image.
And even so, it's unlikely to work straight away because of some parameters that are too strict on the number of tracks.
If you need it I can push a branch in which I relaxed some constraints that enable the reconstruction with markers only (you cannot make these changes from the GUI)

Answer 13 · 2021-11-26T10:17:25.000Z

Hi, thanks for the info and offer. I don't really think this is necessary, as in normal projects, I would not rely on tags only. Was just trying it out to see if the tags are working. In reality, it will always be a combination (tags to help matching in difficult scenes) and that seems to work fine! But thanks for the assistance again and the offer! 🥇

As everything is clear now, i will close the issue. @simogasp you are the GOAT! Thanks!