xmipp3 FlexAlign failed
Closed this issue · 3 comments
I'm just getting started with SCIPION and I have encountered the following error.
I am following the tutorial that can be found here --> (file:///home/em-group/Downloads/tutotial_scipion3_apoferritin.pdf)
I run the "import movies" and "movie gain" and nothing strange seems to happen.
But when I run the "flex align" it fails.
This is the run.stdout
0001: Logging configured. STDOUT --> Runs/000303_XmippProtMovieCorr/logs/run.stdout , STDERR --> Runs/000303_XmippProtMovieCorr/logs/run.stderr
00002: RUNNING PROTOCOL -----------------
00003: Protocol starts
00004: Hostname: Cryo-EM-Worktsation
00005: PID: 1376763
00006: pyworkflow: 3.0.17
00007: plugin: xmipp3
00008: currentDir: /home/em-group/ScipionUserData/Mazzotta/ApoFer
00009: workingDir: Runs/000303_XmippProtMovieCorr
00010: runMode: Continue
00011: MPI: 1
00012: threads: 1
00013: Starting at step: 1
00014: Running steps
00015: STARTED: _convertInputStep, step 1, time 2021-10-10 16:25:07.722353
00016: FINISHED: _convertInputStep, step 1, time 2021-10-10 16:25:07.727931
00017: STARTED: processMovieStep, step 2, time 2021-10-10 16:25:07.738015
00018: Processing movie: Runs/000303_XmippProtMovieCorr/tmp/movie_000001/190117_02_00001_UnderDefocus0.7um_frameImage.tiff
00019: xmipp_cuda_movie_alignment_correlation -i "Runs/000303_XmippProtMovieCorr/tmp/movie_000001/input_movie.xmd" -o "Runs/000303_XmippProtMovieCorr/extra/190117_02_00001_UnderDefocus0.7um_frameImage_shifts.xmd" --sampling 0.495000 --maxResForCorrelation 30.000000 --Bspline 3 --outside wrap --frameRange 0 49 --frameRangeSum 0 49 --max_shift 30 --oavg "Runs/000303_XmippProtMovieCorr/extra/190117_02_00001_UnderDefocus0.7um_frameImage_aligned_mic.mrc" --oavgInitial "Runs/000303_XmippProtMovieCorr/tmp/movie_000001/initialMic.mrc" --gain "Runs/000303_XmippProtMovieCorr/extra/orientedGain.mrc" --minLocalRes 500.000000 --device 0 --processLocalShifts --storage "Runs/000303_XmippProtMovieCorr/extra/fftBenchmark.txt" --controlPoints 9 9 10 --patchesAvg 3
00020: Input movie: Runs/000303_XmippProtMovieCorr/tmp/movie_000001/input_movie.xmd
00021: Output metadata: Runs/000303_XmippProtMovieCorr/extra/190117_02_00001_UnderDefocus0.7um_frameImage_shifts.xmd
00022: Dark image:
00023: Gain image: Runs/000303_XmippProtMovieCorr/extra/orientedGain.mrc
00024: Max. Shift: 30
00025: Max resolution (A): 30
00026: Sampling: 0.495
00027: Solver iterations: 2
00028: Aligned movie:
00029: Aligned micrograph: Runs/000303_XmippProtMovieCorr/extra/190117_02_00001_UnderDefocus0.7um_frameImage_aligned_mic.mrc
00030: Unaligned micrograph: Runs/000303_XmippProtMovieCorr/tmp/movie_000001/initialMic.mrc
00031: Frame range alignment: 0 49
00032: Frame range sum: 0 49
00033: Crop corners (0, 0) (-1, -1)
00034: Use input shifts: 0
00035: Output Binning factor: -1
00036: Bspline: 3
00037: Local shift correction: yes
00038: Control points: 9 * 9 * 1 * 10
00039: Device: 0 (GPU-1634eba0-9769-7002-af74-1da3207a3da2)
00040: Benchmark storage Runs/000303_XmippProtMovieCorr/extra/fftBenchmark.txt
00041: Patches avg: 3
00042: Autotuning: on
00043: Computing global alignment ...
00044: rank|placeness|type|data|X|Y|Z|batch|plan size estimate (MB)|plan size estimate 2 (MB)|plan size actual (MB)|data size (MB)|plan time (ms)|exec time (ms/signal)|exec time of batch (ms)|elements|performance (1k elem of signal/ms)|error message
00045: many|out-of-place|R2R forward|float|3584|3584|1|50|4901.3671875000|4901.3671875000|4901.3671875000|4901.3671875000|3.9909999371|0.6767640710|33.8382034302|642252800|18980.1093750000|
00046: rank|placeness|type|data|X|Y|Z|batch|plan size estimate (MB)|plan size estimate 2 (MB)|plan size actual (MB)|data size (MB)|plan time (ms)|exec time (ms/signal)|exec time of batch (ms)|elements|performance (1k elem of signal/ms)|error message
00047: many|out-of-place|R2R forward|float|432|432|1|1225|1748.2275390625|1748.2275390625|1748.2275390625|1748.2269287109|3.4070000648|0.0037011430|4.5339002609|228614400|50423.3398437500|
00048: Requested scale factor: 0.11211
00049: Actual scale factor (X): 0.120536
00050: Settings for the movie: 3584(1793) * 3584 * 1 * 50, batch: 50, inPlace: no
00051: Settings for the correlation: 432(217) * 432 * 1 * 1225, batch: 1225, inPlace: no
00052: Reference frame: 25
00053: Estimated global shifts (from the reference frame):
00054: X: 17.2953 Y: -5.9634
00055: X: 14.3114 Y: -10.1582
00056: X: 13.0880 Y: -10.7898
00057: X: 13.0037 Y: -9.0692
00058: X: 12.7511 Y: -8.8825
00059: X: 11.9175 Y: -8.9945
00060: X: 11.0651 Y: -8.5285
00061: X: 09.8042 Y: -8.1498
00062: X: 09.6675 Y: -7.0224
00063: X: 09.5900 Y: -6.7392
00064: X: 08.9316 Y: -5.3476
00065: X: 09.2842 Y: -4.9210
00066: X: 07.1799 Y: -5.0082
00067: X: 08.9685 Y: -2.4785
00068: X: 09.8270 Y: -0.2851
00069: X: 07.4875 Y: -1.5587
00070: X: 06.1224 Y: -1.3682
00071: X: 04.1338 Y: -1.2065
00072: X: 04.7149 Y: -0.1178
00073: X: 03.9917 Y: -0.8221
00074: X: 03.6662 Y: -0.2726
00075: X: 02.4825 Y: 01.1774
00076: X: 01.5612 Y: -0.0751
00077: X: 00.6820 Y: -0.4041
00078: X: 02.2072 Y: 01.1533
00079: X: 00.0000 Y: 00.0000
00080: X: -1.3069 Y: 01.3639
00081: X: -2.6441 Y: 03.3311
00082: X: -3.8781 Y: 02.8747
00083: X: -4.4857 Y: 01.2247
00084: X: -4.3198 Y: 01.0124
00085: X: -5.3704 Y: 03.6017
00086: X: -6.5956 Y: 03.2008
00087: X: -7.8508 Y: 03.9878
00088: X: -8.0729 Y: 05.7595
00089: X: -8.6737 Y: 07.4383
00090: X: -9.8171 Y: 07.3239
00091: X: -11.5210 Y: 05.7940
00092: X: -11.2977 Y: 06.5646
00093: X: -12.8008 Y: 07.6063
00094: X: -12.2584 Y: 08.3250
00095: X: -13.9987 Y: 07.7452
00096: X: -15.5791 Y: 07.4312
00097: X: -13.9455 Y: 07.2488
00098: X: -14.9580 Y: 06.9518
00099: X: -16.2124 Y: 08.5430
00100: X: -15.8834 Y: 06.1707
00101: X: -17.4042 Y: 03.9010
00102: X: -16.8344 Y: 06.3778
00103: X: -17.6764 Y: 05.1031
00104:
00105: Computing local alignment ...
00106: rank|placeness|type|data|X|Y|Z|batch|plan size estimate (MB)|plan size estimate 2 (MB)|plan size actual (MB)|data size (MB)|plan time (ms)|exec time (ms/signal)|exec time of batch (ms)|elements|performance (1k elem of signal/ms)|error message
00107: many|out-of-place|R2R forward|float|1024|1024|1|50|400.3906250000|400.3906250000|400.3906250000|400.3906250000|7.3730001450|0.0268760044|1.3438001871|52428800|39015.3242187500|
00108: Using 4 x 4 patches
00109: rank|placeness|type|data|X|Y|Z|batch|plan size estimate (MB)|plan size estimate 2 (MB)|plan size actual (MB)|data size (MB)|plan time (ms)|exec time (ms/signal)|exec time of batch (ms)|elements|performance (1k elem of signal/ms)|error message
00110: many|out-of-place|R2R forward|float|120|120|1|1225|135.7043457031|135.7043457031|135.7043457031|135.7040405273|0.3740000129|0.0004480816|0.5489000082|17640000|32137.0000000000|
00111: No. of patches: 4 x 4
00112: Actual scale factor (X): 0.117188
00113: Settings for the patches: 1024(513) * 1024 * 1 * 50, batch: 50, inPlace: no
00114: Settings for the correlation: 120(61) * 120 * 1 * 1225, batch: 1225, inPlace: no
00115: Computing BSpline coefficients
00116: Solving equation system ...
00117: Iteration 0 R2x=-2.85744e+43 R2y=-3.70516e+43
00118: Found 16 outliers
00119:
00120: Iteration 1 R2x=-2.33641e+46 R2y=-1.64359e+46
00121: We cannot process Runs/000303_XmippProtMovieCorr/tmp/movie_000001/190117_02_00001_UnderDefocus0.7um_frameImage.tiff
00122: Command ' xmipp_cuda_movie_alignment_correlation -i "Runs/000303_XmippProtMovieCorr/tmp/movie_000001/input_movie.xmd" -o "Runs/000303_XmippProtMovieCorr/extra/190117_02_00001_UnderDefocus0.7um_frameImage_shifts.xmd" --sampling 0.495000 --maxResForCorrelation 30.000000 --Bspline 3 --outside wrap --frameRange 0 49 --frameRangeSum 0 49 --max_shift 30 --oavg "Runs/000303_XmippProtMovieCorr/extra/190117_02_00001_UnderDefocus0.7um_frameImage_aligned_mic.mrc" --oavgInitial "Runs/000303_XmippProtMovieCorr/tmp/movie_000001/initialMic.mrc" --gain "Runs/000303_XmippProtMovieCorr/extra/orientedGain.mrc" --minLocalRes 500.000000 --device 0 --processLocalShifts --storage "Runs/000303_XmippProtMovieCorr/extra/fftBenchmark.txt" --controlPoints 9 9 10 --patchesAvg 3' returned non-zero exit status 134.
00123: Erasing.....movieFolder: Runs/000303_XmippProtMovieCorr/tmp/movie_000001
00124: FINISHED: processMovieStep, step 2, time 2021-10-10 16:26:07.071823
Here is where the errors start
00125: ERROR: Movie Runs/000002_ProtImportMovies/extra/190117_02_00001_UnderDefocus0.7um_frameImage.tiff couldn't be added to the output set.
00126: Block: 'frameShifts': Runs/000303_XmippProtMovieCorr/extra/190117_02_00001_UnderDefocus0.7um_frameImage_shifts.xmd
00127: WARNING: Micrograph Runs/000303_XmippProtMovieCorr/extra/190117_02_00001_UnderDefocus0.7um_frameImage_aligned_mic.mrc was not generated, can't add it to output set.
While the run.stderr says:
00006: terminate called after throwing an instance of 'std::runtime_error' 00007: what(): GPUassert: an illegal memory access was encountered libraries/reconstruction_cuda/cuda_gpu_geo_transformer.cpp 227 00008: Aborted (core dumped)
Hi.
We're aware of this issue. It is already fixed in the new version, which will be released probably next week.
The quick workaround is to decrease the number of control points in the protocol.
For that, you have to switch Expert Level to Advanced, scroll down to Local Alignment group, switch Auto control points to No and set lower values for X and Y. You want one control point per 1000A.
Please let me know if that works for you.
See also: I2PC/xmipp#415
Thanks! Now it worked! I inserted 3 for x and y and 5 for t. I'm not sure what exactly it did but it gave me the output :)
OK, I will close this issue as it should be fixed in the next release (available since tomorrow).