TurboSETI not finding bright drifting signal
Closed this issue · 10 comments
This is data made available here:
https://storage.googleapis.com/ata_test_data/turbo_seti_test/guppi_59196_68468_000762_Unknown_0001.rawspec.0000.h5
Plotting using blimpy:
watutil -b 8470.985 -e 8471.010 -p a -l 4 guppi_59196_68468_000762_Unknown_0001.rawspec.0000.h5
The signal has a driftrate ~ -0.8 Hz/sec
Running turboSETI:
turboSETI -l debug -n 256 guppi_59196_68468_000762_Unknown_0001.rawspec.0000.h5 -g y
(note the 256 number, the data were finely channelised using rawspec)
I do not see the candidate in the hits. The only candidates turboSETI finds are in the range 8437-8439 MHz
Attached is a zip log of running
turboSETI /seti_data/turbo_seti_issue_135/*.h5 -n 256 -o /tmp/run_turbo_seti/ -l debug | tee issue_235.log.
I need help in interpreting this debug log (28 MB unzipped). Hopefully, there is enough debug logging information present. If more is needed, the find_doppler.py can be amended to provide more debug logging.
It appears that turboSETI is missing drift rates between -0.9 <-> 0 🤔 ? (the signal at hand is ~-0.8 Hz/s)
Drift_block = -1:
find_doppler.131 DEBUG Drift_block -1 (in range -10 to 11)
find_doppler.131 DEBUG Un-flipped corrected negative drift_block
find_doppler.131 DEBUG drift_block < 0: sub_range=[-1.92509712 -1.90993887 -1.89478063 -1.87962238 -1.86446414 -1.84930589
-1.83414765 -1.8189894 -1.80383116 -1.78867291 -1.77351467 -1.75835642
-1.74319818 -1.72803993 -1.71288169 -1.69772344 -1.6825652 -1.66740695
-1.65224871 -1.63709046 -1.62193222 -1.60677397 -1.59161573 -1.57645748
-1.56129924 -1.54614099 -1.53098275 -1.5158245 -1.50066626 -1.48550801
-1.47034977 -1.45519152 -1.44003328 -1.42487503 -1.40971679 -1.39455854
-1.3794003 -1.36424205 -1.34908381 -1.33392556 -1.31876732 -1.30360907
-1.28845083 -1.27329258 -1.25813434 -1.24297609 -1.22781785 -1.2126596
-1.19750136 -1.18234311 -1.16718487 -1.15202662 -1.13686838 -1.12171013
-1.10655189 -1.09139364 -1.0762354 -1.06107715 -1.04591891 -1.03076066
-1.01560242 -1.00044417 -0.98528593 -0.97012768]
but then, drift block = 0:
find_doppler.131 DEBUG Drift_block 0 (in range -10 to 11)
find_doppler.131 DEBUG drift_block > 0: sub_range=[0. 0.01515825 0.03031649 0.04547474 0.06063298 0.07579123
0.09094947 0.10610772 0.12126596 0.13642421 0.15158245 0.1667407
0.18189894 0.19705719 0.21221543 0.22737368 0.24253192 0.25769017
0.27284841 0.28800666 0.3031649 0.31832315 0.33348139 0.34863964
0.36379788 0.37895613 0.39411437 0.40927262 0.42443086 0.43958911
0.45474735 0.4699056 0.48506384 0.50022209 0.51538033 0.53053858
0.54569682 0.56085507 0.57601331 0.59117156 0.6063298 0.62148805
0.63664629 0.65180454 0.66696278 0.68212103 0.69727927 0.71243752
0.72759576 0.74275401 0.75791225 0.7730705 0.78822874 0.80338699
0.81854523 0.83370348 0.84886172 0.86401997 0.87917821 0.89433646
0.9094947 0.92465295 0.93981119 0.95496944]
The above comment is a bug that needs fixing, but it does not explain this issue. It turns out that turboSETI was actually finding the correct candidates, but reporting the wrong freq_start and freq_end.
This is the ".dat" file output:
# -------------------------- o --------------------------
# File ID: guppi_59196_68468_000762_Unknown_0001.rawspec.0000.h5
# -------------------------- o --------------------------
# Source:Unknown
# MJD: 59196.792453703703 RA: 0h00m00s DEC: 0d00m00s
# DELTAT: 2.097152 DELTAF(Hz): 1.907349
# --------------------------
# Top_Hit_# Drift_Rate SNR Uncorrected_Frequency Corrected_Frequency Index freq_start freq_end SEFD SEFD_freq Coarse_Channel_Number Full_number_of_hits
# --------------------------
001 -0.970128 210.425807 8437.745037 8437.745037 62934 8437.743755 8437.746315 0.0 0.000000 56 1103
002 -0.970128 140.126839 8437.647305 8437.647305 11694 8437.646023 8437.648582 0.0 0.000000 118 321
003 0.000000 208.999073 8437.750000 8437.750000 65536 8437.748718 8437.751278 0.0 0.000000 128 60
004 -0.970128 365.891984 8437.747871 8437.747871 64420 8437.746590 8437.749149 0.0 0.000000 131 9268
005 -0.970128 211.403714 8437.740122 8437.740122 60357 8437.738840 8437.741400 0.0 0.000000 133 151325
006 -0.970128 4735.457039 8437.747871 8437.747871 64420 8437.746590 8437.749149 0.0 0.000000 133 151325
007 -0.970128 206.242927 8437.755623 8437.755623 68484 8437.754341 8437.756901 0.0 0.000000 133 151325
008 -0.970128 383.383817 8437.747873 8437.747873 64421 8437.746592 8437.749151 0.0 0.000000 135 13730
009 -0.970128 156.214405 8437.848440 8437.848440 117147 8437.847158 8437.849718 0.0 0.000000 148 358
010 -0.970128 208.447802 8437.750708 8437.750708 65907 8437.749426 8437.751986 0.0 0.000000 210 1030
If you look at the "channel_number" column, turboSETI is reporting candidates in the coarse channel that I expect my signal to be (i.e. coarse_chan = 133, the rest of the candidates are harmonics).
The freq_start and freq_end are wrongly reported. I think this is because the report_tophit function (https://github.com/UCBerkeleySETI/turbo_seti/blob/master/turbo_seti/find_doppler/file_writers.py#L166) is passed the ind_tuple which describes which fine channel the top-hit is within a coarse channel, and not the absolute fine channel (given we have multiple coarse channels).
That function needs to somehow know which coarse channel it is reporting the hit from, such that it can properly calculate what correct sky frequency it is
Ran a fixed up version of 1.3.0 and the DAT file makes more sense per @wfarah
@luigifcruz - There seems to be something wrong in find_doppler.py.
This is the correct DAT file output, found by turbo_seti version 1.3.0:
# -------------------------- o --------------------------
# File ID: guppi_59196_68468_000762_Unknown_0001.rawspec.0000.h5
# -------------------------- o --------------------------
# Source:Unknown
# MJD: 59196.792453703703 RA: 0h00m00s DEC: 0d00m00s
# DELTAT: 2.097152 DELTAF(Hz): 1.907349
# --------------------------
# Top_Hit_# Drift_Rate SNR Uncorrected_Frequency Corrected_Frequency Index freq_start freq_end SEFD SEFD_freq Coarse_Channel_Number Full_number_of_hits
# --------------------------
001 -0.894336 54.937101 8451.245031 8451.245031 62931 8451.243750 8451.246309 0.0 0.000000 54 14
002 -0.879178 29.750972 8451.737282 8451.737282 58868 8451.736000 8451.738560 0.0 0.000000 56 1985
003 -0.879178 631.124467 8451.745031 8451.745031 62931 8451.743750 8451.746309 0.0 0.000000 56 1985
004 -0.894336 30.741016 8451.752783 8451.752783 66995 8451.751501 8451.754061 0.0 0.000000 56 1985
005 -0.894336 53.702237 8452.245033 8452.245033 62932 8452.243752 8452.246311 0.0 0.000000 58 15
006 -0.909495 31.702241 8466.647301 8466.647301 11692 8466.646019 8466.648579 0.0 0.000000 116 4
007 -0.894336 389.609477 8467.147301 8467.147301 11692 8467.146019 8467.148579 0.0 0.000000 118 674
008 -0.909495 32.834768 8467.647303 8467.647303 11693 8467.646021 8467.648581 0.0 0.000000 120 4
009 0.000000 208.979572 8469.750000 8469.750000 65536 8469.748718 8469.751278 0.0 0.000000 128 119
010 -0.909495 33.927765 8470.481869 8470.481869 56030 8470.480587 8470.483147 0.0 0.000000 131 10934
011 -0.879178 30.955185 8470.490116 8470.490116 60354 8470.488834 8470.491394 0.0 0.000000 131 10934
012 -0.894336 1076.299827 8470.497868 8470.497868 64418 8470.496586 8470.499146 0.0 0.000000 131 10934
013 -0.894336 30.882870 8470.505617 8470.505617 68481 8470.504335 8470.506895 0.0 0.000000 131 10934
014 -0.879178 33.729899 8470.513866 8470.513866 72806 8470.512585 8470.515144 0.0 0.000000 131 10934
015 -0.909495 47.439492 8470.866798 8470.866798 126772 8470.865517 8470.868076 0.0 0.000000 132 10
016 -0.894336 609.250965 8470.990118 8470.990118 60355 8470.988836 8470.991396 0.0 0.000000 133 156692
017 -0.879178 13657.062090 8470.997868 8470.997868 64418 8470.996586 8470.999146 0.0 0.000000 133 156692
018 -0.909495 569.279543 8471.005619 8471.005619 68482 8471.004337 8471.006897 0.0 0.000000 133 156692
019 -0.894336 34.893923 8471.128939 8471.128939 2065 8471.127657 8471.130217 0.0 0.000000 134 4
020 -0.909495 30.772719 8471.481871 8471.481871 56031 8471.480589 8471.483149 0.0 0.000000 135 15386
021 -0.894336 1107.738870 8471.497869 8471.497869 64419 8471.496588 8471.499147 0.0 0.000000 135 15386
022 -0.879178 29.932427 8471.513868 8471.513868 72807 8471.512587 8471.515146 0.0 0.000000 135 15386
023 -0.894336 39.826079 8474.348434 8474.348434 117144 8474.347153 8474.349712 0.0 0.000000 146 5
024 -0.909495 446.308926 8474.848436 8474.848436 117145 8474.847155 8474.849714 0.0 0.000000 148 927
025 -0.894336 43.308256 8475.348436 8475.348436 117145 8475.347155 8475.349714 0.0 0.000000 150 5
026 -0.894336 53.883013 8489.750704 8489.750704 65905 8489.749422 8489.751982 0.0 0.000000 208 14
027 -0.894336 34.197546 8490.242954 8490.242954 61842 8490.241673 8490.244232 0.0 0.000000 210 1884
028 -0.894336 561.703912 8490.250704 8490.250704 65905 8490.249422 8490.251982 0.0 0.000000 210 1884
029 -0.909495 30.978648 8490.258455 8490.258455 69969 8490.257174 8490.259733 0.0 0.000000 210 1884
030 -0.894336 53.602007 8490.750706 8490.750706 65906 8490.749424 8490.751984 0.0 0.000000 212 14
As you can see, coarse channel 133 has a huge SNR value. This hit is missed by turbo_seti 2.0.2.
@texadactyl there are probably some misses because the search between -0.9 -> 0 drift rates was not being done.
The PR #137 fixes the problems reported here by reverting PR #121 and #113.
# -------------------------- o --------------------------
# File ID: guppi_59196_68468_000762_Unknown_0001.rawspec.0000.h5
# -------------------------- o --------------------------
# Source:Unknown
# MJD: 59196.792453703703 RA: 0h00m00s DEC: 0d00m00s
# DELTAT: 2.097152 DELTAF(Hz): 1.907349
# --------------------------
# Top_Hit_# Drift_Rate SNR Uncorrected_Frequency Corrected_Frequency Index freq_start freq_end SEFD SEFD_freq Coarse_Channel_Number Full_number_of_hits
# --------------------------
001 -0.894336 54.942876 8451.245031 8451.245031 62931 8451.243750 8451.246309 0.0 0.000000 54 14
002 -0.879178 29.753387 8451.737282 8451.737282 58868 8451.736000 8451.738560 0.0 0.000000 56 1985
003 -0.879178 631.164027 8451.745031 8451.745031 62931 8451.743750 8451.746309 0.0 0.000000 56 1985
004 -0.894336 30.743493 8451.752783 8451.752783 66995 8451.751501 8451.754061 0.0 0.000000 56 1985
005 -0.894336 53.706244 8452.245033 8452.245033 62932 8452.243752 8452.246311 0.0 0.000000 58 15
006 -0.909495 31.704836 8466.647301 8466.647301 11692 8466.646019 8466.648579 0.0 0.000000 116 4
007 -0.894336 389.637482 8467.147301 8467.147301 11692 8467.146019 8467.148579 0.0 0.000000 118 675
008 -0.909495 32.840153 8467.647303 8467.647303 11693 8467.646021 8467.648581 0.0 0.000000 120 4
009 -0.015158 113.986064 8469.750000 8469.750000 65536 8469.748718 8469.751278 0.0 0.000000 128 59
010 -0.909495 33.931910 8470.481869 8470.481869 56030 8470.480587 8470.483147 0.0 0.000000 131 8681
011 -0.879178 30.959041 8470.490116 8470.490116 60354 8470.488834 8470.491394 0.0 0.000000 131 8681
012 -0.894336 1076.405771 8470.497868 8470.497868 64418 8470.496586 8470.499146 0.0 0.000000 131 8681
013 -0.894336 30.886719 8470.505617 8470.505617 68481 8470.504335 8470.506895 0.0 0.000000 131 8681
014 -0.879178 33.734026 8470.513866 8470.513866 72806 8470.512585 8470.515144 0.0 0.000000 131 8681
015 -0.909495 46.724844 8470.866798 8470.866798 126772 8470.865517 8470.868076 0.0 0.000000 132 7
016 -0.894336 609.307646 8470.990118 8470.990118 60355 8470.988836 8470.991396 0.0 0.000000 133 151147
017 -0.879178 13658.318576 8470.997868 8470.997868 64418 8470.996586 8470.999146 0.0 0.000000 133 151147
018 -0.909495 569.332548 8471.005619 8471.005619 68482 8471.004337 8471.006897 0.0 0.000000 133 151147
019 -0.894336 35.571154 8471.128939 8471.128939 2065 8471.127657 8471.130217 0.0 0.000000 134 4
020 -0.909495 30.775063 8471.481871 8471.481871 56031 8471.480589 8471.483149 0.0 0.000000 135 12916
021 -0.894336 1107.803066 8471.497869 8471.497869 64419 8471.496588 8471.499147 0.0 0.000000 135 12916
022 -0.879178 29.934723 8471.513868 8471.513868 72807 8471.512587 8471.515146 0.0 0.000000 135 12916
023 -0.894336 39.831350 8474.348434 8474.348434 117144 8474.347153 8474.349712 0.0 0.000000 146 5
024 -0.909495 446.319275 8474.848436 8474.848436 117145 8474.847155 8474.849714 0.0 0.000000 148 927
025 -0.894336 43.312686 8475.348436 8475.348436 117145 8475.347155 8475.349714 0.0 0.000000 150 5
026 -0.894336 53.886039 8489.750704 8489.750704 65905 8489.749422 8489.751982 0.0 0.000000 208 14
027 -0.894336 34.201170 8490.242954 8490.242954 61842 8490.241673 8490.244232 0.0 0.000000 210 1884
028 -0.894336 561.750127 8490.250704 8490.250704 65905 8490.249422 8490.251982 0.0 0.000000 210 1884
029 -0.909495 30.982012 8490.258455 8490.258455 69969 8490.257174 8490.259733 0.0 0.000000 210 1884
030 -0.894336 53.609371 8490.750706 8490.750706 65906 8490.749424 8490.751984 0.0 0.000000 212 14
Verified.
Not forgotten!
See issue #141.
Thanks!