This is a dump of the code that I most recently ran on MARS data. Warnings and disclaimers: * I'm still stuck in python2 land. Sorry guys. All of this code will need to be overhauled for python3 compatibility. * Because the MARS internet connection was crap (and because I was lazy), some of the code was copied from git repos without proper version control. Here are some of those repos: https://github.com/nivekg/marion-tools https://github.com/sievers/scio ----------------------------------------------------------------- There are two types of MARS data: 1) Directly computed auto- and cross spectra from two inputs. These data products exist at: hippo:/data/cynthia/albatros/mars2019/data_auto_cross The data are chunked in regular (hour-long, I think?) intervals. There are 5-digit directories corresponding to each day (ctime), and 10-digit ctime subdirectories within each. Within each subdirectory, the files are: - acc_cnt[12].raw : FFT accumulation counter - fpga_temp.raw : FPGA temperature - pfb_fft_of[12].raw : PFB FFT overflow counter - pi_temp.raw : Raspberry Pi temperature - pol00.scio, pol11.scio : autospectra of inputs 0 and 1 (2D array) - pol01r.scio, pol01i.scio : real and imaginary parts of cross spectra (2D array) - sync_cnt[12].raw : some FPGA-related counter, I forget the details - sys_clkcounter[12].raw : some FPGA-related counter, I forget the details - time_gps_start.raw, time_gps_stop.raw : GPS time stamps for the start and stop of each accumulation 2) Baseband data from two inputs. These data products will soon exist on scinet. The directory structure is the same as above, and I...forget the exact details of the data files because I don't have any on hand right now. ----------------------------------------------------------------- Code that's included here: * albatrostools.py : Tools for reading baseband data. We suspect there's a lingering bug in the bit unpacking that was never fully sorted out, so this needs testing and fixing. * coarse_cross.py : Calculate auto- and cross-spectra from baseband, coarsely sampled. If everything is working correctly, the spectra that are derived from the baseband data should be the same as those that are directly computed and saved by the DAQ. * coarse_hist.py : Plots coarsely sampled baseband histograms for 2- and 4-bit data. Use this to verify that all the bits are being properly used and that levels are set correctly. * find_baseband_runs.py : Troll the log files to find baseband runs and their parameter settings. * plot_all.py : Plot directly computed auto- and cross-spectra, concatenated over a specified length of time. * plot_overnight_data.py : Plot directly computed auto- and cross-spectra for all individual subdirectories within a specified length of time. * scio.py : Tools for reading directly computed auto- and cross-spectra * SNAPfiletools.py : Miscellaneous functions for handling data and timestamps * timestamp_loginfo.py : Troll the log files to find system information for a particular time stamp. * utc_ls.py : Poorly written script for converting ctimes into human-readable timestamps. The globbing is fragile and sometimes needs coaxing in order to work. ----------------------------------------------------------------- Other useful files: * logs/ : ALL of the log files. Use these in conjunction with timestamp_loginfo.py and find_baseband_runs.py above. * events.txt : Machine-readable file that describes the system state as a function of time. * notes.txt : Human's log file recording the system states and other idiosyncrasies. events.txt is derived from this. All of our long, good runs of data were taken overnight, when we turned off power at the base (and all the sources of RFI, like the inverters and solar chargers). If you have questions about any of the daytime data and the state of the system, send me a message and I can help you decipher notes.txt. ----------------------------------------------------------------- Note - this version uses (optionally) an improved C-based 4bit unpacker (other bit depths to come). This means you need to compile the C code. A sample compile line, which should hopefully be transparantely translateable to other systems, is at the top of albatrostools.c