Question about datasets
Opened this issue · 6 comments
Hi,
I am wondering if there are any examples where you load empirical images and do fits for those?
I've just completed section 2, and had a quick look through the data, it looks like the data used in the examples is synthetic, and the noise and psf are added/ combined into the dataset object.
I'm trying to fit empirical data, where the noise and telescope artifacts would be intrinsic to the image. Wondering if the process is similar? The al.Imaging.from_fits() function seems to require a noise path and a psf path as well as the data, but all these would be part of the empirical image? Would I instead just need an al.Array2D.from_fits() object?
Thanks,
Sam
I don't fully understand what you mean by "empirical" data?
You can indeed use al.Array2D.from_fits() to load an image by itself (e.g. without a noise_map / psf).
However, if you want to fit the data with a lens model, and compute a quantity like the chi_squared or log_likelihood, you need a noise_map in order to make this calculation valid.
I mean observational data (i.e, collected directly from telescopes/ HST, not synthesized). I.e, are the imaging datasets used in the autolens-workspace synthetic?
I'm trying to use data I've found here: https://lweb.cfa.harvard.edu/castles/#binary (this is just a quick test to get something up and running).
Your last statement is exactly what I'm looking to do, but I guess the noise is already part of the image itself, would I need to add a noise map then? :)
The majority of datasets on the workspace are synthetic indeed, just the easiest way to set it up!
There are a few real datasets here:
https://github.com/Jammy2211/autolens_workspace/tree/release/dataset/slacs
There is a guide to fitting real data here:
https://github.com/Jammy2211/autolens_workspace/tree/release/notebooks/imaging/data_preparation
You pretty much always need a noise map and PSF to model real data, let me know if you need any more help.
Aah that's great, thanks. I'll read into that first and I'll come back if I have any questions.
Do I have a misconception about the Noise Map and PSF? I initially thought we required them because a synthetic image was intended to be a precise mathematical portrayal of the lens effect, which would accurately represent the physics. Consequently, the filters are applied to introduce imperfections and make it resemble real data more closely.
However, in the case of actual data, these characteristics are present in the data itself, so does the use of a noise map appear redundant? Unless I am mistaken about the roles of the noise map and PSF, are they are utilized during the fitting process to help determine the model parameters?
(Perhaps I should read the links you sent first... :D)
When we observe a lens with something like HST, we do not know exactly what flux in each pixel we measured. There is an uncertainty in each measurement (e.g. we might measure a pixel has 0.1 e/s of flux, but the uncertainty means we have an range of values... so 0.1 +- 0.02 e/s, or something).
When we fit a lens model to this observed data, we need to fully account for these uncertainties. Otherwise, the model doesn't know whether fitting the pixel with a value of 0.09 e/s is a "good" fit (e.g. cause its consistent within the uncertaintities) or a "bad fit" (because its offset by 0.01 e/s and the uncertainities tell us it should be much closer.
Do I have a misconception about the Noise Map and PSF? I initially thought we required them because a synthetic image was intended to be a precise mathematical portrayal of the lens effect, which would accurately represent the physics. Consequently, the filters are applied to introduce imperfections and make it resemble real data more closely.
So in answer to this, it is true that noise is added to synthetic images to make them appear like the real data. However, the noise_map is used in both cases, in order to allow the model to quantify whether a fit is a good fit within the errors of each image pixel.
The PSF is a similar thing -- the observed data has a blurring effect due to the telescope optics which the PSF represents. When we fit the real data, we still have to account for this blurring effect in our model in order to ensure we get an accurate lens model fit.
Understood, thank you for for the explanation!
Aah, I see your tutorials you linked to explains it really well too!