Mismatch between posteriors from polychord vs importance sampler

Question

Mismatch between posteriors from polychord vs importance sampler

jessmuir opened this issue 7 months ago · 5 comments

Hello! I've been investigating something that was initially flagged by Noah (@nweaverd), where the posterior reported by the importance sampler and polychord sampler slightly differ in some cases even when they are run with the exact same cosmosis pipeline.

The context is that occasionally we've been running an importance sampler on chains as an approach to save model predictions for datavectors that can be used for making plots, PPD calculations, and some validation tests. In principle, doing this with the exact same cosmosis pipeline, same data, same scale cuts, should produce importance sampler output where all the log_weight entries are 0, meaning the IS post is identical to the original sampler's old_post. We're finding that this isn't always the case.

(There's a thread on this in the cosmosis channel of the DES slack team; let me know if you want to be pointed to that.)

We're still not sure what's happening, but compiling notes on what we've checked so far:

The list and importance samplers produce matching posteriors.
Including or not including extra_output columns in either the original or IS chain doesn't affect this.
Noah did some extensive investigation looking at whether the IS log_weight correlates with anything; it doesn't have any obvious correlation with post, like, prior, old_post, or the handful of individual parameters we checked.
Looking back at DES Y3 chains where we used this, and re-checking with some newer simulated Y6 chains we do see differences in the polychord and IS posts. Generally more constraining chains show less differences. E.g. Planck+BAO+SN has smaller missmatches than Planck alone, DES+external is smaller than DES alone. However, 3x2pt has bigger differences than shear only.
I don't see this mismatch when I ran polychord and then importance sampler for the cosmosis example2 setup for the pantheon SNe likelihood.
I don't see this mismatch when I ran a simplified DES chain sampling only A_s and Omega_m.
I do see the mismatch when I ran a simplified DES chain sampling only one of the shear calibration parameters. This makes me think that this may have something to do with how priors are evaluated, specifically parameters with Gaussian priors. This might explain why 3x2pt shows bigger differences than shear only, since there are more nuisance parameters with non-flat priors.
However, when I try adding a Gaussian prior to parameters for the pantheon SNe chain, it doesn't affect agreement between polychord and IS. So if this is related to Gaussian priors, it's not only that.

We have a couple more ideas of things to check. I'll add notes here as we get to them.

I know that running a test sampler handles priors differently that running a sampler with varying parameters, so it seems plausible that there might be worth scrutinizing how the importance and list samplers do this, but I don't have a good enough understanding to be sure.

Answer 1 · 2024-05-29T08:02:12.000Z

I haven't been able to replicate this directly. Could you confirm from looking at the chain headers that the exact same cosmosis-standard-library version was used in each case? And perhaps give me some more detail on exactly what you ran to replicate?

Answer 2 · 2024-05-29T10:21:36.000Z

Hm. I just attempted to reproduce the vary-m1-only result on my laptop, and am finding matching polychord and IS posteriors.

Since I'd previously ran the test on my office computer, I won't have access to those output files for another week or so (I'm traveling). I'm relatively sure that in that case I was using the same CSL version since I ran polychord then IS in succession with no code updates between them.

Previously I'd seen mismatches when using IS to save extra derived parameters for DES Y3 chains. I'll aim to hunt down the details on those within the next few days.

Answer 3 · 2024-06-13T21:05:01.000Z

Sorry for the delayed response to this! I was actually running with Nautilus rather than Polychord (an on NERSC). I was worried about different versions so ran a new Nautilus chain and then ran the Importance Sampler using the exact same params.ini, just changing the sampler, with only a few hours in between and no code updates.

Here's a portion of the Nautilus output header:

#cosmological_parameters--omega_m	cosmological_parameters--h0	cosmological_parameters--omega_b	cosmological_parameters--n_s	cosmological_parameters--a_s	cosmological_parameters--mnu	shear_calibration_parameters--m1	shear_calibration_parameters--m2	shear_calibration_parameters--m3	shear_calibration_parameters--m4	wl_photoz_errors--bias_1	wl_photoz_errors--bias_2	wl_photoz_errors--bias_3	wl_photoz_errors--bias_4	lens_photoz_errors--bias_1	lens_photoz_errors--bias_2	lens_photoz_errors--bias_3	lens_photoz_errors--bias_4	lens_photoz_errors--width_1	lens_photoz_errors--width_2	lens_photoz_errors--width_3	lens_photoz_errors--width_4	bias_lens--b1	bias_lens--b2	bias_lens--b3	bias_lens--b4	intrinsic_alignment_parameters--a1	intrinsic_alignment_parameters--a2	intrinsic_alignment_parameters--alpha1	intrinsic_alignment_parameters--alpha2	intrinsic_alignment_parameters--bias_ta	COSMOLOGICAL_PARAMETERS--SIGMA_8	COSMOLOGICAL_PARAMETERS--SIGMA_12	DATA_VECTOR--2PT_CHI2	log_weight	prior	post
#sampler=nautilus
#n_varied=31
#module_0=consistency 1.1
#module_1=BBN-Consistency 0705.0290
## CITE Comp.Phys.Commun.178:956-971,2008
#module_2=camb 2.0
## CITE http://arxiv.org/abs/1201.3654
## CITE http://arxiv.org/abs/astro-ph/9911177
#module_3=fast_pt 1
## CITE https://iopscience.iop.org/article/10.1088/1475-7516/2016/09/015
#module_4=load_nz_fits 1
#module_5=photoz_bias 1
#module_6=photoz_bias 1
#module_7=photoz_bias 1
#module_8=IA #name from ini file
#module_9=project_2d 1.0
#module_10=project_2d 1.0
#module_11=add_magnification 1
#module_12=add_intrinsic 1.0
#module_13=2pt_shear #name from ini file
#module_14=2pt_gal #name from ini file
#module_15=2pt_gal_shear #name from ini file
#module_16=shear_bias 1
#module_17=add_point_mass #name from ini file
#module_18=2pt_like #name from ini file
#module_19=shear_ratio_like #name from ini file
#timestamp=2024-02-16T04:25:34.409664
#platform=Linux-5.14.21-150400.24.81_12.0.87-cray_shasta_c-x86_64-with-glibc2.31
#platform_version=#1 SMP Thu Jan 11 20:03:21 UTC 2024 (e30c7c1)
#uuid=3d638c3adfce4848b081c7d452b55670
#cosmosis_git_version=
#csl_git_version=
#cwd_git_version=6cd5ddd11b7ef6219eff042ecef8a85dbd2c6510
#hostname=nid004207
#username=nweaverd
#workdir=/global/cfs/cdirs/des/nweaverd/cosmosis-standard-library
#n_live=1024
#n_update=1024
#enlarge_per_dim=1.1
#n_points_min=81
#split_threshold=100.0
#n_networks=4
#n_batch=256
#seed=-1
#resume=True
#f_live=0.01
#n_shell=512
#n_eff=10000.0
#discard_exploration=False
#verbose=True
## START_OF_PARAMS_INI
## [DEFAULT]
## 2pt_file = likelihood/des-y3/2pt_NG_final_2ptunblind_02_26_21_wnz_maglim_covupdate.fits
## 
## [runtime]
## sampler = nautilus
## verbosity = standard
## resume = T
## 
## [pipeline]
## modules = consistency  bbn_consistency 	camb  fast_pt 	fits_nz  lens_photoz_width  lens_photoz_bias  source_photoz_bias 	IA  pk_to_cl_gg  pk_to_cl 	add_magnification  add_intrinsic 	2pt_shear 2pt_gal  2pt_gal_shear 	shear_m_bias   add_point_mass 	2pt_like  shear_ratio_like
## timing = F
## debug = F
## priors = examples/des-y3-maglim-priors.ini
## values = examples/des-y3-maglim-values.ini
## extra_output = cosmological_parameters/sigma_8 cosmological_parameters/sigma_12 data_vector/2pt_chi2
## fast_slow = F
## first_fast_module = shear_m_bias

And here is the corresponding IS one:

#cosmological_parameters--omega_m	cosmological_parameters--h0	cosmological_parameters--omega_b	cosmological_parameters--n_s	cosmological_parameters--a_s	cosmological_parameters--mnu	shear_calibration_parameters--m1	shear_calibration_parameters--m2	shear_calibration_parameters--m3	shear_calibration_parameters--m4	wl_photoz_errors--bias_1	wl_photoz_errors--bias_2	wl_photoz_errors--bias_3	wl_photoz_errors--bias_4	lens_photoz_errors--bias_1	lens_photoz_errors--bias_2	lens_photoz_errors--bias_3	lens_photoz_errors--bias_4	lens_photoz_errors--width_1	lens_photoz_errors--width_2	lens_photoz_errors--width_3	lens_photoz_errors--width_4	bias_lens--b1	bias_lens--b2	bias_lens--b3	bias_lens--b4	intrinsic_alignment_parameters--a1	intrinsic_alignment_parameters--a2	intrinsic_alignment_parameters--alpha1	intrinsic_alignment_parameters--alpha2	intrinsic_alignment_parameters--bias_ta	COSMOLOGICAL_PARAMETERS--SIGMA_8	COSMOLOGICAL_PARAMETERS--SIGMA_12	DATA_VECTOR--2PT_CHI2	cosmological_parameters--sigma_8	cosmological_parameters--sigma_12	data_vector--2pt_chi2	log_weight	prior	old_post	log_weight	post
#sampler=importance
#n_varied=31
#module_0=consistency 1.1
#module_1=BBN-Consistency 0705.0290
## CITE Comp.Phys.Commun.178:956-971,2008
#module_2=camb 2.0
## CITE http://arxiv.org/abs/1201.3654
## CITE http://arxiv.org/abs/astro-ph/9911177
#module_3=fast_pt 1
## CITE https://iopscience.iop.org/article/10.1088/1475-7516/2016/09/015
#module_4=load_nz_fits 1
#module_5=photoz_bias 1
#module_6=photoz_bias 1
#module_7=photoz_bias 1
#module_8=IA #name from ini file
#module_9=project_2d 1.0
#module_10=project_2d 1.0
#module_11=add_magnification 1
#module_12=add_intrinsic 1.0
#module_13=2pt_shear #name from ini file
#module_14=2pt_gal #name from ini file
#module_15=2pt_gal_shear #name from ini file
#module_16=shear_bias 1
#module_17=add_point_mass #name from ini file
#module_18=2pt_like #name from ini file
#module_19=shear_ratio_like #name from ini file
#timestamp=2024-02-16T12:47:53.710098
#platform=Linux-5.14.21-150400.24.81_12.0.87-cray_shasta_c-x86_64-with-glibc2.31
#platform_version=#1 SMP Thu Jan 11 20:03:21 UTC 2024 (e30c7c1)
#uuid=64f5b4cc3aee40a88d86f80fa2c8f77c
#cosmosis_git_version=
#csl_git_version=
#cwd_git_version=6cd5ddd11b7ef6219eff042ecef8a85dbd2c6510
#hostname=nid005704
#username=nweaverd
#workdir=/global/cfs/cdirs/des/nweaverd/cosmosis-standard-library
#input=output/des-y3-maglim_naut_v2.txt
#nstep=128
#add_to_likelihood=False
## START_OF_PARAMS_INI
## [DEFAULT]
## 2pt_file = likelihood/des-y3/2pt_NG_final_2ptunblind_02_26_21_wnz_maglim_covupdate.fits
## 
## [runtime]
## sampler = importance
## verbosity = standard
## 
## [pipeline]
## modules = consistency  bbn_consistency 	camb  fast_pt 	fits_nz  lens_photoz_width  lens_photoz_bias  source_photoz_bias 	IA  pk_to_cl_gg  pk_to_cl 	add_magnification  add_intrinsic 	2pt_shear 2pt_gal  2pt_gal_shear 	shear_m_bias   add_point_mass 	2pt_like  shear_ratio_like
## timing = F
## debug = F
## priors = examples/des-y3-maglim-priors.ini
## values = examples/des-y3-maglim-values.ini
## extra_output = cosmological_parameters/sigma_8 cosmological_parameters/sigma_12 data_vector/2pt_chi2
## fast_slow = F
## first_fast_module = shear_m_bias

Interestingly the cosmosis and CSL git versions are blank, but like I said they should be identical.
The old_post column of the IS matches the Nautilus chain, as do all the parameter values, but the post column differs with a scatter of ~1.

Answer 4 · 2024-06-14T13:51:58.000Z

Could you send me the full chains, or just the full ini files that generated them?

Answer 5 · 2024-06-14T20:53:33.000Z

Just to follow up: I'm now unable to replicate the mismatch I noted in my post above where I said

I do see the mismatch when I ran a simplified DES chain sampling only one of the shear calibration parameters. This makes me think that this may have something to do with how priors are evaluated, specifically parameters with Gaussian priors. This might explain why 3x2pt shows bigger differences than shear only, since there are more nuisance parameters with non-flat priors.

I don't know if I had an error in my initial test, or some update to my workspace or to cosmosis in the meantime affected this. In any case, now on my simple desktop tests I'm unable to replicate the polychord-vs-IS mismatch. Apologies for adding confusion to the discussion!