Similar to #368 pickle failure

[gannonjs]

General information:

• emcee version: emcee-3.1.0
• platform: Ubuntu 20.04 lts
• installation method (pip/conda/source/other?): conda

Problem description:

Expected behavior: sampler.run_mcmc executes

Actual behavior: sampler.run_mcmc raises error:

_pickle.PicklingError: Can't pickle <function lnprob at 0x7f6a4ced2dd0>: attribute lookup lnprob on main failed

What have you tried so far?: Code has been being used for ~2 years without this issue being raised. The latest run has only involved input file changes.

I've tried downgrading back to emcee-2.2.1 and tried offloading the lnprob to a different file and importing as per #368 . lnprob raises errors as it is written to require information in __main (such as parameter limits).

Minimal example: would prefer not to provide publicly

[gannonjs]

Update: I think my implementation of parallelisation was the cause

Problem code:

with Pool() as pool:
    sampler = emcee.EnsembleSampler(nwalkers, ndim, lnprob, args=(False, False), pool = pool)
    sampler.run_mcmc(pos, 1000, progress = True)

Now running code (if much more slowly than it was):

sampler = emcee.EnsembleSampler(nwalkers, ndim, lnprob, args=(False, False))
sampler.run_mcmc(pos, 1000, progress = True)
```

[dfm]

I can't help if you "don't want to provide" a minimal worked example. Good luck out there!

[gannonjs]

Apologies, I am somewhat new to GitHub and that probably wasn't the right wording on my behalf. The code being run (sans input files) is available -
https://github.com/gannonjs/Published_Code/blob/master/VCC1287_PCA_sky_subtraction.py

Was there an update to the manner in which multiprocessing is being handled as that seems to throw the error?

[dfm]

Please try to put together the simplest snippet of code that reproduces the issue. This shouldn't require data or extra inputs or imports, and I should be able to run it.

Emcee's multiprocessing hasn't changed in many years (since something like version 1), but the Python multiprocessing module has changed regularly, so I'd expect that that's what you're seeing.