Fitting hangs in Prophet 1.1.5

[knkski]

I'm trying to fit Prophet to some data with 1.1.5, and the process hangs, pegging one CPU core. Reverting to 1.1.4 fixes the issue. I'm running this on Ubuntu 22.04 via poetry2nix. Here's the traceback from interrupting the process with ctrl+c:

  File "/nix/store/3rbwh0yg977sw6qggywyjy1jjvkgc3s4-python3-3.11.5-env/lib/python3.11/site-packages/prophet/forecaster.py", line 1232, in fit
    self.params = self.stan_backend.fit(stan_init, dat, **kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/3rbwh0yg977sw6qggywyjy1jjvkgc3s4-python3-3.11.5-env/lib/python3.11/site-packages/prophet/models.py", line 121, in fit
    self.stan_fit = self.model.optimize(**args)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/3rbwh0yg977sw6qggywyjy1jjvkgc3s4-python3-3.11.5-env/lib/python3.11/site-packages/cmdstanpy/model.py", line 681, in optimize
    self._run_cmdstan(
  File "/nix/store/3rbwh0yg977sw6qggywyjy1jjvkgc3s4-python3-3.11.5-env/lib/python3.11/site-packages/cmdstanpy/model.py", line 2086, in _run_cmdstan
    line = proc.stdout.readline()
           ^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt

Here's the command that's hanging. Running it separately, I can reproduce the behavior:

$ /nix/store/5r0d6xzq391p36w1gf2b8n1ngy0gg2n8-python3.11-prophet-1.1.5/lib/python3.11/site-packages/prophet/stan_model/prophet_model.bin random seed=3240 data file=/run/user/1000/tmpl4yoldq_/vbm1e3_7.json init=/run/user/1000/tmpl4yoldq_/0us5yk79.json output file=/run/user/1000/tmpl4yoldq_/prophet_model4k3i9roj/prophet_model-20231102141425.csv method=optimize algorithm=lbfgs iter=10000
method = optimize
  optimize
    algorithm = lbfgs (Default)
      lbfgs
        init_alpha = 0.001 (Default)
        tol_obj = 9.9999999999999998e-13 (Default)
        tol_rel_obj = 10000 (Default)
        tol_grad = 1e-08 (Default)
        tol_rel_grad = 10000000 (Default)
        tol_param = 1e-08 (Default)
        history_size = 5 (Default)
    jacobian = 0 (Default)
    iter = 10000
    save_iterations = 0 (Default)
id = 1 (Default)
data
  file = /run/user/1000/tmpl4yoldq_/vbm1e3_7.json
init = /run/user/1000/tmpl4yoldq_/0us5yk79.json
random
  seed = 3240
output
  file = /run/user/1000/tmpl4yoldq_/prophet_model4k3i9roj/prophet_model-20231102141425.csv
  diagnostic_file =  (Default)
  refresh = 100 (Default)
  sig_figs = -1 (Default)
  profile_file = profile.csv (Default)
num_threads = 1 (Default)

Initial log joint probability = -6.01315
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes

No more output is printed, but here's 1.1.4 completing the same command successfully:

method = optimize
  optimize
    algorithm = lbfgs (Default)
      lbfgs
        init_alpha = 0.001 (Default)
        tol_obj = 9.9999999999999998e-13 (Default)
        tol_rel_obj = 10000 (Default)
        tol_grad = 1e-08 (Default)
        tol_rel_grad = 10000000 (Default)
        tol_param = 1e-08 (Default)
        history_size = 5 (Default)
    iter = 10000
    save_iterations = 0 (Default)
id = 1 (Default)
data
  file = /run/user/1000/tmpl4yoldq_/vbm1e3_7.json
init = /run/user/1000/tmpl4yoldq_/0us5yk79.json
random
  seed = 3240
output
  file = /run/user/1000/tmpl4yoldq_/prophet_model4k3i9roj/prophet_model-20231102141425.csv
  diagnostic_file =  (Default)
  refresh = 100 (Default)
  sig_figs = -1 (Default)
  profile_file = profile.csv (Default)
num_threads = 1 (Default)

Initial log joint probability = -6.01315
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes
      84       143.643   1.61055e-05       2722.96    6.18e-09       0.001      164  LS failed, Hessian reset
      99       143.682   7.57202e-08       2450.56      0.2946      0.2946      183
    Iter      log prob        ||dx||      ||grad||       alpha      alpha0  # evals  Notes
     133       143.689    9.5441e-09       2532.98      0.6641      0.6641      226
Optimization terminated normally:
  Convergence detected: absolute parameter change was below tolerance