numerai/example-scripts

Issue running example_model_advanced.py with v4.1 data

TomCS91 opened this issue · 0 comments

This is not stricty a bug but with the 4.1 data being the current standard I was interested to run the advanced example on it, however it errors on the below neutralization step with an error I can't get to the bottom of.

            # do neutralization
            print("doing neutralization to riskiest features")
            training_data.loc[test_split_index, f"preds_{model_name}_neutral_riskiest_50"] = neutralize(
                df=training_data.loc[test_split_index, :],
                columns=[f"preds_{model_name}"],
                neutralizers=riskiest_features_split,
                proportion=1.0,
                normalize=True,
                era_col=ERA_COL)[f"preds_{model_name}"]

The following is the full error trace.

---------------------------------------------------------------------------
LinAlgError                               Traceback (most recent call last)
[<ipython-input-10-6a1221969f79>](https://localhost:8080/#) in <cell line: 2>()
     65             # do neutralization
     66             print("doing neutralization to riskiest features")
---> 67             training_data.loc[test_split_index, f"preds_{model_name}_neutral_riskiest_50"] = neutralize(
     68                 df=training_data.loc[test_split_index, :],
     69                 columns=[f"preds_{model_name}"],

5 frames
[<ipython-input-2-6459d8dbad0a>](https://localhost:8080/#) in neutralize(df, columns, neutralizers, proportion, normalize, era_col, verbose)
    139 
    140         scores -= proportion * exposures.dot(
--> 141             np.linalg.pinv(exposures.astype(np.float32), rcond=1e-6).dot(
    142                 scores.astype(np.float32)
    143             )

/usr/local/lib/python3.9/dist-packages/numpy/core/overrides.py in pinv(*args, **kwargs)

[/usr/local/lib/python3.9/dist-packages/numpy/linalg/linalg.py](https://localhost:8080/#) in pinv(a, rcond, hermitian)
   1988         return wrap(res)
   1989     a = a.conjugate()
-> 1990     u, s, vt = svd(a, full_matrices=False, hermitian=hermitian)
   1991 
   1992     # discard small singular values

/usr/local/lib/python3.9/dist-packages/numpy/core/overrides.py in svd(*args, **kwargs)

[/usr/local/lib/python3.9/dist-packages/numpy/linalg/linalg.py](https://localhost:8080/#) in svd(a, full_matrices, compute_uv, hermitian)
   1646 
   1647         signature = 'D->DdD' if isComplexType(t) else 'd->ddd'
-> 1648         u, s, vh = gufunc(a, signature=signature, extobj=extobj)
   1649         u = u.astype(result_t, copy=False)
   1650         s = s.astype(_realType(result_t), copy=False)

[/usr/local/lib/python3.9/dist-packages/numpy/linalg/linalg.py](https://localhost:8080/#) in _raise_linalgerror_svd_nonconvergence(err, flag)
     95 
     96 def _raise_linalgerror_svd_nonconvergence(err, flag):
---> 97     raise LinAlgError("SVD did not converge")
     98 
     99 def _raise_linalgerror_lstsq(err, flag):

LinAlgError: SVD did not converge

Any help or advice from anyone would be amazing.