Issue running example_model_advanced.py with v4.1 data
TomCS91 opened this issue · 0 comments
TomCS91 commented
This is not stricty a bug but with the 4.1 data being the current standard I was interested to run the advanced example on it, however it errors on the below neutralization step with an error I can't get to the bottom of.
# do neutralization
print("doing neutralization to riskiest features")
training_data.loc[test_split_index, f"preds_{model_name}_neutral_riskiest_50"] = neutralize(
df=training_data.loc[test_split_index, :],
columns=[f"preds_{model_name}"],
neutralizers=riskiest_features_split,
proportion=1.0,
normalize=True,
era_col=ERA_COL)[f"preds_{model_name}"]
The following is the full error trace.
---------------------------------------------------------------------------
LinAlgError Traceback (most recent call last)
[<ipython-input-10-6a1221969f79>](https://localhost:8080/#) in <cell line: 2>()
65 # do neutralization
66 print("doing neutralization to riskiest features")
---> 67 training_data.loc[test_split_index, f"preds_{model_name}_neutral_riskiest_50"] = neutralize(
68 df=training_data.loc[test_split_index, :],
69 columns=[f"preds_{model_name}"],
5 frames
[<ipython-input-2-6459d8dbad0a>](https://localhost:8080/#) in neutralize(df, columns, neutralizers, proportion, normalize, era_col, verbose)
139
140 scores -= proportion * exposures.dot(
--> 141 np.linalg.pinv(exposures.astype(np.float32), rcond=1e-6).dot(
142 scores.astype(np.float32)
143 )
/usr/local/lib/python3.9/dist-packages/numpy/core/overrides.py in pinv(*args, **kwargs)
[/usr/local/lib/python3.9/dist-packages/numpy/linalg/linalg.py](https://localhost:8080/#) in pinv(a, rcond, hermitian)
1988 return wrap(res)
1989 a = a.conjugate()
-> 1990 u, s, vt = svd(a, full_matrices=False, hermitian=hermitian)
1991
1992 # discard small singular values
/usr/local/lib/python3.9/dist-packages/numpy/core/overrides.py in svd(*args, **kwargs)
[/usr/local/lib/python3.9/dist-packages/numpy/linalg/linalg.py](https://localhost:8080/#) in svd(a, full_matrices, compute_uv, hermitian)
1646
1647 signature = 'D->DdD' if isComplexType(t) else 'd->ddd'
-> 1648 u, s, vh = gufunc(a, signature=signature, extobj=extobj)
1649 u = u.astype(result_t, copy=False)
1650 s = s.astype(_realType(result_t), copy=False)
[/usr/local/lib/python3.9/dist-packages/numpy/linalg/linalg.py](https://localhost:8080/#) in _raise_linalgerror_svd_nonconvergence(err, flag)
95
96 def _raise_linalgerror_svd_nonconvergence(err, flag):
---> 97 raise LinAlgError("SVD did not converge")
98
99 def _raise_linalgerror_lstsq(err, flag):
LinAlgError: SVD did not converge
Any help or advice from anyone would be amazing.