Kar2019 testing score assertion errors
Opened this issue · 0 comments
samwinebrake commented
Score assertion errors are present in the kar2019 benchmark testing. See below for an example:
def test_Kar2019ost_cornet_s():
benchmark = load_benchmark('Kar2019-ost')
filename = 'cornet_s-kar2019.nc'
filepath = Path(__file__).parent / filename
s3.download_file_if_not_exists(local_path=filepath,
bucket='brainscore-unittests', remote_filepath=f'tests/test_benchmarks/{filename}')
precomputed_features = NeuroidAssembly.from_files(
filepath,
stimulus_set_identifier=benchmark._assembly.stimulus_set.identifier,
stimulus_set=benchmark._assembly.stimulus_set)
precomputed_features = PrecomputedFeatures(precomputed_features, visual_degrees=8)
# score
score = benchmark(precomputed_features).raw
> assert score == approx(.316, abs=.005)
E assert <xarray.Score...ay(0.02970973) == 0.316 ± 5.0e-03
E comparison failed
E Obtained: <xarray.Score ()>\narray(0.2408883)\nAttributes:\n raw: <xarray.Score (split: 10)>\narray([0.14566184, 0.12413281, 0.23...\n error: <xarray.Score ()>\narray(0.02970973)
E Expected: 0.316 ± 5.0e-03
test.py:35: AssertionError
This assertion error is with brainscore version 2.0.0, whereas when using python 3.11 and the updated brainscore version, the output score is 0.23875906
which still causes an assertion error. This is possibly due to the continued use of tensorflow within the ost metric.