brain-score/vision

Kar2019 testing score assertion errors

Opened this issue · 0 comments

Score assertion errors are present in the kar2019 benchmark testing. See below for an example:

    def test_Kar2019ost_cornet_s():
        benchmark = load_benchmark('Kar2019-ost')
        filename = 'cornet_s-kar2019.nc'
        filepath = Path(__file__).parent / filename
        s3.download_file_if_not_exists(local_path=filepath,
                                       bucket='brainscore-unittests', remote_filepath=f'tests/test_benchmarks/{filename}')
        precomputed_features = NeuroidAssembly.from_files(
            filepath,
            stimulus_set_identifier=benchmark._assembly.stimulus_set.identifier,
            stimulus_set=benchmark._assembly.stimulus_set)
        precomputed_features = PrecomputedFeatures(precomputed_features, visual_degrees=8)
        # score
        score = benchmark(precomputed_features).raw
>       assert score == approx(.316, abs=.005)
E       assert <xarray.Score...ay(0.02970973) == 0.316 ± 5.0e-03
E         comparison failed
E         Obtained: <xarray.Score ()>\narray(0.2408883)\nAttributes:\n    raw:      <xarray.Score (split: 10)>\narray([0.14566184, 0.12413281, 0.23...\n    error:    <xarray.Score ()>\narray(0.02970973)
E         Expected: 0.316 ± 5.0e-03

test.py:35: AssertionError

This assertion error is with brainscore version 2.0.0, whereas when using python 3.11 and the updated brainscore version, the output score is 0.23875906 which still causes an assertion error. This is possibly due to the continued use of tensorflow within the ost metric.