transcription+velocity metrics fail when there is no matching

[jongwook]

The implementation recently added in #283, it fails with an unexpected exception when there is zero match between the reference and the estimated.

from numpy import array
from mir_eval.transcription_velocity import precision_recall_f1_overlap

precision_recall_f1_overlap(array([[1, 2]]), array([1]), array([1]),
                            array([[3, 4]]), array([1]), array([1]))    

~/anaconda3/lib/python3.6/site-packages/mir_eval/transcription_velocity.py in precision_recall_f1_overlap(ref_intervals, ref_pitches, ref_velocities, est_intervals, est_pitches, est_velocities, onset_tolerance, pitch_tolerance, offset_ratio, offset_min_tolerance, strict, velocity_tolerance, beta)
    289         ref_intervals, ref_pitches, ref_velocities, est_intervals, est_pitches,
    290         est_velocities, onset_tolerance, pitch_tolerance, offset_ratio,
--> 291         offset_min_tolerance, strict, velocity_tolerance)
    292 
    293     precision = float(len(matching))/len(est_pitches)

~/anaconda3/lib/python3.6/site-packages/mir_eval/transcription_velocity.py in match_notes(ref_intervals, ref_pitches, ref_velocities, est_intervals, est_pitches, est_velocities, onset_tolerance, pitch_tolerance, offset_ratio, offset_min_tolerance, strict, velocity_tolerance)
    176     matching = np.array(matching)
    177     # Grab velocities for matched notes
--> 178     ref_matched_velocities = ref_velocities[matching[:, 0]]
    179     est_matched_velocities = est_velocities[matching[:, 1]]
    180     # Find slope and intercept of line which produces best least-squares fit

IndexError: too many indices for array

---

In a related context, it also throws an exception when either is empty, not following the convention in #43 / #44

[craffel]

The implementation recently added in #283, it fails with an unexpected exception when there is zero match between the reference and the estimated.

Thanks, I've made a fix in #297. Can you confirm this solves the issue?

In a related context, it also throws an exception when either is empty, not following the convention in #43 / #44

Can you provide an example of this behavior? This line should prevent that:
https://github.com/craffel/mir_eval/blob/master/mir_eval/transcription_velocity.py#L285

[jongwook]

Thanks Colin, I confirm that #297 fixes the issue.

Regarding the second case, below is an example. The exception is thrown inside the validate() function which is called before the if statement at line 285. I presume switching the order of the two will fix the issue?

In [7]: from numpy import array 
   ...: from mir_eval.transcription_velocity import precision_recall_f1_overlap 
   ...:  
   ...: precision_recall_f1_overlap(array([[1, 2]]), array([1]), array([1]), 
   ...:                             array([[]]), array([]), array([]))                                                  
/home/jongwook/anaconda3/lib/python3.6/site-packages/mir_eval/transcription.py:167: UserWarning: Estimated notes are empty.
  warnings.warn("Estimated notes are empty.")
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-7-03f2bd745b18> in <module>
      3 
      4 precision_recall_f1_overlap(array([[1, 2]]), array([1]), array([1]),
----> 5                             array([[]]), array([]), array([]))    

~/anaconda3/lib/python3.6/site-packages/mir_eval/transcription_velocity.py in precision_recall_f1_overlap(ref_intervals, ref_pitches, ref_velocities, est_intervals, est_pitches, est_velocities, onset_tolerance, pitch_tolerance, offset_ratio, offset_min_tolerance, strict, velocity_tolerance, beta)
    284     """
    285     validate(ref_intervals, ref_pitches, ref_velocities, est_intervals,
--> 286              est_pitches, est_velocities)
    287     # When reference notes are empty, metrics are undefined, return 0's
    288     if len(ref_pitches) == 0 or len(est_pitches) == 0:

~/anaconda3/lib/python3.6/site-packages/mir_eval/transcription_velocity.py in validate(ref_intervals, ref_pitches, ref_velocities, est_intervals, est_pitches, est_velocities)
     81     """
     82     transcription.validate(ref_intervals, ref_pitches, est_intervals,
---> 83                            est_pitches)
     84     # Check that velocities have the same length as intervals/pitches
     85     if not ref_velocities.shape[0] == ref_pitches.shape[0]:

~/anaconda3/lib/python3.6/site-packages/mir_eval/transcription.py in validate(ref_intervals, ref_pitches, est_intervals, est_pitches)
    131     """
    132     # Validate intervals
--> 133     validate_intervals(ref_intervals, est_intervals)
    134 
    135     # Make sure intervals and pitches match in length

~/anaconda3/lib/python3.6/site-packages/mir_eval/transcription.py in validate_intervals(ref_intervals, est_intervals)
    169     # Validate intervals
    170     util.validate_intervals(ref_intervals)
--> 171     util.validate_intervals(est_intervals)
    172 
    173 

~/anaconda3/lib/python3.6/site-packages/mir_eval/util.py in validate_intervals(intervals)
    770     if intervals.ndim != 2 or intervals.shape[1] != 2:
    771         raise ValueError('Intervals should be n-by-2 numpy ndarray, '
--> 772                          'but shape={}'.format(intervals.shape))
    773 
    774     # Make sure no times are negative

ValueError: Intervals should be n-by-2 numpy ndarray, but shape=(1, 0)

[craffel]

That's intended behavior. An empty array of intervals would have shape (0, 2). You're passing an array with shape (1, 0).

[jongwook]

I see; that makes sense. I see that passing np.zeros((0, 2)) works. I guess this issue can be closed together with merging #297.

[craffel]

Thanks for pointing this out.