justinfx/gofileseq

ranges AppendUnique can be very slow

justinfx opened this issue · 0 comments

InclusiveRanges.AppendUnique can potentially be very slow, if previous ranges have been added, and the current range being added is very large.

// ok
var r ranges.InclusiveRanges
r.AppendUnique(1, 1000000000, 1)

// slow
var r ranges.InclusiveRanges
r.AppendUnique(1, 1, 1)
r.AppendUnique(1, 1000000000, 1)

On subsequent appends, a very slow approach is being used, where we loop through the entire range, checking if each value is unique before building it into a range to add. A more efficient approach could be used, to where we detect overlapping ranges and just trim the range to be added.

Example:

var r ranges.InclusiveRanges
r.AppendUnique(1, 100, 1)
r.AppendUnique(50, 150, 1)

In this particular case, we can see the existing range is a stepping of 1 and our new range to append has a stepping of 1. It should be easy enough to tell upfront that we can just do r.Append(101, 150, 1), without needing to loop through all the values.

I'm not sure how to do any better for a bunch of mixed steppings, where we would probably still need to test membership of the values, i.e.

var r ranges.InclusiveRanges
r.AppendUnique(1, 100, 3)
r.AppendUnique(50, 150, 2)

Here, while we still probably need to loop through value to test unique membership, maybe we could at least trim the new potential range to: 101-150x2