Benchmarks need a little clarification and maybe a feature request

[MadBomber]

Please correct me if I'm wrong... The actual parsing of the UA string does not take place until a specific component is requested. At that point only that specific component is extracted from the UA string. So other user agent parsers which extract all components at once will show up slower in your benchmarks.

I can see where your JIT parser approach comes in handy when only one or two components are used within an application. it would be interesting to see how your approach actually compares to a full parse of all components. For example Useragent now supports a #to_h method which provides a nicely structured representation of all of the components of the UA string.

If DeviceDetector had a #to_h feature that did the same thing, how would your regex approach compare benchmark-wise against UserAgent ?

I can see how a #to_h feature may be outside the scope for which DeviceDetector is best suited.

Dewayne
o-*

[matisojka]

Yes, there are three main components, the browser / client (software), the operating system and the device (hardware).

The thing is, UserAgent doesn't do device detection at all, so having a benchmark that compares parsing the client name, os and device (DeviceDetector) with only name & os (UserAgent) wouldn't be fair at all.

The method #to_h that you propose is useful, but it can be achieved by the end-user by casting the results into a hash, so I don't know if there is a benefit of adding it to the gem.

[gshutler]

Hi, I'm maintaining UserAgent nowadays and I'd like to be able to run your benchmark to understand where it's slow. Is user_agent_strings.txt available anywhere?

Also, where do you believe most of your performance comes from? I would have thought the caching could be quite significant but I suppose that depends on the content of user_agent_strings.txt.

[matisojka]

Hello @gshutler!

Most of the performance comes from caching and optimizing the regex parsing and matching. It's no voodoo magic and actually we didn't want to make this a contest, we just wanted to ensure that future PRs do not compromise speed of detection. Our priority is to provide the most accurate detection while not giving up on a decent speed.

Unfortunately, the user_agent_strings.txt is not available, but it has ~200.000 raw user agent strings in it, ~5.000 being unique. This is a pretty common scenario, as the most popular platforms, browsers and devices are used more often than other, more exotic ones. Here, the caching helps a lot.

[gshutler]

If user_agent_strings.txt isn't able to be made available could you run this for me to see how much it helps?

require 'device_detector'
require 'browser'
require 'user_agent'
require 'lru_redux'
require 'benchmark'

user_agent_strings = File.read('./tmp/user_agent_strings.txt').split("\n")

class CachedUserAgent
  DEFAULT_MAX_ITEMS = 5000

  def self.max_items
    @max_items ||= DEFAULT_MAX_ITEMS
  end

  def self.max_items=(value)
    @max_items = value
  end

  def self.cache
    @good_cache ||= LruRedux::ThreadSafeCache.new(max_items)
  end

  def self.parse(useragent)
    cache.getset(useragent) { UserAgent.parse(useragent) }
  end
end

## Benchmarks

Benchmark.bm(15) do |x|
  x.report('device_detector') {
    user_agent_strings.each { |uas| DeviceDetector.new(uas).name }
  }
  x.report('browser') {
    user_agent_strings.each { |uas| Browser.new(ua: uas).name }
  }
  x.report('useragent') {
    user_agent_strings.each { |uas| UserAgent.parse(uas).browser }
  }
  x.report('cached useragent') {
    user_agent_strings.each { |uas| CachedUserAgent.parse(uas).browser }
  }
end

I'd like to get a feel for how effective the cache is. Have you also done benchmarking with different sizes of cache (100, 250, 500, 1000, 2000 for example)? With your default cache size also roughly matching your unique user agent strings it would be interesting to see where the optimum memory/speed trade off lay.

[matisojka]

@gshutler I got some good news for you ;)

Using the cached variant of User Agent it actually wins the race:

Cache size 5000
                      user     system      total        real
device_detector   1.170000   0.010000   1.180000 (  1.179326)
browser           2.010000   0.010000   2.020000 (  2.025717)
useragent         4.600000   0.010000   4.610000 (  4.605785)
cached useragent  0.560000   0.010000   0.570000 (  0.571583)

Also, I have detected that the cache size beyond 1.000 has a minimal impact on detection performance:

Cache size 1000
                      user     system      total        real
device_detector   1.160000   0.020000   1.180000 (  1.179425)
browser           2.030000   0.010000   2.040000 (  2.036658)
useragent         4.640000   0.010000   4.650000 (  4.660174)
cached useragent  0.610000   0.010000   0.620000 (  0.607808)

At Podigee we have noticed that caching user agent strings is beneficial to us. Obviously, each use case is different, but the good thing about caching is that it has a negligible impact on performance even if each user agent string is different (extremely uncommon).

Next thing that would be nice to measure is how good / precise UA detection is. Our code isn't going to get much faster, as the regex parsing and matching through hundreds of different options is quite expensive.

If it is OK with you, I'd like to close this issue.

[MadBomber]

Thank you. Good discussion. Lets close it.

[matisojka]

Thank you too ;)