How to properly use energy, accuracy etc.?

Question

How to properly use energy, accuracy etc.?

Opened this issue 6 years ago · 5 comments

Hi,

I'm currently using OpenTuner for my thesis and I'm not quite sure how to use energy, accuracy etc. for the Result object. If I just use the time field for minimizing a value, it works without a problem. But if I only use energy (e.g. Result(energy=my_value)) for minimizing energy or only accuracy (e.g. Result(accuracy=my_value)) for maximizing a value (this field is for maximizing, isn't it?), OpenTuner complains that time is None and thus can't compare energy/accuracy with time (OpenTuner exits after this).

Do I have to set time manually even though I only want to e.g. minimize energy? Which value should be set in such a case? 0.0 or 1.0 or something different? Unfortunately, this is not really documented and I only could find this line in unitary where this is used, but I'm still not sure which time value should be set here.

Answer 1 · 2018-07-25T05:04:40.000Z

You need to pass a SearchObjective instance to your MeasurementInterface's super __init__ call, or otherwise override MeasurementInterface.objective(). There are some predefined objectives in search/objective.py but none of them mention energy, so you'd have to implement your own.

Answer 2 · 2018-08-05T23:43:49.000Z

Many thanks for your answer.

My goal is to optimize two objectives (time and energy), but I tried it with minimizing energy first. For this I simply modified the MinimizeTime objective:

class MinimizeEnergy(SearchObjective):
  """
  minimize Result().energy
  """

  def result_order_by_terms(self):
    """return database columns required to order by the objective"""
    return [Result.energy]

  def result_compare(self, result1, result2):
    """cmp() compatible comparison of resultsdb.models.Result"""
    return cmp(result1.energy, result2.energy)

  def config_compare(self, config1, config2):
    """cmp() compatible comparison of resultsdb.models.Configuration"""
    return cmp(min(map(_.energy, self.driver.results_query(config=config1))),
               min(map(_.energy, self.driver.results_query(config=config2))))

  def result_relative(self, result1, result2):
    """return None, or a relative goodness of resultsdb.models.Result"""
    if result2.energy == 0:
      return float('inf') * result1.energy
    return result1.energy / result2.energy

The objective is overwritten in my tuner class:

def objective(self):
    return MinimizeEnergy()

When I now use Result(energy=my_value), I still get the error I mentioned in my first post. Here is the message:

[    17s]    INFO opentuner.search.plugin.DisplayPlugin: tests=1, best {'frequency': 1, 'threads': 4}, cost energy=35.2974, found by DifferentialEvolutionAlt
Traceback (most recent call last):
  File "/home/user/PycharmProjects/Tests/FrequencyThreadEnergyParetoTuner.py", line 125, in <module>
    FrequencyThreadEnergyTuner.main(argparser.parse_args())
  File "/home/user/.local/lib/python2.7/site-packages/opentuner/measurement/interface.py", line 299, in main
    return TuningRunMain(cls(args, *pargs, **kwargs), args).main()
  File "/home/user/.local/lib/python2.7/site-packages/opentuner/tuningrunmain.py", line 199, in main
    self.search_driver.main()
  File "/home/user/.local/lib/python2.7/site-packages/opentuner/search/driver.py", line 275, in main
    self.run_generation_results(offset=-self.args.pipelining)
  File "/home/user/.local/lib/python2.7/site-packages/opentuner/search/driver.py", line 227, in run_generation_results
    self.wait_for_results(self.generation + offset)
  File "/home/user/.local/lib/python2.7/site-packages/opentuner/tuningrunmain.py", line 217, in results_wait
    self.measurement_driver.process_all()
  File "/home/user/.local/lib/python2.7/site-packages/opentuner/measurement/driver.py", line 212, in process_all
    self.run_desired_result(dr)
  File "/home/user/.local/lib/python2.7/site-packages/opentuner/measurement/driver.py", line 113, in run_desired_result
    desired_result.limit = self.run_time_limit(desired_result)
  File "/home/user/.local/lib/python2.7/site-packages/opentuner/measurement/driver.py", line 81, in run_time_limit
    return self.default_limit_multiplier * best.time
TypeError: unsupported operand type(s) for *: 'float' and 'NoneType'

This is the OpenTuner function where the error appears (in the last line):

def run_time_limit(self, desired_result, default=3600.0 * 24 * 365 * 10):
  """return a time limit to apply to a test run (in seconds)"""
  best = self.results_query(objective_ordered=True).first()
  if best is None:
    if desired_result.limit:
      return desired_result.limit
    else:
      return default

  if desired_result.limit:
    return min(desired_result.limit, self.upper_limit_multiplier * best.time)
  else:
    return self.default_limit_multiplier * best.time

I first thought this error appeared because I set --stop-after (although it doesn't appear when using the time field of the Result object), but I still get it when I run OpenTuner without any time limit. Then I tried to set the time field to 1.0 (namely Result(time=1.0, energy=my_value)) and it now works correctly (OpenTuner minimizes the energy value).

I debugged it a bit and found out that the best object always had None for the time field when I didn't set it myself (the energy value was properly set). Is this a current "limitation" of OpenTuner?

My second question would be: What methods have to be implemented for optimizing multiple objectives? I looked at the implemented examples in objective.py and think that def result_order_by_terms(), def result_compare() and def result_relative() always need to be implemented.

But some objectives like MinimizeTime also define config_compare() which isn't defined by MaximizeAccuracy or MaximizeAccuracyMinimizeSize. Is this only needed in special cases?

I hope you can also shed some light on this.

Answer 3 · 2018-08-06T03:30:47.000Z

run_time_limit isn't related to --stop-after, I don't think. It's used in run_desired_result to set the limit passed to run_precompiled or compile_and_run. The idea is that some configurations may be dramatically worse than others, but past a certain point, there's no meaningful difference between 'awful' and 'extremely awful'. run_time_limit is a heuristic for setting a cutoff after which the trial can be aborted.

I didn't actually look at the history/blame but I guess that code was written before customizable objectives were introduced. The objective's result_order_by_terms isn't enough information to decide what to use in place of time, because we don't know whether we're minimizing or maximizing those terms, and if we have multiple terms, we don't know how to trade off between them (or if we should have two separate cutoffs). The proper thing to do is to move it onto SearchObjective; the pragmatic thing to do is to return default if best.time is None, just as if best itself is None.

For your second question, MinimizeTime.config_compare seems to be unnecessary because the superclass implementation should do the right thing by delegating to result_compare. ThresholdAccuracyMinimizeTime's override adds objective_ordered=True, which adds an order-by to the query, but it's not obvious to me why that's necessary. To sort acceptable before unacceptable results, perhaps?

I've never had to look at this code before; in the Mario example which uses an arbitrary fitness function, I just returned the negative fitness as time and minimized "time". (See also #74.)

Answer 4 · 2018-08-08T05:11:48.000Z

Thanks for the clarifications.

I have a few more questions about optimizing multiple objectives.

The result_relative() method of MaximizeAccuracyMinimizeSize always returns None (there is also a comment that it's unimplemented for now). Is this method important for the "goodness" of the final result? Or does it only help finding better configurations in less time? For single objectives this is pretty easy, but I have to admit that defining this for multiple objective may not be that easy.
There is also a stats_quality_score() method which is only overwritten by MaximizeAccuracy (somehow it's still the same as the default one). Is this method needed?
Is OpenTuner using any algorithm to determine the "best" configuration when optimizing multiple objectives? Or does it just use the Python cmp() function which - as far as I know - compares the first value of each tuple and if they are the same it compares the next entry in the tuple with each other?

I ask because I want to find Pareto-optimal points which means that any of the target values can't be improved further without worsening one of the values. There is an algorithm called Simple Cull which I'm going to use to find all these points (the points tested by OpenTuner will be the input), but it would be interesting to know if OpenTuner returns one of these points.

Answer 5 · 2018-08-09T00:29:17.000Z

result_relative is only called from SearchObjective.relative, and that's currently only called from the simulated annealing technique. But the result of that capability check is then unconditionally set to false. So it seems it was intended to be used by techniques, but nothing is currently using it.

stats_quality_score is only called from utils/stats.py and utils/stats_matplotlib.py for generating reports/plots. You'd only need to implement it if you want to use those utilities. (I suspect they're not useful for multi-objective optimization, because you'd actually want three-dimensional plots showing the progress of the Pareto frontier over time.)

Techniques use the SearchObjective as a comparator (grep for cmp=objective.compare and objective.lt to find usages). The "best" configuration provided to MeasurementInterface.save_final_config is the minimum configuration as judged by objective.lt. Some of the code instead uses SearchObjective.result_order_by (which, by default, delegates to result_order_by_terms) to perform the sort in the database.

I don't think OpenTuner has any direct support for finding multiple configurations from a tuning run (Pareto or otherwise). You'll need to write code to look over all the results in the database. You should be able to do this with the SQLAlchemy-related code (that's resultsdb/models.py), but if not, you can also open the database directly in whatever SQLite client and extract the results. (To generate a movie file showing the progression of new-best Mario runs, I shell out to the sqlite3 command-line utility. I'm sure I could do this with the ORM code but I've never worked with SQLAlchemy, so I thought this was easier.)