semiotic-ai/autoagora-agents

Implement the metrics module

Opened this issue · 6 comments

Implement the metrics module
  1. Elementary metrics:
  • served queries
  • revenue (= served queries * price)
  1. Derivative metrics:
  • query ratio (= served queries / total queries)
  • regret (= customer budget - agent's revenue)
  • regret ratio (= agent's revenue / customer budget)
  1. time to convergence
  • epsilon_1 = accepted difference between the customer budget b and agent's price p (mean)
  • epsilon_2 = defines how wide the gaussian can be (stddev)

agent converged IF |b - p| <= epsilon_1 AND stddev <= epsilon_2

note: think/discuss about the warm-up phase (dynamic budget and/or no queries)

Price efficiency = Indexer price / customer budget