commonsense_constraint error

[yananchen1989]

hi team.

As we see in the code, temperature is zero, but you know, sometimes the response from llm is not deterministic.
Therefore, sometimes, though not that often, LLM does not generate a valid plan.

for example, in the number 48 example in validation set, query:

I require a travel itinerary for a seven-day trip beginning on March 2nd and ending on March 8th, 2022. The trip will begin in Philadelphia and involve visiting 3 cities in Virginia. The available budget for the trip is $2,900.

most of time, it generates a valid plan, like this below, which can be parsed into json format for further eval.

Day 1:
Current City: Philadelphia
Transportation: Self-driving from Philadelphia to Richmond

• Duration: 4 hours 2 mins
• Distance: 407 km
• Cost: $20
  Accommodation: Not applicable

Day 2:
Current City: Richmond
Transportation: Self-driving from Richmond to Jamestown

• Duration: 1 hour 1 min
• Distance: 92.4 km
• Cost: $4
  Accommodation: Not applicable

Day 3:
Current City: Jamestown
Attraction: Historic Jamestowne

• Address: 1368 Colonial Nat'l Historical Pkwy, Jamestown, VA 23081, USA
• Phone: (757) 856-1250
• Website: http://www.historicjamestowne.org/
  Lunch: Beyond Breads
• Average Cost: $95
  Dinner: Mama's Fish House
• Average Cost: $63
  Accommodation: Not applicable

Day 4:
Current City: Jamestown
Transportation: Self-driving from Jamestown to Charlottesville

• Duration: 2 hours 2 mins
• Distance: 206 km
• Cost: $10
  Accommodation: Not applicable

Day 5:
Current City: Charlottesville
Attraction: Monticello

• Address: 1050 Monticello Loop, Charlottesville, VA 22902, USA
• Phone: (434) 984-9800
• Website: https://www.monticello.org/
  Lunch: Mama's Fish House
• Average Cost: $63
  Dinner: Restaurant Andre
• Average Cost: $250
  Accommodation: Not applicable

Day 6:
Current City: Charlottesville
Transportation: Self-driving from Charlottesville to Philadelphia

• Duration: 4 hours 24 mins
• Distance: 411 km
• Cost: $20
  Accommodation: Not applicable

Day 7:
Current City: Philadelphia
Accommodation: Not applicable

However, a few times, under the exactly same prompt, it generates

Apologies, but it seems that the budget provided is insufficient to cover the travel expenses for a seven-day trip involving multiple cities. If you could provide a higher budget, I
would be more than happy to assist you in creating a detailed travel itinerary.

and parsed by LLM, it will be json: {"error": "Insufficient budget provided"}

so for this case, it will cause error in eval.py

File "C:\Users\ITSupp\Downloads\codes\TravelPlanner\tools\planner\sole_planning.py", line 169, in
scores, detailed_scores = eval_score(args.set_type, tested_plans)
File "C:\Users/ITSupp/Downloads/codes/TravelPlanner/evaluation\eval.py", line 80, in eval_score
commonsense_info_box = commonsense_eval(query_data,tested_plan['plan'])
File "C:\Users/ITSupp/Downloads/codes/TravelPlanner/evaluation\commonsense_constraint.py", line 523, in evaluation
return_info['is_reasonalbe_visiting_city'] = is_reasonalbe_visiting_city(query_data, tested_data)
File "C:\Users/ITSupp/Downloads/codes/TravelPlanner/evaluation\commonsense_constraint.py", line 134, in is_reasonalbe_visiting_city
city_value = tested_data[i]['current_city']

is there any suggestion to not trigger this error and maybe for this type of cases, the eval system directly count them as 0 delivery ?
thanks.

[hsaest]

Hi,

This issue also exists in our experiments. However, the best solution we've come up with is to manually revise these instances. Of course, directly counting them as zero is a simpler method. However, identifying these exceptions poses a challenge since the exceptions arising during parsing by LLMs vary greatly. Given the rigorous nature of this research work, we cannot afford to assign a score of zero every time an exception occurs, especially considering the variety of other exceptions that might arise.

[yananchen1989]

@hsaest very helpful. make sense. thanks.