Here's some resources on value of information. Including utilities to calculate it and a card game made out of it.
There are several pdfs in this repo for set cards, but if you want to make your own, you can use generate_value.py
cards_nine_per_page.pdf is a new version of the game ready for printing. If you've played the game with me before this is the one you can use. I would recommend printing them all on 300gsm paper single sided, then cutting them out with a stanley knife, and gluing them back to back to make thick cards.
cards_one_per_page.pdf is the same as the above but each card side gets it's own page in the document. You can use this if you want to make cards of different sizes.
cards_old.pdf represents an old (and buggy) version of the game.
You play by giving a player, or a team of players, n (I'm currently using n=10) tokens. These tokens represent money. Then laying down 1 evaluator card and 7 intervention cards in front of them. Make sure the intervention cards are faced down (has the ? symbol facing up, rather than the lighbulb) and that the player does not look on the other side of the card.
The tokens I use are glass pebbles that you can often find in home rennovation stores and are used for water features and paving. They are usually extremely inexpensive.
The goal (as is life) is to create the most utility possible. You can create utility by spending the tokens (money) on an intervention. By spending money on an intervention, you get the utility that is printed on the other side of the card. However, you do not know the utility on the other side of the card, but what you are given is an 89% confidence interval on bottom of the card (lognormal distribution, if it matters to you). You can distribute your tokens in any way you wish, for instance you could put it all on one card or spread them out to each intervention. When you put multiple tokens on a card, you get utility equal to the number on the other side multiplied by the number of tokens you put on the card.
What makes this game interesting, is that if you want, you can put a token on the evaluator card. Putting a token on the evaluator card lets you flip over one of the intervention cards and learn its true utility. However, you must work out how much that information is worth to you, because every token you put on the evaluator card does not go towards an intervention.
Have a go!
Extra notes for the curious that ask questions:
- The numbers were generated by a computer program, so the game is fair. Also, the names/logos and cause areas were also generated by a computer program and are entirely decorative. They should not play a part in your decision making, nor do they represent my opinion of the value of these cause areas.
- Why 89%? It's a reference to Statistical Rethinking. It's a bit of a Bayesian joke to puncuate the arbitraryness of the number, as the more commonly used 95% for statistical significance is equally arbitrary.
- Why lognormal? We use lognormal distributions a lot at QURI. We like them because there interventions are calculated by the product of a lot of positive random variables, that by an extention of the central limit theorem, comes out to be lognormal.
I usually first ask "What make you decide what you wanted to do?", and then go into one of three discussion topics based on what they say. I generally find that people have good intuitions when they come to calculating the value of information, and hearing what they say and how
The first thing to realise about the game is how to calculate expected value (EV). Although this is not exact, a good approximation is simple the average of the high and the low number to get the EV. One question to ask to talk about this would be "If you didn't have the evaluator card, what would you do?". Often, people will say they would just put all their money on the one with the highest EV. I often also notice people will only evaluate charities with a high EV, and I encourage that and say that that's because it's likely to change your mind.
Hedging is when one choses to put tokens into multple places. If one wants to go for the highest EV, you shouldn't hedge and you should simply put all your money into the card with the highest EV. You may want to hedge if:
- You have loss aversion
- You have diminishing marginal returns (aka, "If I gave 10 Billion dollars to AMF, they wouldn't be able to use it all effectively because they wouldn't know what to do with it")
- Your goal is to beat another player/score over getting the highest EV.
I usually then prompt people with "What would you do if you had 100 tokens rather than 10". People will usually correctly answer that they would evaluate a lot more. I then support that to say that that's exactly the case, that VOI is (linearly) proportional to the amount of resources you have.