Estimating True Skill in a Single MTT Structure - Gambling and Probability

Sherman · 02-02-2012, 12:17 AM

So I'm not sure how crazy this sounds or if I have the math right, but I figure the folks here can tell me if this idea is way off base. The idea works like this. Take a player who has finished N MTTs with an identical structure (think Stars 4/180s) where N is some number greater than 1. Given this information, can we make an accurate estimate of the player's true skill level? I believe that it may be possible. This is because in MTTs with an identified structure we have known information about the population Mean and SD.

For example, in 4/180s, we know that the population mean result must be -$0.40 with a variance $447.0234.

Let's further say we have the following sample of results from a player who played 20 4/180s.

Code:

 0.00   0.00   0.00   0.00   0.00   0.00   0.00   0.00   0.00   0.00 216.00   0.00   0.00   0.00   0.00  18.72   0.00   0.00   0.00   0.00

Over these 20 MTTs this player average $7.336 with a variance of $2212.588.

From this information we can compute an estimate of this player's true skill as follows:

((popMean / popVAR) + (obsMean / obsVAR)) = Numerator
(1/popVAR + 1/obsVAR) = Denominator

Dividing these two results we get an estimate of the player's true skill. Using the numbers from this data we would have the following:

((-.4 / 447.0234) + (7.336 / 2212.588)) = Numerator
(1/447.0234 + 1/2212.588) = Denominator

This gives us a result of .900255, or about $0.90 per tournament. This is a far cry from the original observed estimate of $7.336 per tournament. If certain assumptions are met (which I am not sure they are) and my math is correct, this number should represent our estimate of this player's true skill (in terms of $ won) in this type of MTT given the results we have seen from him so far.

Further, this estimate has a variance:

1 / [ (1/popVAR) + (1/obsVAR) ]

and of course the square root of this number is the standard deviation of this true skill estimate.

So what do you think? Does this make sense?

For what it is worth, I have already created an R function that computes these numbers in case someone wants to see the math that way.

Code:

MTT.skill <- function(pstruct, BI, prizes, ignore.prob=FALSE) {
  if(sum(pstruct < 0) > 0) {stop("pstruct must not be below 0.")}
  if(length(prizes) < 2) {stop("Must have more than 1 prize.")}

  popProfits <- pstruct - BI
  popMean <- mean(popProfits)
  popVAR <- sum(popProfits^2 * (1/length(pstruct))) - popMean^2

  profits <- prizes - BI
  obsMean <- mean(profits)
  obsVAR <- sum(profits^2 * (1/length(profits))) - obsMean^2

  N <- length(prizes)
  skill <- ((popMean / popVAR) + (obsMean / obsVAR)) / ((1/popVAR) + (1/obsVAR))
  skillSD <- sqrt(1 / ((1/popVAR) + (1/obsVAR)))
  SE <- skillSD / sqrt(N)
  outcash <- rbind(obsMean, skill, skillSD, N, SE)
  outBI <- rbind(obsMean/BI, skill / BI, skillSD / BI, N, SE / BI)
  out <- cbind(outcash, outBI)
  rownames(out) <- c("Obs. Mean", "Est. True Mean", "Est. SD", "N", "Est. SE")
  colnames(out) <- c("Results $", "Results BI")
  return(out)
}

Example:

Code:

Stars4_180.prizestruct <- c(216, 144, 85.68, 57.60, 46.80, 36.00, 25.20, 18.72, 12.24, rep(8.64, 9), rep(0,162))
samp <- sample(Stars4_180.prizestruct, 20, T)
samp
MTT.skill(Stars4_180.prizestruct, 4.40, samp)

AaronBrown · 02-04-2012, 08:16 PM

The procedure is reasonable, but arbitrary. You are putting shrinking the observed result toward the known population average, weighting by inverse variance. Shrinking is a good idea, but your degree of shrinkage is arbitrary.

A less arbitrary way to shrink would be to get data on the distribution of results of actual players. If it matched the population distribution, you would want to shrink a lot, because it would appear that there are few players with significant edges. If it was much more spread out than the population distribution, then you wouldn't be inclined to shrink as much.

Sherman · 02-04-2012, 11:28 PM

Quote:

Originally Posted by AaronBrown

A less arbitrary way to shrink would be to get data on the distribution of results of actual players. If it matched the population distribution, you would want to shrink a lot, because it would appear that there are few players with significant edges. If it was much more spread out than the population distribution, then you wouldn't be inclined to shrink as much.

Thanks for your thoughts Aaron. I don't quite understand what you mean in this part though. By definition (because it is a fixed tournament) the population distribution of results for any single tournament is known. And that is where my population numbers come from. So we know the mean and variance are right. I think the issue might be whether using the variance is appropriate because of the shape of the distribution.

Could you explain a bit more about what you mean here?

AaronBrown · 02-05-2012, 11:57 AM

If there were no skill involved, every players' distribution of returns would look like the population distribution.

If it were all skill, each player would have zero variance.

The ratio of the average players' variance to the population variance is a measure of the amount of skill. This is the insight that your estimator exploits. However, you are looking only at a single player to estimate the degree of skill. There is information in how all the players do.

Consider a simpler application, heads-up, freeze-out play. Suppose someone has won 30 and lost 10. What odds would you put on him winning versus a random other player? Saying 1:3 is giving too much credence to the history. If he were 30 and 0, you wouldn't say he was certain to win. But saying 1:1 is ignoring the data. So you want something in between.

Suppose I told you that there were 1,000 players and he had the best record. Well, you expect more than one 30-10 player even if all the matches are coin flips. So you're likely to assign his performance almost totally to luck, and say his odds are close to 1:1 in the next match. You would feel even more confident if the records of all players looked exactly like a Binomial distribution.

On the other hand, suppose I told you that 20 players out of 1,000 had records of 30-10 or better. That virtually proves that there is some skill, so you're likely to put the odds closer to 3:1. You won't go all the way there, the players with good records on average have had good luck, but you won't think of 1:1, maybe you'll shrink to something like 5:2.

Sherman · 02-05-2012, 12:12 PM

Quote:

Originally Posted by AaronBrown

If there were no skill involved, every players' distribution of returns would look like the population distribution.

If it were all skill, each player would have zero variance.

The ratio of the average players' variance to the population variance is a measure of the amount of skill. This is the insight that your estimator exploits. However, you are looking only at a single player to estimate the degree of skill. There is information in how all the players do.

Ah. This makes sense to me. While I have the total variation if everything were luck, I do not have the variation due to skill (in the game) for all players. I just have this one player's variation. It would be better if I had the actual variation of all players and set it as the population variation.