Sorta a weird question. Everyone wonders about variance. I think it tends to get exaggerated on these message boards. However its occured to me lately that there must be a lottery-winner equivalent in poker on both the lucky and unlucky side.

As a probabilist do you think the odds of winning a state lottery on a 1 dollar ticket or turning 100 into 1 million and thinking your a pro poker player when in fact you only have average skill?

This might be a dumb ill defined question if you think its lame just ignore it.

Jerry Yang.

Well, I just saw someone with over 1100 posts, and they were still an adept. I guess I duped myself. Oh well, it's been and continues to be a fun thread for me. So I'm not bothered.

I never reached what I consider a financially serious level, but I did take the game very seriously.

Not really. Although I do think it gave me a slight advantage in things like bankroll management and statistical analysis of my winrate (which are of course related). For example, what is the subtle problem with the maximum likelihood approximation formula for the standard deviation, as presented by Mason in his well-known essay, and how might this affect the results? How does Poker Tracker compute your standard deviation, why might its result be inaccurate, and how can you fix it? How do you do proper bankroll management when you regularly play at different levels and even in different games? How can you incorporate qualitative information about your skill level into a statistical estimate of your winrate? What size downswings can I expect, and after how many hands can I expect them to appear? These questions seem to elude a lot of people. In fact, I have seen very smart people who confidently give the wrong answers to them. I have written a lot about these topics here on the forums, and also in my private notes. When I find the time, I hope to compile it all and make a long article/short book about it.

haha, you should really formally compile some of that and post it here if you wouldn't mind. i know at least about myself that i would probably get a lot of those wrong. especially the downswing and bankroll management stuff, i have generally been fairly conservative in my opinions of both but i don't think i could give a qualitative analysis of either.

Off the top of my head, I can't think of anything beyond the standard, well-known examples. See here. Other than that, people are sometimes thrown for a loop by the basics. For instance, roll a die and consider two events:

A = "the result is 4 or less",
B = "the result is a prime number".

Some of my students are wholly confused as to why these events are independent. If I think of something better, I'll post it.

Some of my colleagues might, but I have never discussed string theory with them, so I don't know. I certainly don't. In fact, I don't really know anything about string theory and didn't even know there was a controversy. Is it just that string theory supposedly doesn't produce any testable claims?

I think you are right that this question is ill-defined. My ill-defined answer is that I think the chances of winning the lottery are substantially lower.

I really don't understand how those are not dependent. It seems given A, the probability of B would 3/4 instead of the usual 4/6. Doesn't that mean they are dependent?

Edit: wait, apparently 1 is not a prime number. Therefore the probability of B given A is still 2/4 and the probability of just B is 3/6. I guess they are independent.

Here is a similar one. Roll two dice. Let's make them red and blue, so that they are distinguishable. Let

A = "The sum is odd",
B = "The red die is odd".

Just as before, A and B are independent. But then, by symmetry, A is also independent of

C = "The blue die is odd".

But how can A be independent of both B and C? After all, if B and C both happen, then the sum is necessarily even. What is the explanation?

This is language ambiguity: (A is independent of B) and
(A is independent of C) but NOT (A is independent of [B&C]).


Here's a strange question someone gave me years ago:

There are three "independent" observers A,B and C around
a lake and they see a small boat sink somewhere in the
middle of the lake. Their lines of sight are not coincident,
i.e., don't intersect at a point. Also, assume for each of
the lines of sight, 1) they aren't "true", i.e., none of the
lines of sight actually pass through the real location where
the boat sank; 2) each line of sight has a probability of
1/2 to be slightly off to the right or slightly off to the left
of the "true line"; and 3) no two lines are parallel.

Thus, the three lines of sight form a triangle on a map of
the lake. What is the probability that the boat sunk
inside this triangle?


Show that the following naive solution is incorrect:

Since the observers are "independent", and the boat has
equal probability of being on either side of a line of sight,
the probability of the boat having sunk inside the triangle
is (1/2)(1/2)(1/2)=1/8.

I'm assuming this is because

I recently started toying around with Benford's law on sets of naturally occuring numbers.

Now am I correct in saying Benford's law holds because we have a decimal number system and we start counting from 0? Thus when a lot of numbers don't rise above 200, more than half of these will already start with a 1.

Can Benford's law also be used to give a chance to a set of numbers, that doesn't follow this distribution? That is in fraudcases, where the numbers didnt naturally occured, but were modified/created. Can you say: these numbers are off by more than 3 standard deviations so the chance that happens naturally is z?

So not: something fishy is going on, but: there is only a 0.05% of these numbers being distributed like this in a natural way?

Also does (and why would) nature follow a logaritmic scale?

Thanks. That's what I was looking for. Great presentation. Cobwebs are clearing. I may have questions later.


PairTheBoard

Most of the criticisms are that it may not even be possible to connect it with experiment and the fact that some vocal and well known string theorists are claiming that the anthropic principle is necessary to describe the universe. Some opponents say that string theory belongs in the math department, others say it should be in its own department separate from math and physics! As a probabilist, you are pretty far away from all this, but if you have friends interested in representation theory or topology they probably have a stronger opinion.

This question reminds me of the situation we encounter when we build a confidence interval. Invariable, people ask, "What is the probability that the true parameter falls inside this interval?" When asked this way, one might get the impression that the interval is fixed and the true parameter is random. But it is the other way around. The proper question is, "What is the probability that the (random) confidence interval covers the (fixed) true parameter?"

Similarly, in this case, the boat is fixed and the triangle is random. So it might be better to ask, "What is the probability that the (random) triangle formed by the lines of sight of the observers covers the (fixed) location of the sunken boat?" I would suspect a lot of people are led astray by thinking about it the other way around.

Benford's law is not actually a part of probability theory. That is, it is not a mathematical theorem that can be stated and proved mathematically. (In contrast, the law of large numbers and the central limit theorem are mathematical theorems.) Instead, Benford's law is simply an observation about the statistical behavior of a wide variety of empirical data sets. It does not hold universally, nor does it hold exactly. Many data sets do not follow Benford's law, and those that do will follow it only approximately, with varying degrees of "closeness."

In my opinion, Benford's law is not very surprising. Let's take some arbitrary data set:

1691397
14925359
25902260
1358541
1153750
956000523
3603519
619955
4989127
383790212
...

Now take the base-10 logarithm of each number on the list:

6.228245556
7.173924786
7.413337658
6.13307275
6.062111714
8.98045813
6.556726816
5.792360167
6.698024559
8.584093895
...

Now drop the integer part of these numbers:

0.228245556
0.173924786
0.413337658
0.13307275
0.062111714
0.98045813
0.556726816
0.792360167
0.698024559
0.584093895
...

Would you be surprised to discover that this new list of numbers is approximately uniformly distributed on [0,1]? Well, this is just Benford's law! Here is why.

Take some positive integer N. Let X be the fractional part of its base-10 logarithm. Then the leading digit of N is just [10^X], where "[a]" denotes "the greatest integer less than or equal to a." Can you figure out why this is true?


Now, what is the probability that the leading digit of N, which is [10^X], is d? It is just the probability that

d <= 10^X < d + 1,

or

log(d) <= X < log(d + 1).

But if you are sufficiently convinced that X is uniformly distributed on [0,1], then this probability is just

log(d + 1) - log(d),

which is Benford's law.

I have never seen anyone attempt to do something so quantitative, and I would be extremely skeptical of any such analysis.

The claim that "nature follows a logarithmic scale" is, in my opinion, way too broad and sweeping, and just not true. Logarithmic relationships are certainly very common in nature, and can result from physical principles such as the exponential growth of a physical quantity or the memoryless nature of a physical phenomenon. But in this case, the appearance of the logarithm seems to be related more to the connection between the leading digit of a number in base b and the base-b logarithm. This, of course, is a mathematical relationship and not a physical one.

"I have never seen anyone attempt to do something so quantitative, and I would be extremely skeptical of any such analysis."


This guy uses it to detect accounting fraud.
http://nigrini.com/

That doesn't mean he claims to quantitatively determine the probability of "numbers being distributed like this in a natural way." One can detect fraud by simply detecting when "something fishy is going on."

By the way, here is an interesting experiment to try, if you have Excel. I created an artificial data set, ranging over several orders of magnitude, by entering "=INT(10000*(10^RAND())^6)" into cell A1, and then filling that down to cell A10000. Then, in cell B1, I put "=LOG10(A1)", and filled this down to the last row of data. This takes the base-10 logarithm of the data. Next, in cell C1, I put "=B1-INT(B1)". This drops the integer part of what is in cell B1. I filled this down to the last row of data. Finally, in cell D1, I put "=INT(10^C1)". This returns the leading digit of the number in A1. I filled this down to the last row of data. Finally, I used Excel to check the proportions of 1's, 2's, ..., 9's, and I found a very nice match to Benford's law.

Okay, if you could do that, then now try the interesting part. Go back to cell B1 and change it to "=SQRT(A1)". Fill this down to the last row of data. Change nothing else. You will see a new list of digits between 1 and 9 in column D. But this time, they will not appear to have anything to do with the data in column A. Nonetheless, you should still have a very close match to Benford's law.

Now do another tweak. Go back to cell B1 and change it to "=ASINH(A1)". Fill this down to the last row of data. Change nothing else. You will again see a new list of digits between 1 and 9 in column D. And again you should still have a close match to Benford's law.

Lastly, try changing cell B1 to "=PI()*A1", fill it down, and again you should have a very good match to Benford's law.

The point is that the frequencies predicted by Benford's law are coming directly from the fact that the fractional part of the logarithm of the data is uniformly distributed. Just as the fractional part of the square root of the data is uniformly distributed. And the fractional part of the arcsinh of the data is uniformly distributed. And the fractional part of pi times the data is uniformly distributed.

tyvm for these clear explanations!

I am trying to figure out now if certain languages follow a sort of Benford's law in that for example the letter E is followed by the letter N about 10% of the time etc. Then when I feed it a new text from a language I hope it can classify to its closest match.

I have never heard this before. Thanks for the info. I will keep my ears open for this.

Hi jason,

I recently have come to a point where I wanted to measure the variance of the variance of a sample.

after some internet searching, a found a possible answer of:



I also found this paper (pdf) where some guy derives it. He takes all the possible different subsets of the sample, calculates the variance for each subset, then calculates the variance of all these calculated variances. Seems logical I guess, though I'd be easy to fool here

So yeah, any recomendations for how I should be calculating variance of the variance? I am using data sets with between like 5 and 10,000 numbers, messing around in excel and am using vba

I should maybe note the expected distributions will not be normal (forex price changes to be precise)

The paper you linked to assumes your data set is a uniformly selected subset from some larger population. That may not be the best way to be thinking about your data, depending on where it is coming from. It is often better to regard your data points, X_1, X_2, ..., X_n, as realizations of a sequence of iid random variables. The common distribution of these variables may be unknown to you.

Here, it is important to distinguish between the variance and the sample variance. The variance, call it s^2, is the variance of one of the X_j's. It is unknown to you, since the distribution is unknown to you. The sample variance is S^2. This you know, because you just calculate it. The sample variance is itself a random variable, since it is a function of X_1, ..., X_n. One can use the central limit theorem to prove that, when n is large, S^2 is approximately normal (even when the X_j's are not). Its mean is s^2 (provided you use a factor of 1/(n - 1), and not 1/n).

It is then natural to ask, what is the variance of the sample variance? If m = EX_j and r = E[|X_j - m|^4], then the answer is

(r - s^4(n - 3)/(n - 1))/n.

If you knew the value of r, then you could use this result to build a confidence interval for s^2. The trouble is that r is unknown. You will have to estimate it either from the data, or from some other source of information you have about the data set. You can get reasonable looking confidence intervals from either an absolute bound on r of the form r < C, or a relative bound of the form r/s^4 < C.

Somewhat open-ended question ...

When some process is modeled probabilistically, what exactly is happening in physical reality (in your opinion at least).

(The same type of question has been debated for decades about quantum theory.)

In my opinion, the most common scenario is that we either do not know what is happening, or we know, but it is so complicated, or there are so many unknown variables, that a deterministic model is not practical and/or not useful.

It might be worth pointing out that any probabilistic model built in the Kolmogorov system (that is, all of mainstream probability theory) is, in some sense, a model with unknown and/or hidden variables. Our processes may depend on many things, such as time, space, state, etc. But they all necessarily have one variable in common: \omega. A reasonable interpretation in many circumstances is that all of our unknown/hidden variables have been lumped under the umbrella of \omega. We do not know the exact value of \omega, so we impose a probability measure to model our partial knowledge about these unknowns. However, if we were to discover the value of \omega, then, by definition, we would know the exact outcome of our experiment.

Thank you jason, that was extremely helpful

are you suggesting I should assume each X_j has it's own probability distribution?

I believe in my specific situation, it's better to assume each independent X_j is drawn from the same distribution. In fact that is sort of the point.

I am trying to predict the volatility of for example the EUR/USD between Monday 2:00 pm and Monday 2:01 pm, using as data points what has happened during that time interval of the week every week over the last year. So I would have in this example about 52 data points.

So yeah, I am assuming the price change in that interval has the same distribution every week.

So then (r - s^4(n - 3)/(n - 1))/n would be relatively straightforward to calculate yes?