Open Side Menu Go to the Top
Register
How does increasing variance of a distribution affect the tails of that distribution? How does increasing variance of a distribution affect the tails of that distribution?

07-19-2017 , 02:23 AM
I always assumed that higher variance meant more of the distribution would go to the tails instead of less but I'm running into conflicting info from some experts. Who is right?
How does increasing variance of a distribution affect the tails of that distribution? Quote
07-19-2017 , 04:03 AM
How do you increase it? Do you increase a parameter or change the very distribution function itself? If the distribution is normal, increasing variance while maintaining the avg the same you end up having more likely events that were rare before unless you measure in terms of standard deviation units, that remain equally likely. If you were betting 1 unit in a negative ev game and you bet instead everything, you increase the chance to reach an exit target.

Eg at 49% win probability in a win 100% or lose 100% game you should bet all you have if you want to reach 100% gain and quit, instead of betting a little every time trying to reach the same 100% profit exit target. The chance to reach 100% gain is substantial now vs very low if you bet say 10% of your money every time.


If you have a war in a country that is weak and the invader is a superpower you gladly would accept your king (or best soldier) to fight their king (or best soldier) in a duel that ends the war.

You can have a distribution that has parameters in it that affect the tails. If your distribution is remaining functionally the same though then the distribution of tails is the same in relative terms (eg to standard deviation units a 3sd event is similarly rare but it is now a much more distant place than before) its just that what was outliers before have become less rare now etc.

You can also have something like k*(1/((x-a)^2+b^2)+1/((x-c)^2+d^2)) that increasing a and c increases variance and so does reducing b and d.

Last edited by masque de Z; 07-19-2017 at 04:16 AM.
How does increasing variance of a distribution affect the tails of that distribution? Quote
07-19-2017 , 05:53 AM
Quote:
Originally Posted by masque de Z
How do you increase it? Do you increase a parameter or change the very distribution function itself? If the distribution is normal, increasing variance while maintaining the avg the same you end up having more likely events that were rare before unless you measure in terms of standard deviation units, that remain equally likely. If you were betting 1 unit in a negative ev game and you bet instead everything, you increase the chance to reach an exit target.

Eg at 49% win probability in a win 100% or lose 100% game you should bet all you have if you want to reach 100% gain and quit, instead of betting a little every time trying to reach the same 100% profit exit target. The chance to reach 100% gain is substantial now vs very low if you bet say 10% of your money every time.


If you have a war in a country that is weak and the invader is a superpower you gladly would accept your king (or best soldier) to fight their king (or best soldier) in a duel that ends the war.

You can have a distribution that has parameters in it that affect the tails. If your distribution is remaining functionally the same though then the distribution of tails is the same in relative terms (eg to standard deviation units a 3sd event is similarly rare but it is now a much more distant place than before) its just that what was outliers before have become less rare now etc.

You can also have something like k*(1/((x-a)^2+b^2)+1/((x-c)^2+d^2)) that increasing a and c increases variance and so does reducing b and d.
Here is the info that prompted me to post this

https://twitter.com/nntaleb/status/8...120833?lang=en

I think you raised a similar point as him, you brought up the idea of a threshold in terms of being a 100% winner or winning a war. Im still somewhat confused by what he is saying overall, can you explain it better than he did?

Last edited by Bulrathi; 07-19-2017 at 06:11 AM.
How does increasing variance of a distribution affect the tails of that distribution? Quote
07-19-2017 , 06:01 AM
I have to read more the sequence there but for example if a population has avg IQ 105 and sd 15 and another population has avg IQ 100 but sd 20 then there will be more often IQ >160 IQ people in second population even with smaller avg because now say a 2.4sd event puts you to 160+ in second but it is 2.75sd in the first.

The bell curve opens up more with higher variance making rare events before more likely now if one uses the same old absolute scale to measure "rare".
How does increasing variance of a distribution affect the tails of that distribution? Quote
07-19-2017 , 06:33 AM
Quote:
Originally Posted by masque de Z
I have to read more the sequence there but for example if a population has avg IQ 105 and sd 15 and another population has avg IQ 100 but sd 20 then there will be more often IQ >160 IQ people in second population even with smaller avg because now say a 2.4sd event puts you to 160+ in second but it is 2.75sd in the first.

The bell curve opens up more with higher variance making rare events before more likely now if one uses the same old absolute scale to measure "rare".
I see. So he is saying *IF* IQ helps people to succeed, then success of a population would depend more on the variance and the frequency of significant outliers rather than average IQ of the population. So, as an example, when researchers try to measure avg IQ of countries to see if it helps explain differences among countries, they are probably looking at the wrong thing.
Is that what he is saying?
How does increasing variance of a distribution affect the tails of that distribution? Quote
07-19-2017 , 03:45 PM
On the IQ issue, Taleb is being a loser. Yes, variance matters if means are the same or if it's very different. But take a population of average 85 IQ. There variance would have to be enormously larger to merely match the 105 IQ population at the high end tails which Taleb claims (and I agree) are what matter for how a civilization progresses. Very different variances are not the case, in reality. It's the mean that matters here as the variance is the same. So he's just throwing out red herrings because he's a coward.

As for the rest, a bell curve is just a statistical approximation of an underlying phenomenon. It's not real. Where does the greater variance come from, is what matters. If you're in Africa and childhood encephalitis is an IQ killer, then there'll be more variance due to this, but it'll only fatten the downside tail. Similarly if a population has more Jews, it'll fatten the upside tail without a particularly noticeable difference in variance.
How does increasing variance of a distribution affect the tails of that distribution? Quote
07-19-2017 , 06:31 PM
Quote:
Originally Posted by ToothSayer
On the IQ issue, Taleb is being a loser. Yes, variance matters if means are the same or if it's very different. But take a population of average 85 IQ. There variance would have to be enormously larger to merely match the 105 IQ population at the high end tails which Taleb claims (and I agree) are what matter for how a civilization progresses. Very different variances are not the case, in reality. It's the mean that matters here as the variance is the same. So he's just throwing out red herrings because he's a coward.

As for the rest, a bell curve is just a statistical approximation of an underlying phenomenon. It's not real. Where does the greater variance come from, is what matters. If you're in Africa and childhood encephalitis is an IQ killer, then there'll be more variance due to this, but it'll only fatten the downside tail. Similarly if a population has more Jews, it'll fatten the upside tail without a particularly noticeable difference in variance.
No. His broader point is that in a winner take all economic/scientific world *IF* you want to prove that IQ correlates to success, you have to look at tails and see if that relates to success. Not average IQs. Looking at the average IQ ignores winner take all dynamics/Pareto 80-20

Last edited by Bulrathi; 07-19-2017 at 06:38 PM.
How does increasing variance of a distribution affect the tails of that distribution? Quote
07-19-2017 , 06:32 PM
I understood his point perfectly and covered that in my post. My post is responding to exactly what you're summarizing Taleb as saying. And I agree with him that the tails matter, to some extent.

However, average intelligence matters too. A society with an average IQ of 85 doesn't have enough skilled/trainable workers even if it has genius IQs on the fat tail. Average IQ of 85 simply can't sustain the intellectual and economic culture for the high IQs to thrive and build a society.
How does increasing variance of a distribution affect the tails of that distribution? Quote
07-19-2017 , 06:54 PM
Quote:
Originally Posted by ToothSayer
I understood his point perfectly and covered that in my post. My post is responding to exactly what you're summarizing Taleb as saying. And I agree with him that the tails matter, to some extent.

However, average intelligence matters too. A society with an average IQ of 85 doesn't have enough skilled/trainable workers even if it has genius IQs on the fat tail. Average IQ of 85 simply can't sustain the intellectual and economic culture for the high IQs to thrive and build a society.
You said that Taleb claimed that the "population at the high end tails" are "what matter for how a civilization progresses", which you agreed. He never made such claim, in fact, he sounded quite skeptical of the idea of using IQ for such things as he believes the ability to tinker is more important than IQ. So it looks like you didn't understood his points "perfectly". Furthermore, you make claims about what society cant or cannot do based on IQ merely by stating, without backing much of anything up
How does increasing variance of a distribution affect the tails of that distribution? Quote
07-19-2017 , 07:50 PM
He appears to be right behind the intelligence tail thesis. He just doesn't like the term IQ, hence he uses "IQ", by which he means intelligence/talent. He has exactly the opinion I put him on:
Quote:
4) This threshold phenomena is NECESSARY in the presence of winner-take-all in scientific & economic success (s. a. Pareto 80/20).
He even quotes his own theory of 80/20.

Anyway, this is his tweet which I criticized:
Quote:
Originally Posted by Questioner
Assuming "IQ" helps success; the performance of a population would depend >>more on the variance than average. (convexity, #Antifragile)
It's a cowardly red herring, because it's completely irrelevant to the point at hand, yet he uses it to dodge, which was my point. Incredibly cowardly. When someone brings up the obvious, he retreats further into cowardice:
Quote:
May be a silly question, but what if the variance is the same in two popns, one high IQ, the other low IQ?
Quote:
Originally Posted by Taleb
The more in the tails, the less the mean matters.
How does increasing variance of a distribution affect the tails of that distribution? Quote

      
m