What is the formula for finding how often a big bet needs to be called with the nuts for it to be better than a smaller bet. So say I have the nuts and I bet 2x pot and its called 30% of the time vs I bet 2/3 pot and its called 60%. I know in theory if they defend correctly the bigger sizing is the best but people don't defend correctly.

Thanks

Pre river betsize for me is a function of stack sizes, implied odds, and the way these factors determine how I can best increase my share by influencing the opponents range through potsize manipulation.

River betsize for me is split into a few categories of situational ranges:

Out of position after calling the turn facing a static river card. (Empty betting ranges)

Stack to pot = <3:1 after betting turn, (polarized with extremely variable size)

Stack to pot >3:1 after betting turn, (polarized with slightly variable bet size not exceeding 2xpot)

After the turn checks through (polarized with slightly variable bet size)

When you ask for a formula, you are naturally looking for a math approach. Since EV is a prime criterion for making a math-based poker decision, here is the EV equation for making a bet that is either called or folded by a single villain:

EV = fe*Pot + (1-fe)*(eq*(Pot + 2*Bet) – Bet)

Pot = pot amount prior to your bet
Bet = bet amount that villain must call
fe = fold equity = villain fold probability
eq = hero win probability if called

You stated two cases: A. 2x bet and fe=0.70 B. 2/3x bet and fe = 0.40. While you specified the nuts, you did not specify a value for eq. So, we’ll do so and assume eq = 0.85. Then applying the formula, we get the following, assuming pot size = 1:

Case A: EV = 1.38 Case B: EV = 1.19

For Case A villain is indifferent to calling or folding at eq= 60%. For Case B indifference is at eq = 71.4%.

Note that these results are exact if the call closes the action, e.g. a river call. Otherwise it is only a first cut analysis.

There is a lot more math that can be done such as finding the bet or equity value that determines break-even (EV=0). Or, for a more complex analysis, you can develop a fold equity function that depends on bet size and your assessment of villain’s folding tendencies. Here is an example of one I have used, which is S-shaped:
fe = 0.982 - 1/(1 + exp(L*R -S)))

L and S are location and shape parameters that depend on how you assess villain’s folding tendency. R is a risk/reward measure and equals Bet/(Pot+Bet), the higher the risk relative to the reward, the less likely villain will call. Using this model in conjunction with the EV equation, enables one to develop general criteria for a good math play.

Naturally, a pure math-based decision criterion should be adjusted to account for factors not explicitly considered such as stack sizes, position, tournament status, etc.

OP specified he has the nuts, which I assume means he wins 100% of the time when called.

The EV maths is fairly simple from that point. In basic language,
EV = (How much you win when he folds * how often he folds) + (how much you win when he calls * how often he calls)
EV = (folding frequency * pot) + (calling frequency * [pot + betsize])

EV of betting 2x pot with the nuts and getting called 30% of the time = (70% * pot) + (30% * [pot + 2pot])
= 0.7p + 0.9p = 1.6pot

EV of betting 2/3 pot and getting called 60% of the time = (40% * pot) + (60% * [1p + 2/3p]) = 0.4p + 1p = 1.4p

Conclusion: Bigger is better in this example.