Action:
HJ(168bb) open 3bb (pot 4.4bb)
CO(80.2bb) 3bet to 9bb (pot 13.4bb)
BTN(44.4bb) 4bet and is all in for 44.4bb (pot 57.8bb)
Hero in BB(168bb) 5bet and is all in for 168bb

Estimations:
HJ will continue 10% of the time, when he does he has 71% equity vs Hero
CO will continue 50% of the time, when he does he has 68% equity vs Hero
BTN is already all in and has 53% equity vs Hero

I thought how to put it all together and this what I came up with

first:
BTN and Hero are both all in so they will go to showdown 100% of the time
Pot is 57.8bb
Hero will win 47% of the time
Hero needs to put 43,4bb in the pot to match BTN cold 4bet to 44.4bb
therefore
+(0.47*57.8bb)-(0.53*43,4bb)=4.16bb

second:
HJ calls Hero's shove 10% of the time
Hero has 29% vs HJ calling range
when HJ calls Hero will win 57.8bb+165bb
Hero and HJ are both 168bb deep so Hero would lose 167bb
therefore
0.1*(+(0.29*(57.8bb+165bb))-(0.71*167bb))=-5.39bb

third:
CO calls Hero's shove 50% of the time
Hero has 32% equity vs CO calling range
when CO calls Hero will win 57.8bb+71.2bb
Hero and CO have 80.2bb effective, so Hero would lose 79.2bb
therefore
0.5*(+(0.32*(57.8bb+71.2bb))-(0.68*79.2bb))=-6.28bb

so we have:
+(0.47*57.8bb)-(0.53*43,4bb)+0.1*(+(0.29*(57.8bb+165bb))-(0.71*167bb))+0.5*(+(0.32*(57.8bb+71.2bb))-(0.68*79.2bb))=-7.51bb

Am I calculating EV of this spot correctly?




Also fwiw I'm gonna put in the spoiler actual hand and my estimations in terms of ranges

One thing you aren't considering from what I can tell at a galnce is when both hj and co call jointly.

Also their calling decisions aren't necessarily independent in other ways. For example HJ folding means CO can call with all of his hands that are at or above the equity threshold. If HJ calls some of those hands could possibly no longer have the correct equity and pot odds to call.

In addition to JG’s comments, you made other mistakes.

For example, you have hero vs. hijack as a confrontation. But Button has to be in every confrontation. And the occurrence probability of Hero vs HJ and Button has to include the probability that Cutoff has folded. In addition, the win probability of Hero vs. two opponents is not specified. In general, it is not equal to the product of the individual win probabilities. If you have estimates of opponent ranges, use a poker equity calculator to get the multiway win probabilities.

I can fix this

So the situation when they both call would only occure when HJ calls, thus we need to include these calculations in this part of the formula: 0.1*(+(0.29*(57.8bb+165bb))-(0.71*167bb))=-5.39bb

I estimate that CO will call a shove when HJ already called 30% of the time
versus both CO (the change of his range when HJ called is considered) and HJ ranges Hero has 18% equity
also, when CO calls, money loss does not change, but to money won we need to add the rest of CO stack, that would be +71.2bb
therefore this part of the formula would look like this:
0.1*(0.7*(0.29*(57.8bb+165bb)-0.71*167bb)+0.3*(0.18*(57.8bb+165bb+71.2bb)-0.82*167bb))=-6.29bb

When I was constructing CO calling range I didn't account for the probability of HJ calling, so that range doesn not change (as if HJ is already folded), when HJ calls I made CO calling range tighter and included new estimations in the necessary part of the formula above

this taken into account here, I added 0.7* and 0.3* according to my estimates in the second part of the formula



So in every confrontation we need to include BTN range, thus we need to change second and third parts of the formula:
0.1*(0.7*(0.29*(57.8bb+165bb)-0.71*167bb)+0.3*(0.18*(57.8bb+165bb+71.2bb)-0.82*167bb))=-6.29bb
and
0.5*(+(0.32*(57.8bb+71.2bb))-(0.68*79.2bb))=-6.28bb

versus HJ range when he calls and BTN range Hero has 22% equity
versus HJ range when he calls, CO range when he calls after HJ calls and BTN range Hero has 15% equity
therefore second part of the formula would look like this:
0.1*(0.7*(0.22*(57.8bb+165bb)-0.78*167bb)+0.3*(0.15*(57.8bb+165bb+71.2bb)-0.85*167bb))=-8.62bb

versus CO calling range and BTN range Hero has 24% equity
therefore third part of the formula would look like this:
0.5*(0.24*(57.8bb+71.2bb)-0.76*79.2bb)=-14.61bb

So when we put it all together we get this:
(0.47*57.8bb)-(0.53*43,4bb)+0.1*(0.7*(0.22*(57.8bb+165bb)-0.78*167bb)+0.3*(0.15*(57.8bb+165bb+71.2bb)-0.85*167bb))+0.5*(0.24*(57.8bb+71.2bb)-0.76*79.2bb)=4.16bb-8.62bb-14.61bb=-19.07bb

Is EV formula correct now?

If you did the things you said you did and did them correctly then I would say you are probably Ok.