Say we get to river with ~second-fourth nut-flush (or worse). Villain bets. We raise. Villain shoves.

This question is mostly for hold'em but playing PLO has made me wonder about this more.

From a GTO perspective, how do we figure out which hands we're supposed to call with? For sake of argument, suppose our range after raising is mostly flushes, some weaker hands (maybe sets or straights), and maybe an occasional bluff with dry ace. Now let's say that villain shoves with mostly some mixture of nut or strong flushes and an occasional dry ace. So basically there are no (or very few) bluffs in villain's range when we have the nuts, but if we don't have the nuts, villain can have a bluffing range.... I'm just not sure how the math works out when ranges interact with each other this way.

It's almost never true in that spot that there are flushes we fold 100% and flushes we call 100%. That said, what you want to look at is if your hand blocks bluffs, and in the case where it's 3 card flush on board, you want the flush combo that block the most number of Axss flushes from villain's range.

If there is a 4 card flush on board, then obv we don't block the nuts with any combo- so we just wanna unblock potential bluffs he has.

I'm thinking of 3-flush boards, and not necessarily what's right in practice (because we can probably be somewhat exploitable in these spots against most player pools), but only how to figure this out from a game theory perspective. I'm a bit lost when blockers come into play b/c on one hand it seems like we don't want villain to be able to always exploitably shove and get folds when he has the dry ace, but on the other hand, dry ace should be a small part of villain's range in most situations - especially in hold'em... If villain is able to profit from bluffing with dry ace, does that necessarily mean we're overfolding GTO-wise?

In terms of perfect GTO then yes, you should be playing a strategy that villain cannot profit from in any given spot. You're not going to be able to play GTO though, it just isn't possible, so I wouldn't get too bogged down in it.

If you're asking purely out of interest, then from a theoretical standpoint you should always have combos that aren't A high flushes to call with here to be unexploitable. You can do a load of maths to determine the precise number of flushes you need to call here but I think it is more interesting discussing the redeeming features of certain combos than the maths.

If villain was the PFR or 3 bet preflop, your best candidates for calling are obviously K high flushes as you block AKs and are at the very top of your range (KQs would be the best K high candidate as you block AQs too). If villain overcalled your raise preflop, you could potentially make an argument that from a purely theoretical standpoint, a flush like QJs would be a better candidate as you block AQs and AJs and villain will normally be 3 betting AKs. This is only valid if the river action is truly polarising (ie. villain is never doing this with a K high flush).

Although this is an interesting thought, I don't think it is very applicable. Typically, certainly at lower stakes, people will be underbluffing this spot and either overfolding/overcalling. As a result, i probably overfold this spot vs GTO. I think there's more EV to be made by basing your decisions on your opponent and his tendencies than to worry about particular blockers.