How can I adjust the preflop ranges vs different player types? I'm not looking for the common things like, open tighter and 4bet lighter vs an aggro 3bettor, but a more sound and theoretical approach. So if I tweak a range I'd like to know why and that should be supported by mathematics. The problem is that I got no idea where to look and what to look for.

Let's take an extreme example, we are OTB and we want to open vs a big whale in the BB. Postflop he donks 3 streets atc, how can I find my optimal range and my max exploitative range?

I think this is too difficult to calculate by hand. You need a solver.

Luckily, you're not the only one that can't do this kind of math on the fly, so as long as you do it better than your opponents you're good.

Start with your hand, not your range. A max exploitative range is just a collection of max exploitative lines with individual hands.
The two main things you should take into account:
-How often you face aggression either now or on future nodes.
-How often you get a fold either now or on future nodes.

The first one will shift your threshold up, while the second one will shift it down.

To know how to play vs this whale from preflop, you have to think about the river and go backwards.
What made hands yield high EV (>50% equity vs betting range) vs the river jam with A2C?
What hands can I continue on flop with prospects of being strong enough to call river?
What hands can I open preflop with prospect of having enough value to continue on flop?

I guess you would just want hands that flop >50% equity vs A2C very often