When's a better time than Christmas to do some EV calculations? Pretty much any time but, alas...

I'm trying to work out what proportion, p, of hands I should raise from the BTN if:

P(Villain folds) = f, Effective stack = S

(I also made the assumption that they only shove or fold and that I call 20% of the time and when I do I have 60% equity against their range - not sure if they're valid assumptions either).

E(raising BTN with proportion p) = I raise, they fold + I raise, they shove and I call - I raise, they shove and I fold - I fold
= 1\*pf + S\*p(1-f)\*0.2\*[0.6-0.4] - 2\*p\*(1-f) - 0.5\*(1-p)

When plugging in numbers of them folding 50% of the time with a stack size of 10bb I get that this is greater than 0 if p > 2.5 which is obviously nonsense, where am I going wrong?

The whole premise of your calculation is wrong. The specific % of hands you minraise is not a direct function of your opponent's strategy because each hand in that % has different EVs. Your approach has led to blunders like including the expectation of open folding in your computation for the expectation of minraising--clearly this is wrong since once you've minraised, you can't open-fold.

Instead you should be looking to find cut-off hands--that is find hands that are very similiar in EV minraising and open-folding, and then all hands worse are clear folds and all hands better are clear opens. Total up all the hands you're opening and you will find your open %.

With your assumptions of no flatting and 50% 3-bet shove you'll find that all hands that minraise/fold have the same EV minraising as open-folding (though in practice there will be card-removal effects that will change this a bit...). All hands that can call a shove will be strictly better minraised then open folded. This means your strategy can open 100%, or open just your min-raise call hands, or something in betwen--ev of these strategies will be the same.

That being said villain not flatting is a horrible assumption

Yeah seems like an unnecessary calculation. The only thing you need to care about is card-removal when you minraise fold against someone with no flatting range oop.