I think it always affects the game if people are doing sidebets and also both involved in the game. The most extreme example is if you manage to get staked by more than one opponent and then try to lose the maximum.
Like lets say it's a 3way pot on the river vs villain A who's crossbooked vs you and villain B who is not.
You're 1st to act and bet for value. Villain A is 2nd and has a decent hand which he's either gonna raise or flat. Usually he has to evaluate if raising is more valuable than flatting, given that raising opens him to a 3bet and flatting has the chance of getting an overcall. But with the crossbook then the raise becomes relatively more lucrative because the overcall isn't as valuable as getting the extra bet from you.
As long as you declare it then it's not dodgy imo. But it changes the dynamics of the game and some people, especially nits, don't like that. They like their comfort zone.