Specifically bishop x bishop, early in game where the only benefit would be a less than great pawn structure my opponent would end up with.

Make the trade or let it go?

Trade pieces if it spoils the pawn cover of the opponent's king and you're left with yet enough pieces to sustain a mate attack.

Spoiling the pawn structure in the centre or on the opposite flank is irrelevant until the endgame, and can even backfire on you in the middlegame if it opens a key line for rooks, so don't trade pieces there if you think you'll have a bigger edge (or a smaller disadvantage) in the middlegame (e.g. initiative and attack) than in the endgame (which yet has to be converted).

Trade your inactive pieces for opponent's active ones. When you're up material, trade pieces because it's easier to grab pawns then; when you're down material, trade pawns to prevent their chains from being eaten and increase the chance of a pawnless (hence usually drawn) endgame.

These are just standard rules of thumb as I understand them, it always depends on many nuances of the position, not only the pawn structures.

In general, don't discuss things abstractly, post specific positions where you were hesitant.

It depends. The simple fact of giving an opponent doubled pawns is not always advantageous. Sometimes doubled pawns are beneficial for covering key squares. Sometimes the side with doubled pawns can benefit by using an open file. Like everything in chess it depends on the specifics of the position in question.

The key phrase in your initial question is "if it adversely affects". Given that, and no other information, then the answer is probably "all other things being equal, yes, trade."

When it becomes a real position, though, it's not that simple. You have to be sure you're not giving up a strong piece for a weaker one (relatively speaking). You have to be sure the effect on your opponent's pawn structure actually is "adverse" (not just superficially so). You have to be sure you're not providing your opponent an upside (like an open file for a rook) that might offset any downsides of the pawn structure. You have to be sure you're not conceding a positional weaknesses in your own camp that might matter more. And of course, as with every chess move you ever evaluate, you have to be sure there's no tactical response your opponent can play that might make all your strategic concerns irrelevant.

All of that being equal, though? Sure, create the weakness.