Consider the comparison between this issue and the corresponding issue for interracial weddings. Supports of freedom for baker and florists to refuse gay weddings have in my opinion a fairly strong burden to differentiate the two (presuming they agree that bakers shouldn't be able to refuse service to interracial weddings; not all libertarians are going to agree to that obviously).
From what I can tell, the difference, such as it is, comes down to something like this: Our society does not just have a general claim to freedom. At specific places it specifically enumerates particular protections to particular classes, above and beyond a general spirit of freedom. In the US it isn't just free speech, but free religion and free press specifically. Canada makes this duality particularly clear in the equivalent of the equal protection claus:
Quote:
Originally Posted by Canadian Charter
15. (1) Every individual is equal before and under the law and has the right to the equal protection and equal benefit of the law without discrimination and, in particular, without discrimination based on race, national or ethnic origin, colour, religion, sex, age or mental or physical disability.
Advocates in Canada obviously want "sexual orientation" added to the underlined part.
What I sometimes question is the importance of the enumerated freedoms. As in, is the bolded sufficient, and do we need the underlined at all? If the underlined has an effect - and practically it has in the legal canon - then it is prioritizing particular enumerated differences over others.
Returning to the question, the distinguishing argument between the interracial and gay marriage cases seems to be that religion is not just protected, but specially protected. As a society we have agreed that a religious reason is in some sense more valid than a general reason. In this case, they might argue that a religious opposition to gay marriage is something we should protect but that a not religious opposition to interracial marriage is something we don't need to. Even if the interracial marriage opponents is making some seemingly positive intending argument about trying to improve our society, we have decided that those classes of arguments just don't manage to trump the need to protect racial minorities.
So the gay marriage differs from the interracial marriage in two ways. Firstly, those not wanting to support gay marriages are doing it for explicitly religious reasons. Secondly, sexual orientation doesn't have the explicit protections that race has. So such a person needs to argue that both are necessary. Namely, they need to maintain that we should keep freedom of religion as being somehow greater than a generic non religious argument, that it deserves to retains its privileged sets of protections. And that gay people should not be added to the list of those with privileged protections the way races or genders do.
One part of me says to dispense with the entire issue of enumerated freedoms and enumerated protections. We support general freedom, and general nondiscrimination, and that society and the legal system has the sophistication to work out the balances from there. Perhaps this is naive. Certainly, I think that if we do enumerate protections, sexual orientation should be on that list as certainly as race is. I am seeing less reason to continue with enumerated freedoms. As in, while clearly I don't want discrimination against religious minorities, i don't see what is added to free speech and assembly by free religion. In particular, I don't see that religious reasons should be sufficient to trump a protected class where non religious reasons are not.