I agree that it excludes a lot of us (your first paragraph I match totally, not the second because I am in no sort of relationship at all). However, I disagree that it is prejudiced. It just doesn't apply. Neither of us can say how it affects our marriage, because we don't have a marriage. The most we could accurately say is something like "If I were to be married, the probability of which is so close to zero as to be meaningless, then I believe that what other people are also permitted to be married would not affect my marriage", which is so weak a statement that there's no point in saying it.

It's like saying that a meme about how one feels as a driver is prejudiced against non-drivers, or one about the extension of pub drinking hours if one is not a drinker. They simply don't apply, and there is no reason for them to apply. If you have an opinion on the subject, state it freely, but you can't state it as being from something you're not. I couldn't say "as a golfer, I'm against the closure of golf courses" because I'm not a golfer.