I agree with you generally, but there are two things operating here - reading (or listening to) a work created when the sensitivities of the world were very different, and being aware of those issues is not quite the same as promoting a contemporary work where the author(s) views are out of step with the prevailing morality.

Nevertheless, I've never thought that the views (or sex) of a creator should be taken into account when accessing their work, particularly when those views may not be evident in the work. As you say, make up your mind on the basis of what's in front of you, not the baggage it comes with.