For everyone who loves this novel, there is someone who hates it. In most cases, this is simply a matter of personal choice, though I think this is a novel more for adults and shouldn't be taught until at least senior year in high school.
I don't think the material is objectionable for anyone younger than 17, but that it is an age thing. In many ways, the book is nuanced and, sometimes it is taught wrong. It's about sin. It's important to remember this. Hester is not innocent and does not see herself as such. It's about levels of sin.
Some people feel that the adultery aspect of the novel is one that gets people's panties in a twist. But that's life, and how often is that plot point used in all the media around us every day? At least Hawthorne deals with it in an adult manner.
Personally, I think that book banners (and challengers) get upset with Hawthorne's comment on hypocrisy - a greater sin in Hawthorne's eyes. There is only one sin worse - the willful destruction of a human heart - hmm, maybe that on gets them upset to.