while i see the merits of banning material that is harmful to society, i think it is dangerous to give the state the authority to decide what is or isn't harmful. many people think that the western world's culture of overconsumption will be its downfall (not just of resources, but buying things people can't afford in general), so for instance would you be in favor of censoring the amount of advertisements to which we're introduced to make sure we don't continue to consume at this rate? i think it becomes difficult to decide where to draw the line.
Your first point would be a contradiction because other than the hard-line anarchist, all individuals in free societies give consent to government to decide issues on their behalf. Hence, representatives are employed by the state, be it in government or law, to decide what is harmful to society. Additionally, most governments go further than merely legislating against harm-to-others, and use paternalistic legislation as well.
Getting back to the where the line is drawn. Now it is difficult to imagine given that we don't have a concrete example in front of us, but it is perfectly possible, given the involvement of enough bodies in a decision-making process, to decide whether or not an act or publication should be deemed impermissible. And there are bodies that exist that do provide a check and balance against this procedure. In the UK there is the Lords and additionally there is the European Parliament. Both can overrule decisions made by government or councils that are to become law. I expect your American constitution provides similar cover (but update it TS, please).
also i think its somewhat absurd to be threatened by specific ideas. for instance, when someone reads a piece of racist literature, they should have the ability to tell themselves why they disagree with the ideas expressed, otherwise people go about believing ideas solely because they've been told to. (i don't believe people should not be racist just because they think its the way they should be, i'd rather they understood why racism is bad). it may be idealistic but i'd rather the state foster peoples ability to discern good ideas from bad, and i'd prefer ideas be discussed in open forum, so society would be less resistant to outside ideas.
I don't think ideas are threats in themselves, but what can be done with an idea can be the threat. What would have happened, for example, if anti-semitism had been criminalised in Germany in the '30s? Would 6 million Jews still have suffered at the hands of the Nazis? Okay, I'm using a "from small acorns' argument here. But it really was from a small acorn that anti-semitism took off in the way it did. And look where it took Germany.
You'd probably get on with JS Mill (or you would do if he was still alive). Mill's idea was the state only legislates against acts that will cause harm to others. Self-regarding wrongs, might be wrong, but are not the business of the state. The problem with Mill is that only the most trivial act can be said to be solely self-regarding. If I take drugs, my family suffers and perhaps I support a vast criminal empire in Africa (and with all that entails). If I publish non-inciting racist literature, I help promote that world in which the evil is allowed to fester relatively unchecked. If I'm a gambler, I threaten the comfort of my children. I think you get the picture and can quite easily play counterfactuals or cause-and-effect yourself.
Now, I think Mill was aware of this (although bizarrely he didn't mention that he was). His solution was to teach morality (citizenship) at school. This would be the Religion of Humanity. You seem to be going in that way yourself with the idea that the state fosters an environment that encourages people to be able discern bad ideas (or that is what it appears from above). That sounds all well and good, but then are we not just getting back to the state telling us what to believe? You can hardly expect each individual to go through years of training in the art of philosophy. The best we can get is the state teaching good ideas, nothing more, nothing better. Some might say we live in a very Millian world already, with political correctness reigning supreme in our schools, councils and media outlets (ask TX Bill what he thinks about that). And let's face facts on this, some of the morons that exist in these establishments and institutions are far from competent to be our moral or political guardians.
But there are no easy answers to any of this, which I think you agree with.
Anyway, I'm starting to bore myself now