"In any society freedom of thought will probably be of direct significance for a small minority. But this does not mean that anyone is competent, or ought to have power, to select those to whom this freedom is to be reserved."