Unfair?

Is it fair to kids whose parents aren't exactly classical musicians?  How early do you have to start to not be considered as going in a stupor when you hear orchestral (classical or pop) music?