If you are swigging out of a bottle inside a brown paper bag then yeah, probably low class. If you are drinking it out of a crystal double old fashioned glass in your library then maybe not.
The only person I knew who drank Wild Turkey (and loved it), also lived in a tiny cabin in the mountains, was a self-described proud redneck, hated "city folk" and made peach moonshine. He would also keep his Wild Turkey hidden in the bushes of his In-laws home so he'd have a stash whenever he had to go there.