I love New Orleans. I stayed at an ancient hotel in the quarter for two weeks several years back and could have stayed a month if I were independently wealthy.

You don't have to go during any special festivals (and I'd actually recommend NOT going during Mardi Gras - it's just too crowded). It's always fascinating and fun. More so if you enjoy jazz, zydeco, and hurricanes (the drink kind), but even if you're just a people watcher you'll have a blast.