From seaside inns to boutique shops, locals balance small-town charm with global stardom as speculation swirls around Swift’s mansion.