I get a remark every now and then when I wear my Confederate do-rag while riding my bike. They say "people will think your a racist". I tell them no I'm just a proud Southerner. Theres a difference.
Of course they will sometimes say it represents slavery, but if thats true then I guess the same can be said for Old Glory as well since it was around before the Civil War.
People will find a reason to be offended about anything only because they want to be offended.
Live and let live.