Here's what you do: get a bike rack and bungee core a milk crate to it. Bonus points for a milk crate that looks well worn and like it lived an entire lifetime before it took up residence on your bike. The look is timeless, a classic of style, unquestionable. From there you can store anything you want, hat included.
Or don't. You wouldn't ask a stranger on the street for their opinion, so why care what they think?
It's been my experience that the areas most often referred to as "bad parts of town" are the areas with the most people squeezed in without consideration for anything else. Small homes can be fine of there are other outlets in the area such as community centers, parks, libraries, stores, etc. Without those you just concentrate too much human suffering in one area.