Remove/hide the hats
Remove the hats, it doesn't bring any value to the player. Worse, in early game, other users tend to put the robber on the users with a hat more often then not. Hats put a big target on the players' back and it is very frustrating. For that reason, I tend to try and play games with other hats.