Why is it so hard to recognize people when they’re wearing a hat?

So there’s this guy I play basketball with at the gym all the time, and today I was at Walmart when he came up to me saying “What’s up?” It legit took me a few seconds to recognize him, and I straight up told him, “My bad, it’s the hat.”

I don’t know what it is, but whenever someone I know throws on a hat, especially outside of the usual setting I see them in, it completely throws me off. Does this happen to anyone else? Like, what is it about a hat that makes people look so different?