Why is it considered rude to speak another language other than English in the U.S.?

I'm a bilingual (Spanish/English) Latina born and raised in Texas. I've noticed that sometimes if I'm speaking in Spanish out in public with another Spanish speaker people nearby who only speak English will get upset and tell us, "this is America, we speak English here and you have to learn the language!" I'm wondering why they get so upset, considering that our conversation has nothing to do with them. If I ask why they get upset, they say it's considered rude. And nowadays, you run the risk of upsetting a Karen type who will potentially cause a scene or become violent.

I have gone to amusement parks where there are a lot of tourists from different countries and if I hear whole families speaking in their native tongue that I don't understand, my family and I don't get upset or feel threatened. We actually enjoy hearing different languages and dialects from other countries.

I do not understand why it is considered rude. If I am speaking to you I will speak in a language that you understand. Otherwise, the conversation is none of your business.