Then you must accept that medieval cultures of the Muslim world and stone age African cultures cannot be considered equal to modern Western society. They are, in fact, worse.
And any newcomers from such a culture must be educated and integrated before being allowed free access to XXI century people.

Certain cultures are inferior to others in their ideology and thinking.

Just not seeing Hindus or Christians running around burning people to death or blowing them up.

Is there a big problem with time-traveling Muslims and Africans on the loose in the countryside?

That's the problem actually. They don't travel with time. Their cultures are stuck.