Well, it's not like the people had control over the empire. The government did, specifically queen victoria. The people who hate immigration the most are citizens of the UK, which hardly have control over this stuff that the government does.