The complicated love-hate relationship of immigrants from former colonies with the British Empire cannot be ignored if lessons are to be learned in post-Brexit Britain