The Prime Minister is said to have been in two minds about remaining in the EU, but Iggy Ostantin reveals he supported far-right figures who opposed it