The UK Independence Party has gained its first elected MP, with Douglas Carswell taking the seat of Clacton by 12,404 votes reported BBC.
Mr Carswell, who defected from the Conservatives, knocked his old party – which enjoyed a 12,068 majority at the 2010 election – into second place.
UKIP leader Nigel Farage said he had “shaken up British politics”.
In the night’s other by-election, Labour held on to Heywood and Middleton but UKIP slashed its majority to 617.
Mr Farage told BBC Breakfast UKIP could hold the balance of power after next year’s general election.
He said UKIP’s second place in Heywood and Middleton was “even more significant” than its win in Clacton, saying the party was now the main opposition to Labour in northern cities.
“Something big is happening here. People want change, they have had enough of career politicians in three parties.”
Mr Farage said he expected more Conservative MPs to join UKIP following the defection of Mr Carswell and Mark Reckless, who has triggered a by-election in Rochester and Strood, Kent.
“I think it would be very surprising if more people did not come across”, he said, saying he had also spoken to Labour MPs “frustrated that they are not able to change things in British politics”.
Conservative Party Chairman Grant Shapps told BBC Breakfast the result in Clacton was a “wake-up call” saying that while UKIP were a threat to all the Westminster parties, they “cost Conservatives seats”, so put Labour leader Ed Miliband “one step closer to Downing Street”.