Olney is running against Zac Goldsmith, who held the seat for Prime Minister Theresa May's Conservative Party but quit in protest after the government backed expanding the nearby Heathrow Airport.
He is now standing as an independent candidate.
"Speaking to voters, what was becoming increasingly clear was that while some people feel very strongly about Heathrow expansion, lots more people feel much more strongly about Brexit," Olney told AFP.
"That's really alarmed and upset people and they want to use this opportunity to send a message."
In the June referendum on Britain's membership in the European Union, 52 percent nationwide voted to leave.
But in the well-heeled borough of Richmond in southwest London, 69 percent voted to remain in the bloc, the 19th highest of 326 voting areas.
The 82 per cent turnout, one of the highest in the UK, showed it was an issue locals felt passionate about.
The centrist and unambiguously pro-EU Lib Dems, reduced to a rump in the 2015 general election, are eyeing a comeback by filling the void for disgruntled 'Remain' voters.
"The things people want to talk about in this campaign are things we've had a very clear position on," said Olney, meeting commuters outside Richmond railway station.
The 39-year-old accountant only joined the Lib Dems last year and was rapidly selected as their next Richmond Park candidate for the House of Commons.
Former Lib Dem leader Paddy Ashdown, 75, said Thursday's vote could "send a message right to the heart of Downing Street".
"Richmond can speak for the people of Britain, the millions who want the government to change course and don't want a hard Brexit. If they vote Liberal Democrat, that's a message Downing Street will hear," he told AFP on the campaign trail.
However, the Lib Dems face a tough task in overturning Goldsmith's majority. Both the Conservative and Brexit-cheerleading UKIP parties are giving him a clear run.
Goldsmith won Richmond from the Lib Dems in 2010 and retained it in the May 2015 general election with 58 percent of the vote.
Bookmakers have Goldsmith as the 1/3 favourite, with Olney 5/2, and then the main opposition Labour candidate at 200/1. Five others are also standing.
Wealthy, suave, affable and soft-spoken, 41-year-old Goldsmith is Brexit royalty.
Zac Goldsmith made a bid for London mayor in May but he was beaten by Labour's Sadiq Khan and was criticised for the tone of his campaign, often focusing on Khan's Muslim faith.
AFP sought an interview with Goldsmith but he is focusing on private grassroots campaigning.
And his understated, local approach seems to have won him support around Richmond.
"There's a lot of loyalty to Zac round here," said Jane McCready, 52, who sat enjoying a hot drink overlooking the River Thames.
But she added that Brexit was an issue: "I do have friends who are using this as another vote for 'Remain' (in the EU) by voting Lib Dem.
(This story has not been edited by Business Standard staff and is auto-generated from a syndicated feed.)