Demonstrators in the rally in central Trafalgar Square waved Union Jack flags and held up signs reading "Let's stay together" and "Scotland we love you, don't go."
"I am beside myself with worry that Scotland might be separate from the rest of the UK. It will be an absolute travesty and terrible," said Shona Milne, 56, a longtime resident of England from Glasgow.
"I am distraught and I know millions of Scots are feeling the same," added Milne, who was wearing a "Proud to be Scottish and British" t-shirt.
Polls have shown the result could go either way when Scots cast their votes on Thursday in the historic referendum on the 307-year-old union.
Organiser Dan Snow, a broadcaster and historian, told the crowd: "We think that unity is better than division, and cooperation is better than competition."
Irish rock musician Bob Geldof, a long-time resident of England, said the United Kingdom was "one of the greatest ideas ever invented for the modern age".
The rally was held as Conservative Prime Minister David Cameron visited Aberdeen to urge a "No" vote to independence, warning in a speech of dire consequences and a "painful divorce".
Scottish National Party leader and Scottish first minister Alex Salmond argues the country would be better ruled by a national government and has the resources to prosper.
Student Tim Fell from Edinburgh, 34, explained he understood the resentment over Scotland's economic struggles with de-industrialisation but that it was time to move on.
