Researchers from the King's College London in brain tests and analysis of health and lifestyle data of a group of over-50s found that smoking affects the brain negatively even more than high blood pressure and obesity.
Participants took brain tests like learning new words or naming as many animals as they could in a minute, the BBC News reported.
They were all tested again after four and then eight years.
Researchers found a "consistent association" between smoking and lower scores in the tests.
The study of 8,800 people also found that high blood pressure and being overweight also seemed to affect the brain, but to a lesser extent.
Scientists involved said people needed to be aware that lifestyles could damage the mind as well as the body.
The results showed that the overall risk of a heart attack or stroke was "significantly associated with cognitive decline" with those at the highest risk showing the greatest decline.
"Cognitive decline becomes more common with ageing and for an increasing number of people interferes with daily functioning and well-being. We have identified a number of risk factors which could be associated with accelerated cognitive decline, all of which, could be modifiable," researcher Dr Alex Dregan, said.
"We need to make people aware of the need to do some lifestyle changes because of the risk of cognitive decline," he said.
Researchers do not know how such a decline could affect people going about their daily life.
They are also unsure whether the early drop in brain function could lead to conditions such as dementia.
"Research has repeatedly linked smoking and high blood pressure to a greater risk of cognitive decline and dementia, and this study adds further weight to that evidence," Dr Simon Ridley, from Alzheimer's Research UK, said.
"We all know smoking, a high blood pressure, high cholesterol levels and a high BMI [Body Mass Index] is bad for our heart. This research adds to the huge amount of evidence that also suggests they can be bad for our head too," The Alzheimer's Society added.
"Eating a balanced diet, maintaining a healthy weight, exercising regularly, getting your blood pressure and cholesterol checked and not smoking can all make a difference," it said.
The study was published in the journal Age and Ageing.