Mohammad Imran, a caretaker of a mosque in Karol Bagh and resident of Seelampur locality, was arrested when he was going to attend a Janta Darbar (public meeting) at the chief minister's residence on Monday, they said.
He is a maujjim at the Masjid Bawli Wali.
Last week, a man threw chilli powder at Kejriwal inside the Delhi Secretariat.
Imran had come to meet the chief minister at the Janta Darbar around 11:15 am, a senior police officer said.
He had come with 12 imams and maulvis to discuss the issue of increasing salary of staff working at the Delhi Waqf Board, he said.
During the search, a live cartridge of .32 bore was recovered from his purse by security staff deputed at Kejriwal's residence, the officer said.
Imran was handed over to the local police and accordingly, a case was registered at the Civil Lines police station under relevant sections of the Arms Act, he said.
During interrogation, Imran said he was a maujjin at Masjid Bawli Wali in Karol Bagh and two to three months ago, he found a cartridge in the mosque's donation box. He said he had planed to throw it in the Yamuna river but did not do so and kept it in his purse, the officer said.
Further investigation is underway.