The gunmen on a motorbike opened fire at Colonel Abdullah Shamba, killing him immediately before driving off, the official said.
Shamba headed a local anti-Al-Qaeda militia in Abyan, the source added.
Late yesterday, Saudi-led coalition jets attacked Al-Qaeda positions in Abyan's provincial capital of Zinjibar and in the nearby town of Jaar, military sources said.
Government forces backed by the Arab coalition began an all-out offensive in March against jihadists in south Yemen, recapturing main cities they had held.
But they later retreated from Zinjibar after Al-Qaeda militants struck back.
They are often accused by the authorities of carrying out deadly attacks targeting government officials.
The Arab coalition which backs the Yemeni government has also turned its sights on the jihadists, and the United States has pressed its drone war against them.
Washington considers the Yemen-based Al-Qaeda in the Arabian Peninsula, or AQAP, to be the extremist network's deadliest franchise.
