Gunmen led by a Salafist local chief known as Abu al-Abbas blew up the mosque of Sheikh Abdulhadi al-Sudi on Friday night, the official told AFP, confirming media reports of the attack.
Yemen's commission for antiquities and museums condemned the destruction of the site that is considered the most famous in Taez.
It said the mosque's white dome was "one of the biggest domes in Yemen and one of the most beautiful religious sites in old Taez".
Images of the site before destruction showed a white square-shaped, single-storey structure topped by a large central dome circled by smaller ones.
Sufism is a mystical movement of Islam that is frowned upon by the ultraconservative Salafist brand of Islam.
Residents have complained of the growing influence of radical Salafists, who have been imposing curbs on mixing between men and women.
While Al-Qaeda and Islamic State group jihadists have been under attack by both government and rebel forces as well as US drones, Salafists operate under the banner of pro-government militias fighting the Iran-backed Shiite rebels.