The decision to ask Zuma to step down or face being stripped off his office was taken at a specially convened emergency meeting, late on Monday evening, according to The Guardian.
A meeting of the ANC's national executive committee (NEC) was called after it became clear that nearly five days of talks between Zuma and the deputy president, Cyril Ramaphosa, who took over the leadership of ANC in December, had failed.
His tumultuous nine years in power have been marred by economic decline and multiple charges of corruption.
However, the 75-year-old retains significant support inside the party and at a local level in many parts.
The former ANC security chief is also facing a no-confidence motion in the parliament set for February 22.