Clinton, UN leader Ban Ki-moon and Andrew Mlangeni, who was a prisoner with Mandela, honoured the legendary freedom fighter in a special ceremony at the UN headquarters as Mandela spent his 95th birthday in hospital.
Clinton said Mandela ended his 27 years in an apartheid jail "a greater man than he went in" but also told how the ordeal had left its mark on the legendary figure.
"Every day was a struggle, I could see it in his eyes even after he became president. Some old demon would rise up and somebody would do or say something stupid, but he fought it every day," Clinton said at the UN General Assembly.
"Tell me the truth: when you were walking down the road that last time didn't you hate them?" Clinton said he questioned Mandela.
"He said briefly: 'I did. I am old enough to tell the truth.' He said: 'I felt hatred and fear but I said to myself, if you hate them when you get in that car you will still be their prisoner. I wanted to be free and so I let it go.'
"He looked at me and smiled and said: 'Neither should you'."
Clinton said he was also one of the rare people alive who had seen Mandela "in a less saintly role" when the two were arguing as presidents of the United States and South Africa.
"On rare occasions he would even get angry with me when he disagreed with our positions. But what I know is his heart was so big and his humanity so great we often had trouble keeping our official roles apart from our personal friendship," Clinton said.
"We are united in concern. We are also joined in admiration for a towering figure in the worldwide fight for equality and justice," the UN secretary general told the ceremony.
