Mathews returned home after losing the Test series 2-0, the five match ODI series 3-0 (with one tie and one no result) and also conceding the only T20 in chilly English conditions.
"Fitness was a big issue, as the captain I am disappointed with our battling, bowling and fielding," Mathews told reporters in Colombo. "We have to improve in all departments."
However, he thanked loyal local fans for not mocking them for their dismal performance.
Asked about preparations for the upcoming home series against Australia, Mathews said he wanted to address fitness issues plaguing the young team which saw five bowlers suffer injuries during the England tour. Mathews himself was injured.
He could not say how long the team would require to fully recover from the void created after the retirement of legends including Kumar Sangakkara and Mahela Jayawardene.
"But we are doing the right things at the moment. We are not winning games, but we are seeing improvements day by day."
His comments come with the Australian team set to arrive Saturday in Sri Lanka, where they are due to play three Tests, five ODI matches and two T20 games.
Mathews said the Australian team was "tough", but he hoped the home side advantage in sultry conditions would help local players.
"It (the series) is in our home conditions, but we have to prepare well and prepare for a tough challenge, and to overcome the tough Australia side we have to play our best cricket," he said.
