He was 57.
Fahim was an ethnic Tajik who was an ally of Ahmad Shah Massoud, the charismatic Northern Alliance commander who was killed in an al-Qaida suicide bombing two days before the September 11, 2001, attacks.
President Hamid Karzai's office said Fahim died from an illness.
Karzai called Fahim a true patriot and said his death was "a huge loss for Afghanistan." He also expressed his condolences to the Afghan nation and Fahim's family.
Fahim served as a vice president and defence minister in Karzai's first administration and most recently was the first of two vice presidents.
Karzai's spokesman Aimal Faizi said the government has called for a three-day mourning period and flags would be lowered to half-staff.
Presidential candidate Hedayat Amin Arsala said Fahim's death "will affect the future political development," but he didn't elaborate.
