The blaze, which broke out in the middle of the night, left a further five people injured, police said, with the fire and disaster management agency saying four of them were in a serious condition.
Video footage aired on public broadcaster NHK showed firefighters surrounding the hospital in the city of Fukuoka, in the southwest, as smoke poured from the front entrance of the partially gutted building. Dark streaks of soot were smeared across the second floor windows.
All of those who died were between the ages of 70 and 89, police said.
The fire raged for around two hours, having started at 2:20am at the four-storey building, which opened as an orthopaedic hospital in 1970.
The hospital has 19 beds, with suites for orthopaedics, rheumatism and rehabilitation.
Hospital staff lived on the top floor, a local fire station official said.
"We did our best in fire fighting to save lives... But it was a difficult situation," a fire station official told a hastily arranged press conference.
"Patients on the second and third floors were exposed to a lot of smoke because fire doors that would have stemmed the flow had been left open," he said, referring to the first and second floors in British English.
"We will start an investigation" into the cause of the fire, the official said.
Another fire station official told AFP that the fire appeared to have begun somewhere on the ground floor, but he did not know the specific place or cause of the fire.
Local media reported the fire may have started at a treatment room which had a laser device and thermal therapy equipment that used a water boiler.
