Dramatic footage on local television and social media showed one crumbled building that had fallen onto its side and another lurching dramatically after the quake.
Formosa TV reported that there were 150 trapped in one 16-storey building with those inside crying for help.
Officials said that 69 had been rescued but could not confirm how many were trapped inside.
At least 30 people have been rescued from another residential seven-storey block that collapsed, with no fatalities so far.
The shallow quake struck at a depth of 10 kilometres (six miles) around 2000 GMT Friday, according to the US Geological Survey, 39 kilometres northeast of Kaohsiung, the second-largest city on the island and an important port.
The quake was initially reported as having a magnitude of 6.7, but was downgraded to 6.4.
The Pacific Tsunami Warning Center said a destructive Pacific-wide tsunami was not expected.
Taiwan lies near the junction of two tectonic plates and is regularly hit by earthquakes.
A 7.6-magnitude quake struck the island in September 1999 and killed around 2,400 people.
