The 22-year-old off-spinner had Corey Anderson leg-before, Bradley-John Watling caught behind and Doug Bracewell caught in the slips off successive balls at the Zahur Ahmed Chowdhury stadium.
Gazi's hat-trick, which gave him his sixth wicket in the innings, saw the Black Caps slump from 260-4 to 260-7 in their second knock after lunch on the fifth and final day.
The tourists declared the innings closed at 287-7, leaving Bangladesh a target of 256 runs in a minimum of 45 overs to win the Test and take the lead in the two-match series.
Gazi, who hit an unbeaten 101 on Saturday, joined Alok Kapali as only the second Bangladesh bowler to claim a Test hat-trick.
Kapali had achieved the feat against Pakistan in Peshawar in 2003 when he dismissed tailenders Shabbir Ahmed, Danish Kaneria and Umar Gul in three consecutive deliveries.
The draw loomed after New Zealand made 469 in their first innings and Bangladesh replied with 501 on a newly-laid pitch that provided no assistance to the bowlers.
Bangladesh have yet to win a Test against New Zealand, having lost eight and drawn one of their nine previous matches against the Black Caps.
The second Test will be played in Dhaka from October 21 to 25, followed by three one-day internationals and one Twenty20 match.
