The 24-year-old singer revealed on the E!'s 'Fashion Police' that he dumped the 'Love Story' hitmaker, followed by few others, reported Ace Showbiz.
When fashion critic Joan Rivers asked him if he dumped Taylor Swift over a phone call.
"I think it's me and a few others, but yeah," Joe joked, referring to Swift's other famous exes.
It is said that the country singer wrote three songs about her relationship to boy band singer and when questioned about it, Joe's brother Kevin quipped, "We still haven't seen the royalty check. It's weird."
Swift has also dated Taylor Lautner, Jake Gyllenhaal and singer Harry Styles among others.