James extended his lead as the competition's top scorer with a penalty 10 minutes from time to register his sixth goal in five games.
However, by then the damage had already been done for Colombia as goals either side of half-time from Thiago Silva and David Luiz booked Brazil a semi-final meeting with Germany.
"Unfortunately, the referee didn't help a lot," said the Monaco playmaker, who had been singled out for some rough treatment by the Brazilian players.
And Brazil were later to pay the price for the physical nature of the game as star striker Neymar was ruled out of the World Cup after suffering a fractured vertebrae when he clashed with Juan Zuniga two minutes from time.
"I think the referee influenced the game a lot, but that is how it is and we need to look forward.
"We are sad because we wanted to go further in the World Cup. That is what hurts most, but Brazil are a good team."
And the 22-year-old said Colombia could be proud having reached the last eight for the first time in their history.
"I want to thank the people of Colombia because they have always been there for us and had faith.
"We are sad but we also ought to be proud because we gave everything to try and continue.
