"Organised criminals and others are likely to be quick to exploit any perceived weaknesses or gaps in the enforcement regime," the report said.
"This, combined with the UK's potential loss of access to EU security, law enforcement and criminal justice tools, could create security weaknesses which the government would need to address urgently."
Brexit talks have stalled over a disagreement on the so-called Northern Irish backstop, an insurance policy to ensure there will be no return to a hard border on the island of Ireland if a future trading relationship is not agreed in time.
The NAO said planning for border operations in the event of a deal was less developed than that for no deal because of uncertainty over the future UK-EU relationship.