The 28-year-old "The Perfect Man" star was contacted by officers in Beverley Hills after they suspected an unnamed man with Duff's photographs, reported Femalefirst.
The whole incident started this month when the guy hired a limo driver and had a picture of Hilary's house but didn't know the full address.
However, the sources claimed that former "Lizzie McGuire" star's security team know who the man is and what he looks like as they have had issues with him in the past.