never leaves a message, upon recalling the number get voice asking if I want to remove the number

Has called my number 10 times in the last 2 weeks. I just called it back and an older black sounding man says he dialed wrong number, that it is similar to his friend's and says my name. I asked how he knew it, he said it came up and said "it is your name right?". Then I looked online and saw this post.  I hope I didn't just screw up by answering his question. I said "yea" but not "yes". I called him back and told him I was reporting his number... now what do I do?