I have heard of a few victims, locally who actually claim to have had fraudulent charges after a "yes" telephone encounter.  But I agree that the "yes" is not necessary to be scammed.  I found a fraudulent charge on a statement, once, from an online Florida Co.  I had never spoken to them, written to them  (and at the time) had never used a computer.  There is also the "Yes" sales closing trick .

Caller did not realized that he was speaking to my answering machine! Asked if I could hear him, however my machine did not answer and subject hung up!

Called yesterday, didn't answer. Added message here re call. Tonight got another call, didn't answer. ID said Switzerland. Lol Same number just spaced differently. 41 51 428 86 7. Guess the scammers are getting creative.

Same...

Phone rang just 2 times. Did not answer and no message left. Caller ID showed INVALID NUMBER. Another search showed the caller as "Non Such Photography" which appeared to be a porno-photo provider.