There are other ways for spammers to get your email address without hacking you and without your going to a bad site.
First and foremost, *some* websites will sell your email address as a revenue stream.
So for example let's say you sign up for a forum that is free and helps you post messages regarding your favourite pokemon character. (Hey, I had to make something up that I wasn't likely to get sued over :D ). The website then sells off your email address to others. Very quickly your email address gets out there. Since spammers make actual real money, they can easily afford to buy email addresses. As already pointed out, they can use your email address in the send box. I can do that from any real email client quite easily. As already pointed out, the full internet headers would show that while the reply-to and sent from boxes might say one thing, the actual headers will show the real info.
Now yes, that doesn't mean they have access to your address book. Obviously accessing that would require more of a hacking approach.
One thing that comes to mind and troubles me is that there are sites like Google where Gmail and other things nag at you to upload/sync/etc your contacts from your email. Facebook, etc etc. The further outwards you go from internal Rogers and your home, the more likely that data will be intercepted. These sites are growing as people try to replace/supplant Gmail etc.
My approach to resolve or at least track some of these things is that for the price of a cheap fast food meal now you can buy your own domain and have at least email service. I thus have separate email addresses for every website I register with. When eventually I start getting some spam issues, I look at which email is involved. That typically narrows down the culprit quickly and also allows me to selectively stop using it. All the emails I use are automatically forwarded to my Rogers one, so I only ever use my Rogers email for accepting forwarded emails- never directly. I've caught quite a few websites this way.