Asbestos is mostly hazardous if you breathe it in. As in fine particles. Tearing/cutting/removing old insulation will spread this stuff in the air, no question about it, and you'll end up breathing it if you don't have the right kind of protective gear. As for bringing it home, if you brush off your clothes before going home and immediately put them in the wash and shower so you're not spreading the stuff around the house, I'd bet you'd be ok at home.
But without the right equipment, you're breathing that stuff any time the original form of the stuff is disturbed. As originally installed, the asbestos is fixed in place by other material. When it gets old, it often separates and comes apart. If you cut it, it comes apart. Any time it comes apart, you're probably sending the fibers into the air and breathing them in. Bad stuff.
That said, my Dad worked with asbestos as a roofer and working for dupont making freeway bumps, and he didn't get cancer. Some people are much more naturally resistant to cancer than others and he got lucky.