I'll leave the safety on until the rifle comes to my shoulder. It is the way I was taught and is the way I train/practice. While fine motor skills do deteriorate under stress training and practice, the development of muscle memory, can overcome that. You will do under stress what you have developed as muscle memory (which is why it is very important to train like you fight, fight like you train)
The safety on an M4/AR15 is much easier to operate than the retention devices on the l holster I carried on duty, I never had any issues drawing the weapon and did it under much greater stress than this encounter the other evening….I practice with the M4 nearly every day so I'm not concerned about being able to put it in action if needed.
I understand this type of training. A good friend of mine was a small arms instructor for the Vermont State Police. Retired, he now works for a private firm, doing the same type of training. The police have certain liabilities that civilians don't. More than the military as well. Thus, they train a bit differently than the military, with greater emphasis on caution. I was trained that when I expect a confrontation, safety off, finger alongside the trigger. One need only to bend the finger. Why safety off? Because, a potential adversary just 6 feet distant can cover that distance in the time it takes to operate the safety and shoot. Less than a second. Muscle memory is great, but time is the most uncontrolled factor. You go through a door and the bad guy is four feet away... Time becomes THE major factor.
Nonetheless, you're better off doing what you've been trained for. Besides, circumstances will always vary and no one can tell you what to do if not actually in your situation.