No, there should not be plastic deformation at Limit Load.
I can tell you that from first hand experience.
The factor of safety is a completely arbitrary number. It is a factor of ignorance, to account for the difference between real life and the engineer's mathematical model and laboratory experiments. You figure out what load you need to carry safely, call that the limit load, then multiply by your factor of safety, call that the ultimate load, and design the part so that it will not fail (deform plastically or fracture) at the ultimate load.
Being loaded "for a period of time" (fatigue strength) is a different discussion altogether. Steels have a fatigue limit, a stress level below which they will never fail, regardless of the number of stress cycles. Aluminum alloys have no fatigue limit. At even the lowest levels of stress, they will eventually fail after experiencing enough stress cycles. Most critical airframe components on WWII aircraft were aluminum, so the airframes had a finite life, regardless of loading.