If you are using USB stick then I may have similar problem.  From time to time, my stick will get lazy on me and the inputs will be delayed for like 5 seconds.  I unplug the stick, wait a few seconds and plug it back in and it re-initializes and works just fine afterwards. 
By the way, I am using the Saitek X-36.