No, not at all. I am saying do not assume anything about the state of your computer. Check it from top to bottom.
If the CPU is not being clocked back, due to heat, nor the video card slowing down due to heat, then something must be taking the CPU away from the game, unless the slow downs are simply due to the hardware not keeping up within the current settings of the game.
These types of things are always problematic to debug.