Are you talking about the cache being rebuilt everytime? If any one of the files in the cache got marked read-only, then it will retry rebuilding every time you start the game.
Just kill the 'cache' folder and allow it to be rebuilt, without interruption and that should take care of that.
If you are talking about the loading that is done prior to going into an arena, then uncheck the "Preload textures into system RAM" option in the "Video Settings". Not sure why you would want to do that as it could cause more stutters to happen in the game.