I think the player uses the graphics that are currently setup in game.
What you could try before watching/recording the film is setting the grapphics and texture detail up in game, then let it rebuild the cache, then try launching the viewer. Im not sure that works, so try at your own risk

.
You can also swich anti-aliasing on while running the viewer, that smoothes the edges a lot.