Which athletes don't use equipment/clothing that enhances their performance?
Football pads/helmets may look like "safety" equipment, but in truth they allow the players to perform in a manner that would be impossible to sustain without the pads. What about skis, kayaks, tennis rackets, high-tech bicycles, etc? Don't they contribute to an enhanced performance?
As for the swimsuit, as a former competitive swimmer I'll tell you that you missed the most obvious performance enhancing piece of equipment the swimmers use. Without goggles their times would suffer hugely. The fastest part of the race is the start, followed by the turns. Clear vision is massively important to properly approach and time your flip turn. A few inches too far from or close to the wall and your turn will suffer. A goggle malfunction can easily cost you the race.
Wearing the low-drag suit isn't any more "artificial" than it would be to wear a big draggy pair of trunks. If anything, reducing drag to a level more consistant to skin is less "gamey" than applying the artificial drag of the trunks.
As far as the evolution of sports goes, athletes have always and will always be looking for an edge. Heck, even our modern knowledge of diet, mental and physical conditioning, and the easy, rapid transmittal of information gives modern athletes an "unfair" advantage over our "ancestors".