According to what I read, it is one of those "80/20" kinds of things... you gain maybe 20% benefit for around 80% more work.
From personal experience helping my Dad "grind" a reflecting mirror for an astronomical telescope, working by hand with fine grit abrasives is very, very, VERY boring!
If a heatsink has a nice mirror-smooth finish, it's good enough for me.
YMMV