Hm. I think, with every down-vote you should have to give a reason before the vote is submitted. If the reason isn't good, you click the 'report vote' button and ta-da, the vote gets removed. May cause more work for moderators if they have to troll through more reports, but it's the only way to stop the vote-trolls.
!!! Why would you assume that!? That is for people who think they're saves are not good enoiugh to stay or are vote-hungry and want publicity. I know most users I know on the forum would never do that.
Not to be mean or anything, but stop complaining about down votes. I get them all the time. Sometimes I understand why, and sometimes i don't. You should just live with it.