Well, you want the ability to downvote (as you alluded to), then be prepared for downvoters. Ignore them. Besides, deleting saves doesn't change your vote average, I think.
@Lynxrufus(View Post) The voting and score system is mainly there to provide you with feedback on your ability and quality of work... If you know someone is just being a troll, then psychologically, it should make no impact on a mature person... are you a mature person? The moderators will likely dismiss it unless there is an inappropriate comment or tag (there does not seem to be inappropriate language on the tags/comments you describe) Please refrain from using inappropriate language on the forum.
...It's just a game. And yeah, it's upsetting but, best you can do is ask simon if any of the same people are downvoting your stuff, or check the names of who set the tags.