How do you remove a "thumb down" or a "thumb up" that you mistakenly gave an answer ?

You can't, once done it is saved for eternity (or at least as long as Yahoo saves it). In the overall scheme of life is it really that big a deal though?