Are you deleting the tag from the left-hand Directory column that lists all tags or are you selecting the specific picture and then in the right-hand Info column right-clicking the tag and selecting Remove Tag? I think you may be doing the former rather than the latter.
I am adding tags to my photos in Photo Gallery. It was going great until I put a tag on a photo by mistake. I can't remove it! I can remove the tag name from the list but then it will remove it from every photo that it was added to. Please someone tell me I've overlooked something! Thanks