When Word or Excel open a file, they create one of the $ files you refer to. If you have a crash or reboot while these are open, they're left there and only get deleted when you manually delete them. This is just one of many reasons why you shouldn't shut down a computer any way but the normal way. Now the thumbs files are created in any folder where you have pictures and use the thumbnails or icon view of them. Neither of these have anything to do with Windows Live though - that's the category you posted in.
You can delete all these files without harm to your system, although if you have Word or Excel running, there will be a few that're in use and therefore can't be deleted.
`
Good luck.
For a while now when I am creating or working with existing Word files (I believe also Excel), duplicate files will appear in the same directory that have a squiggly line, a $ ( a system file) and the remainder of the file name. Also, I get 'Thumbs' files that appear out of nowhere. Don't know what these things are. I was afraid to delete them for a long time but now I just delete them despite warnings that they may hurt the system. I would like to stop this from happening as it is a nuisance. Anyone give me any advice on how to stop this from happening? Thanks

Chowhound
Comic Vine
GameFAQs
GameSpot
Giant Bomb
TechRepublic