I don't sure about XP, however I do know that in Windows 98, you couldn't delete them, but it only held a limited number of desktop images, and then started rewriting over top of the ones already saved. Some might use that option to overwrite images that might be... no longer wanted lol.
My father mistakenly added a long list of .jpg's to his desktop backgrounds. How can these be deleted? (Not "themes" but backgrounds). I spent two hours reading the articles provided by Google and Microsoft. There is plenty on how to change backgrounds, but I cannot find anything on how to delete, and a simple rt-click won't do it. Instructions on how to remove these files will be greatly appreciated. Thanks.
XP Home

Chowhound
Comic Vine
GameFAQs
GameSpot
Giant Bomb
TechRepublic