The differences are huge, but small at the same time...
1) USB drive:
This can actually be said to be anything you can plug into your USB port that shows up as a drive. If someone says they have a USB drive, it's like saying they have a car, instead of saying it's a Ford or a Suzuki.
2) Flash drive:
[some people refer to these as Memory Sticks]
Typically, these drives are non-volatile memory, meaning they won't erase by removing them from a power source. These type of "drives" include (but are not limited to) SD/xD cards, MMC cards, etc. These can be stored away for years without losing the data on them. They are mostly used in cameras, etc., and write the memory "in a flash" without using any energy to actually keep the device going. These have a limited number of writes to them before the data becomes unreadable. (Don't worry... it's in the thousands of writes... you'd have to use your camera for years to write to it so many times as to start seeing the data corrupted. You'll probably get a new camera or larger flash card before that happens regardless.)
4) Compact Drives:
These, unlike the flash drives, do use energy. Another person mentioned these can be damaged by removing them without disconnecting them properly. These are usually about the size... wait for it... of your thumb. These are good for carrying around more data than the flash drives, can take more writes than a flash drive, but will (eventually... years...) lose the data due to loss of power. It's not actually removing the drive from the computer that damages them in the way described above, but data corruption because the computer hadn't finished writing the directory information. It's always good to disconnect the drives correctly... but power outages, etc. can cause the same problem as removing the drive without "safely removing" the device first. Chances are you'll be fine... but there are lottery winners, people who get by lightning, etc. These do have a limited number of writes as the memory chips they use will wear out, but like the flash drives, you'll probably replace them long before that happens.
5) External hard drives:
This is the only actual DRIVE type... These are slowly being replaced by the flash drives and thumb drives as they grow larger, but will probably never fully be replaced. These are external enclosures or bays that allow you to hook an actual hard drive, be it IDE/PATA or SATA, to the computer and write to it. This is your best bet for long term use as a hard drive, and not just a courier for data, such as copying files from home to take to the office. They're bulky compared to the others, but more reliable long term, and more likely to allow you to recover data in the event of a problem, like the one that started this thread... the deletion of a folder.
I'm probably missing options, but these are the most common. Just remember to keep a backup of anything you put to these, even if it's to another of the same type of device. After all... you don't want to lose those important files because of a wayward delete. (You also want to be able to say "Oh - I needed that" two weeks after, and be able to get it back.)
Hopefully that clarifies the differences a bit.