This isn't exactly true any more (it was 10 years ago in MS-DOS times).
Assuming you're talking about Windows 95 or higher (98, ME, 2000, XP) and delete the file from Windows Explorer (not with MS-DOS del/erase command) a file by default is MOVED to the recycle bin. And from the recycle bin you can 'undelete' it (move it back to where it came from), until you either empty the recycle bin or delete any individual file from it. This will really delete it and free the disc space.
You need special file recovery software to undelete it then, and that only works if the disc space happens to be not reused for another file.
There are two ways you can influence this default behavior:
1. Press shift-del to delete the file in Windows Explorer. This will bypass the recycle bin and delete the file and free the disc space.
2. Use the properties of the recycle bin (right click on the icon on the desktop, then choose properties) and set the "don't use the recycle bin, delete all files immediately" checkbox. That works for ALL files, until you uncheck it.
And deleting from a command prompt circumvents the recycle bin also. It's strictly an Explorer feature.
I would recommend to use the default settings, so generally files go to the recycle bin (you can set the maximum disc space it uses). Only when you're VERY sure you don't need the file, use shift-del. But you may prefer it otherwise.
There are Explorer replacements that offer the 'immediate delete' from the menu bar (Powerdesk does), or maybe as an right-click choice or as a setting, but personally I'm happy with the shortcut keys del and shift-del.
Hope this helps.
Kees