I presume the window closing is open on a folder on the drive being ejected? If so, that is intended behaviour. If it's closing any window (say you have a Finder window open to your Documents, you eject a drive and that window closes), then that's not right. If it's the latter, you can try clearing Finder preferences (Home->Library/Preferences/com.apple.finder.plist) and see if that makes any difference.
When I click on the eject symbol in a finder window or highlight an item and press the eject key it will do the eject process and also close the finder window. Running and 2006 20" i-Mac with 10.5.5. It's kind of annoying and I don't know when it first started but somewhere after upgrading to 10.5. Any ideas?