With dial up you would find downloading quite frustrating. And most legal sources online are poor value, in my view.

Assuming your computer has a CD drive you can rip your existing CD collection to MP3s. I think you can do it using later versions of Windows Media Player, though I'm sure there are better purpose-built programs available as shareware or freeware.

MP3 players usually connect via the USB port, and most (iPod excepted) allow you to transfer files by drag and drop. Most have bult-in rechargeable batteries which (particularly in the case of the iPod) can be expensive to replace when they finally fail.

If you prefer replaceable batteries I'd recommend you start with one of the really cheap stick-type players of at least 1gig size -- they're pretty horrid to look at and to use but are robust and usually support albums as folders -- which saves you tagging each MP3 file.