You can do this from your host, as they often provide you with one for free if you're hosting with them, but I recommend purchasing it from someone other than your host, so you have the flexibility of easily changing hosts down the road (without having to deal with transferring your domain).
2. Purchase web hosting
A shared hosting account is great for starters, because they are inexpensive and very easy to maintain. You'll want to find a host that allows you to install content management systems like WordPress.
3. Install a content management system
There are a lot of options, but from what you've told us, WordPress seems like something you should consider. It provides everything you're looking for out of the box, but it has lots of customization options that let you take your website wherever you want to go in the future.
From 0 to 100%, I reckon that all of this shouldn't take you more than a few hours to setup. Once that's all done, you can start tinkering within the WordPress administration screen to figure out how to categorize things, allow users to register, comment on articles, etc. WordPress is pretty intuitive, so you may get everything setup in a day.