There are extensions for Joomla that allow you to integrate a forum. Here are a few: http://extensions.joomla.org/extensions/communication/forum
What tools do I really need?
You need a Web host that will support the language (PHP/ASP) and content management system (CMS) you wish to use, hence it's better to decide on what it is you want to accomplish first. You mentioned a forum and shopping cart. Both of them are available open source (free) as stand-alone products, but if you are looking for a more integrated solution, you may want to pick a CMS that has these things available. Most of the time you can visit a CMS' Web site to see what they have available.
If you didn't want to use a CMS, which is a separately installed system that you have full control over, you could use the site builder that is sometimes Web host specific. This is not recommended because if you ever needed to change hosts, you're Web site would be a hassle to transfer and could not be easily maintained.
The other solution is to use a software Web builder such as Dreamweaver, Expression Web, etc, which has a learning curve as well. The benefit of that is that you can work on your Web site without being connected to the Internet, however if you're online all the time anyway, a CMS would probably be the most efficient solution.
You also mentioned something about "cheap hosts don't necessarily work out too well". I find that $4 or $5 a month is pretty cheap and you can get a reliable and friendly Web host for that price.
And what is a php?
PHP is a server-side language. You will most likely need to be familiar with either PHP or ASP if you want to make heavy customizations to your CMS, Shopping Car or Forum, however most everything has a user-friendly GUI that let's you change the basics.