Olympus is very good. These are just a few things to think about. Olympus and several other companies like Leica, Sigma, etc decide to commit to the new 4/3 system, which is new and different from the 35 mm format. It has a smaller sensor than other D-SLRs, with a 2x crop factor. And they will not likely make full frame 35 mm format camera and lenses. Yes, even the top-of-the-line E3 has a smaller sensor than the entry level Canon XTi or Nikon D40. But Olympus' image quality is still very good, though its low light performance may not be as good as the Canon or Nikon counterparts.
The main disadvantages of Olympus are that it limits you to smaller sensors with crop factors, smaller line of accessories than Canon and Nikon, and fewer choice of lenses (though you can use other 4/3 manufacturer's lenses since the 4/3 mount is universal). If you shoot action in low light and needs a fast lens, the Olympus/Zuiko fast lenses can be expensive (with high quality and you do get what you pay for) and there are fewer alternatives. Although Olympus pioneers live LCD view and dust removal systems in D-SLR, these are found in many new D-SLRs nowadays.
If you do mostly still photography, portraits, landscapes, then Olympus is as good as other competitors. If you do low light actions, then it can get expensive and have fewer alternatives to choose from, compared to Canon and Nikon. There is also not a lot of room for you to upgrade. Its top of the line E3 boasts fastest autofocus in advertisement but it is not the fastest D-SLR when its top continuous drive is only at 5fps. The mid-range Canon 40D is even faster than Olympus' top-of-the line E3. Don't get me wrong, the Olympus line is very good and you won't be disappointed with their quality. The above may not apply to most people, just some points to think about when you choose a D-SLR system.