Since the mouse button didn't work, hold down the option key at boot. After a few seconds you should see a mouse cursor, maybe a button or two, around the middle of the screen, and then an icon for your HDD should show up. Give it a little bit longer and you should see an icon for the install CD show up. You can click on the CD icon, and then hit the eject button: The one on screen or on the keyboard, either should work.
If all else fails, the paperclip trick already mentioned should work, but if even that fails, then you're going to have to take the unit somewhere to be disassembled to extract that disc. It will have to be an AASP, an Apple Store will just tell you it's too old. This is also one of Apple's least DIY friendly units because of the narrow stem for the display. That can be bent or broken very easily, plus it means the weight distribution of the unit is very lopsided when you try and lay it down, so you can't just have it hang off the edge of a table or something. Apple sold a special positioning block for these units, and unfortunately you probably won't find any places that have those around anymore since the iMac G4's are a nearly extinct species.