New router won't connect to the Internet

Problem: Cannot log onto the Internet with new router. My cable modem connects to the Internet just fine without the router. I think I am missing something in the set-up. I want to make sure before I start thinking that I've got a defective router. I'm in college, and I sure would appreciate some help.

Details:
Router: Linksys BEFSX41 Broadband Firewall Router (brand new)
ISP: Comcast (Cable Internet), DHCP dynamic IP address
Cable Modem: RCA DCM105, (works great without the router.)
Operating System:
Computer #1: Windows XP Professional
Computer #2: Windows XP Media Center Edition (brand new PC)
Network Hardware:
PC#1: SMC Compact USB to Ethernet Converter (works great without the router.)
PC#2: 1394 Net Adapter, Intel PRO/100 VE Network Connection

I turned off my ZoneAlarm software Firewall. (But I did not turn off the firewall in XP. Could that be the problem?)

The router's documentation (70+ pages on a blurred PDF) says that I need to get the set-up info from my ISP. I tried that, but my ISP rep brushed me off after I declined to buy their network set-up services; the rep only told me that it was a DCHP and that I needed to have it Configured Automatically; she then flashed a link to a forum and ''poof'', both vanished. Someone on the ISP chat forum told me that I was to leave the router Set-up domain host and domain name fields blank. Another told me that I had to clone the MAC address. I tried that. No Internet.

I've tried power cycling the modem. I've tried powering up the modem first--wait for full boot, then power up the router--wait for it to initialize, then power up the PC. No Internet. I even ran a cmd > ipconfig/all and then manually entered the IP address as a Static IP Configuration just to see if that would work. No Internet. But, I can connect without the router.

I would appreciate any help. Thanks. ~GZ25