I don't expect it would change if you upgraded one of the machines to, say, Windows 7 or Linux. Then it isn't a XP issue.
It seems the router loses the connection with whatever it's connected to on the Internet side. That could be a separate modem (unless it's a modem-router combination) or it could be the ISP. In both cases, it's an issue you probably can't solve yourself.
Kees
Hi
I have 3 XP pro machines connected on an ADSL router.
Everything worked fine but in last time the connection on the router suddenly drops and there is no internet.
Also, when I connect an additional comuter to the network the same happens, the router has no internet connection.
There is a small lamp on the router indicating "Internet", this lamp is turned off when there is no connection.
Any ideas what is here the problem?
Regards
Mirza

Chowhound
Comic Vine
GameFAQs
GameSpot
Giant Bomb
TechRepublic