Windows 98, by design, is limited to 512 Mb RAM (although some report they have it working with 1 Gb).
The solution:
Remove enough memory sticks to get Windows 98 running. Then Start>Run msconfig, and in General>Advanced ... limit the memory to 512 Mb.
Then reinstall the full 3 Gb and see what happens.
A recent discussion in this forum on the same subject: http://reviews.cnet.com/5208-6618-0.html?forumID=6&threadID=58143&messageID=691124
By the way, what do you do with the machine to need 3 Gb RAM in Windows XP?
Kees
Hello, I am running a dual boot system, with Win98 on one hard drive, and XP on another,the issue I am having has to do with the amount of memory I have installed on my mobo.With XP, I am able to run 3 Gig,with no problems, yet if I try to boot up Win98 with more than 1 gig installed,I get an error message saying that there is not enough memory to initialize Windows, is there a software problem?I have had this happen on more than 1 computer, so know it is some sort of software issue, and not a hardware problem, anybody know why this is?

Chowhound
Comic Vine
GameFAQs
GameSpot
Giant Bomb
TechRepublic