The "core" package requires you to use memory cards to save games, and anything you may download from Xbox Live. The premium package includes a 20GB HDD to store game saves, and other content from Xbox Live. It also comes with wireless controllers as opposed to wired. The new Elite package includes a 120GB HDD, but is otherwise identical to the premium package save for an HDMI port.
Aside from these minor differences, they're exactly the same. Same CPU, same quantity of RAM, all that.
What are the major differences in the Xbox 360 core system and the Xbox 360 with the 20GB hard drive

Chowhound
Comic Vine
GameFAQs
GameSpot
Giant Bomb
TechRepublic