I haven't had an opportunity to use an SD card with my Sansa, but other reviews, and posts show conflicting data on this subject.
Some claim exactly the same problem as you describe, while others say start up is less than 10 seconds.
Most say the card eats bateries.
It would be an additional power drain for sure.
Are all 2 gig SD cards hi speed? Could that be the issue?
The Sandisk manual implies that the Sansa ignores directories, so I wouln't think that would be the issue.
Sorry I can't be more help.
I have a Sansa with a 2 GB SD card and it seems to take forever to start up, plus it seems to use up a lot more battery with the card in.
I tend to keep my music in directories (by artist and album). I know this isn't necessary for the Sansa but it makes it easier for me to add and remove albums.
Do you think I would see better performance by putting all the songs in the root directory? Would that make it easier for the Sansa to check for changes?
Thanks in advance for the help.

Chowhound
Comic Vine
GameFAQs
GameSpot
Giant Bomb
TechRepublic