I haven't had an opportunity to use an SD card with my Sansa, but other reviews, and posts show conflicting data on this subject.
Some claim exactly the same problem as you describe, while others say start up is less than 10 seconds.
Most say the card eats bateries.
It would be an additional power drain for sure.

Are all 2 gig SD cards hi speed? Could that be the issue?

The Sandisk manual implies that the Sansa ignores directories, so I wouln't think that would be the issue.
Sorry I can't be more help.