Personally I think you are being very wise.
Like Internet Explorer with it's Add-ons, anyone can create Gadgets for Windows 7 and unless you are sure that the developer is genuine, I believe you are right to be cautious.
That is not to say that all 3rd party gadgets are suspect of course, and there are some very good ones out there. But I would make sure that they are 'recommended' by some established resource, and that reviews are favorable.
I installed clock and calendar gadgets, both of which came with my Windows 7. I don't allow third-party programs on Facebook and am wondering if I should allow these third-party gadgets on my computer? Am I being super paranoid?
Windows 7 64-bit