If you are using a smart phone there is a lot of discussion about how intrusive these are. Most folk SYNC their data, contacts and more with Apple, Google or another so the answer is no, these after the sync are now up in the cloud where no one can guarantee these are protected.
That said, there are now trillions of such numbers in those clouds so why do you worry about your caller numbers?
Also, if you call those numbers the carriers log that so again, the data has been put into the cloud and subject to leakage.
which application should I use to protect my caller numbers?