Modular will come with price. It will be bigger, heavier, less reliable and more expencive because of design complexity.
Do you really need it? Are you going to upgrade it?
For example. Probably you have laptop. If you really need, you can upgrade many components there much less expensive then in modular phone. Did you do that? Do you need upgrade for your phone?
It covers most of the reasons we know of and most of is simply no one sold or bought them.
-> That said how about the Fair Phone?
Read about it at http://www.techradar.com/reviews/phones/mobile-phones/fairphone-2-1315989/review
Almost modular, but too pricey in my view.