A little from column A and a little from column B. In particular it sounds like you're using an all wireless setup, which is not a good idea for more than 1-2 people. While you do have more and more MIMO routers coming out lately, there's a great deal of older single antenna (i.e. half-duplex) routers out there which will really eat into your bandwidth. Essentially every time you connect another device, the amount of bandwidth gets split an equal number of ways. Two devices: 50/50 split. Three devices: 33/33/33. Four: 25/25/25/25.
The issue isn't your DSL connection, the issue is that you're likely using an older el chepo router with an all wireless setup, and/or some of your mobile devices are sucking up a large amount of bandwidth running some kind of background sync operation. Some cell phones will only update installed apps when connected to wifi, because as much as the rates DSL and cable companies charge for going over bandwidth caps are highway robbery, the amount cellular carriers charge is like handing over your first born, and a couple of limbs... As a down payment. Apple saw fit to hard code an artificial limitation into iOS, so that any app of sufficient size will not download unless you're on wifi.
The quick and easy solution, is to make sure you're always connected via an ethernet cable. That will help significantly. Even better is connecting everything possible via ethernet cable. Leave your tablets and cell phones to connect to wireless alone. Basically, you need to give yourself a crash course in some of the basics of networking.
We have DSL (ok, first problem, yes I know) at my house and several computers. I use my computer the most and am sometimes the only internet user. When someone uses another computer I really notice no slowdown with the internet. However, when my mom uses her IPad or when I have my smartphone connected to the wireless router there seems to be a rather clear slowdown on my computer.
The slowdown isn't necessarily constant, but it feels like sometimes I will have the internet cutoff for short periods at a time - to the extent where some mmorpg clients I have open will shut.
At our other house in the mountains, where we have DSL also but its much slower than in the city, I would sometimes lose internet completely and would have to reset the connection. I couldn't figure out why and then realized my smartphone was connected to the wireless - when I turned off its WiFi the problem stopped occurring.
All this made me wonder if Cloud devices are perhaps bandwidth hogs or for some other reason can completely take over a slow connection for short periods (10 seconds, perhaps) of time. Just wondering if anyone else has any information on this or has experienced this themselves.