"when someone would watch a video on their phone or I would play a game online my download speed would drop to around 5 Mbps.."

Sounds right. Either activity uses up the bandwidth and on WiFi this is half duplex so gaming which uses both up and down over the same link, well, nothing sounds wrong here.