Update (09/30/2014): Google has added a way to manually mute individual tabs in Chrome to the Dev and Canary builds. For more information, check out this post.
Back in January, Google Chrome added a feature that shows which tab is making noise by displaying a speaker icon on it. This is really helpful when you have many tabs open and suddenly a video starts playing on one of them. Unfortunately, the browser doesn't tell you which element on the page is making noise, so you may still have to look for it among the ads and text.
Instead of switching tabs and manually turning off the sound, Ghacks.net recommends using a Chrome extension called Mute Inactive Tabs. This extension takes the guesswork out of silencing tabs and does it for you automatically.
Step 2: Click Add to Chrome. When you see the pop-up confirming the install, click Add.
Step 3: To test, open a YouTube video, and then open another tab next to it. The YouTube video should pause while that tab is inactive.
I found some YouTube videos went back to the beginning with Mute Inactive Tabs turned on, but this shouldn't be too much of a problem if you're working in another tab instead of watching the video.
Although the extension offers a convenient toggle button, it might be best to open streaming music services in another window. That way you can experience muted tabs in the main window without interrupting your tunes in the other.