Need to hold down the shift key as well to de-select. Thanks.