Nothing seems wrong to me. RDC is not instant, never has been.
I use a 2003 Server set up on an older PC. When I connect to this Server using Remote Desktop, I have a lag between the time I click and when the system responds. It's not long, but just enough to be irritating. (Server 2003 with 4 Gig of RAM)
I used to have this server set up on a KVM switch, but the KVM switch was PS/2 Keyboard and mouse which my Windows 10 PC cannot use. I'd really like to use Remote Desktop on this if anyone can figure out what's going on.