The usual is the server is directly on the router via Ethernet for best speed.
Your setup would have speed issues from the onset since WiFi airspace is shared. You chewed up some of your airspace with the WiFi link from the router to the server plus a second WiFi on the server. Why you did that is unclear what 99.99% of the servers I've seen are on the router and don't need their own WiFi. They are on the LAN so they are accessible via WiFi.
I'm thinking of setting up a home server/network and did this diagram for visual reference. My question is, do I plug a WiFi network adapter into a switch connected by cable to the ethernet port, or do I do it the other way round; that is connect the adapter to the ethernet port and then the switch to the adapter? Thanks if anyone can advise me here