If the date is the Date Taken, each file must be opened, read and decoded for the date taken. 500 images at 30 seconds so using that as a baseline to do 8 times more files should hover around 4 minutes.
To speed this up you can look at faster PCs and solid state drives.
Here I took it another solution. I did the sort and then make folders to hold smaller groups by date.
I have a library folder with 4,000 images. The folder is set to view in date order. When I open it in File Manager a green bar crawls across for up to two minutes before the folder is displayed in the correct order. This doesn't always happen but when a third party program wants to access the folder, it always happens.
I have another similar folder but with under 500 images and the same happens, although it resolves faster at 30 seconds.
Each folder is set to "general" and detail view so no thumbnails are needed.
My system is extremely fast and is used for image processing and video editing. I have experienced this problem with many versions of Windows before.
Are there any settings I need to alter to stop this? It is extraordinary that this happens in this day and age.