26 total posts
Since the OS is corrupted.
Why not reload the OS and fix it?
Boot up taking 3 - 20 minutes
We are not sure if it is OS or Hardware related.
We have deployed about 350 of these systems and the problem seems to be spreading.
This sounds like the case of ...
Mark Russ... has troubleshot this often. Here's the one that was close to the symptom.
http://blogs.technet.com/b/markrussinovich/archive/2010/01/13/3305263.aspx for example.
Now that you have read a few of Mark's troubleshooting method you are ready to begin your own investigation.
But the one repeating thing I have encountered are IT departments that field an image and then dig in their heels in fixing that image or creating a new one. I can't explain human nature here but it is interesting.
Re: mup.sys hang
http://www.aitechsolutions.net/mupdotsysXPhang.html (just the first google hit) leaves me with the impression that it's most often a hardware issue. But that's so very unlikely if it happens to so many machines at the same time, that it must be software. Your experiment of cloning the disk leads to the same conclusion, of course.
Assuming these desktops are connected to a server to log in into a domain, seeing what happens if you disconnect from the LAN and boot (and then login as a local administrator) might be an interesting experiment. But it can hardly be expected it leads to the solution.
Spreading might mean it's a malware infection. A rootkit, a worm, that spreads via the LAN; an infected page on your intranet, a mail attachment that is forwarded and opened. Although I assume you have a good companywide antivirus active, it wouldn't harm to check such a machine (and all servers) with 2 different cd-based antivirus disks.
Maybe a bad (Windows) update that is delivered from a central location.
And after all that research, a clean install of the OS probably is indicated. If that helps, the question is for how long. If the issue reappears, it's likely to come from another PC or your servers. Then it might be time for a very big cleaning.
The other consideration.
Let's say this is some corporate install. All the machines may be aging at the same rate so if all things are equal, a power supply or other part should fail in one large group over a short period of time. This sort of failure is well known to most IT staffers but those that are new to this failure might get lost as they never saw it before.
Usually the cranky grey hairs will help on this one.
So they should add a few new machines to the mix and see if it spreads to them also.
Boot up taking 3 - 20 minutes
UPDATE - We are now getting reports from our other 12 hospitals that have rolled out these Dell Computers the same time we did that are having the same problem. The other hospitals use their own image. This rules out an image problem.
Sadly it does not.
As time passes some bit of code that uses the date or some antivirus update could change in such a way that a new image needs to be made.
Be sure that we don't completely rule out any option. For example last year I encountered an image with some Norton item and they had not removed it because it still worked as they changed to McAfee. While it was working at the time later McAfee changed in some auto update and problems began.
Eventually their IT staff caved and made a new image.
How did we find it? With the help of the methods from Mark R.
We have found the problem - I hope this helps someone else
The issue begins with the installation of Intel_Unified-AMT-7-Manageme_A00_R292087.exe or the newer version Intel_Unified-AMT-7-Manageme_A01_R298379.exe. These are standard driver installs from the Dell (and Intel) websites for the Chipset that is in these Dell 790's.
This executable installs the proper drivers for the PCI Serial Device and the PCI Simple Communications Controller devices. The problem is that this executable also installs 2 applets that run as Services on the machines. These are the LMS (Local Manageability Service) and UNS (User Notification Service) services. It is the UNS service that is accessing and writing to the Default registry hive file found C:\windows\System32\Config\DEFAULT file. This is causing it to grow in size over time in seeming a random manner. By disabling these 2 services the file growth will stop.
We ran an SCCM query checking this file size on all of our PC's in house. We found that the file size in all of the new Dell PC's (and a couple other PC's) have grown to various sizes. During bootup the OS will parse through this file and load it into the registry. The larger the file size is the longer it takes to parse it resulting in longer boot times.
The default file size for C:\windows\System32\Config\DEFAULT in a WinXPsp2 and above machine is 512 KB. This should not change. The file size in our problem PC's ranges from
20,xxx KB to 390,xxx KB. You probably would not notice the delay in boot times until this file reaches say 50,xxx KB. Typical numbers and boot times are:
80,xxx KB = about a 5 minute boot time
160,xxx KB = about a 10 minute boot time
250,xxx KB = about a 20 minute boot time
390,xxx KB = about a 45 minute boot time
Fixing the problem is a 2 step process.
Step 1: stop the bleeding
Step 2: replace the corrupt C:\windows\System32\Config\DEFAULT file with a good one.
Step 1: Stop and disable the LMS and UNS services:
We are pushing out a script via SCCM that stops these 2 services, and then disables them from starting again. If you uninstall the Intel Management System Components via control panel add/remove then you will remove the LMS/UNS services, however you will also remove the drivers for 2 PCI devices. This is indicated by the bangs in Device Manager for the PCI Serial Device and the PCI Simple Communications Controller.
The proper drivers for these 2 devices can be installed with running the ....R292087.exe executables. Just explode the R292087 (or R298379) and there is a driver folder that contains the drivers. Then in Device Manager you simply select to update the drivers for these to devices and browse to the driver folder to get the drivers.
Batch file we are scripting via SCCM to stop the services and then make the require registry changes to disable the LMS and UNS on all Dell PC's:
sc stop UNS
sc stop LMS
regedit -s c:\AMT.REG
AMT.REG contains the following 2 registry entry changes. Default value is 2 for automatic Startup of these services. Changing the value to 4 disables these services.
Windows Registry Editor Version 5.00
Step 2: Replace the damaged DEFAULT registry hive file with a good one that is 512 KB.
To do this you have to somehow boot-up outside of the PC's native OS as this file is loaded in the registry at bootup and cannot be moved/renamed or deleted in the native OS.
We are using a bootable flash drive to boot in Windows PE and then we run a batch file that will overwrite the bad file with a good file.
Plug the flash drive into the PC, restart it, use F12 to get to the Dell boot options menu, tell it to boot from the USB flash drive. When booted run your batch file to copy the good file to the location. Pull your flash drive and type EXIT. The system will automatically reboot normally.
This process takes 1.5 minutes total per unit if you want to stick around and see it reboot normally.
When you get tire of doing that just type the exit command and move to the next PC. (30 seconds per PC)
Hey could you post the batch file you used to copy the new file over the old? We have 120 machines that we ordered from dell, all 790s, and about 40 have this problem so far. I have been searching for a solution for a while thank you so much!
If we just disable the services do we need to install the updated driver?
It seems to me
that SAETECH supplied that batch file and also the registry changes required in his post.
He also explained the other steps he had to take.
I also have to package and automate this fix via SCCM. Overwriting the default file has to be done via a boot-up outside of the PC's native OS as this file is loaded in the registry at bootup and cannot be moved/renamed or deleted in the native OS. Is there a batch file for this process?
Pushing above fix via SCCM:
If there is no known script for above post, Unable to locate processes responsible. Can chkdsk.exe be taken out of reg so HD does not boot into OS. Then the default file can be dropped in and chdsk put back in the reg and OS boots with fresh default file?
Doesn't your IT staff create scripts and tools?
Above reply does not really help me really! Any knowledge out there re: replacing a sys32 file without going outside the OS? Suggestions appreciated.
I use an OS to do this.
So I can boot up BartPE or other OS and since I'm inside the OS I can do this.
Sorry but some companies are trying to get by without competent staff and are suffering for that. Some of them will be upset with me since I call them out on that issue. Good to get them upset and THINKING ABOUT THIS!
Again, your competent IT staff would use this article.
Yes it may read as if I beat up IT staff a lot. I admit it. I expect good IT staff to know their stuff, write code, scripts and more.
Outstanding work, I have been looking for a resolve for over 6 months.
Fixed it perfectly.
this worked for us...
Registry size: subkey ... Digest\hosts
the reason of registry size is in the subkey
HKEY_USERS \. DEFAULT \ Software \ Microsoft \ Windows \ CurrentVersion \ Internet Settings \ Digest \ Hosts
In our case there is only type subkeys "digest02410000n2: digest02410000n2" or similar.
Instead of replacing the registry file we've solved removing records with the command
reg delete "HKEY_USERS \. DEFAULT \ Software \ Microsoft \ Windows \ CurrentVersion \ Internet Settings \ Digest \ Hosts"
SLOW-BOOT on HP 8200 PC
We have similar issues with HP8200 PCs. What we did was rename the DEFAULT file and edited the UNS registry and it worked. We did not edit the LMS registry and still boots-up normally. But, over time, especially after updating Windows secuirty through Microsoft Update website, some of the slow-boot PCs before came back to booting-up so slow because the DEFAULT file increased in size again. Do we have to edit the LMS registry? Or is there anything that causes this?
This old post has good answers.
You didn't reveal if you tried them.
Thanks! Problem with Lenovo M81 fixed.
To SAETECH - THANKS! I owe you a beer...
We have many Dell 755's, 760's, and 780's in use...as well as Lenovo M81's and M82's.
Some of our M81's w/ XPSP3 have been experiencing the "Slow Boot" issue for months....with reimaging as our final option. As the issue began to be seen in our other models, we knew we had a growing problem that had to be solved....and, although we were getting close...we were stumped.
Thanks for taking the time to post your hard-earned findings for the rest of us. You rock, and I hope your employer is paying you what your worth!!!
Wondered if anyone has had repeated problems of this issue.
I have run the above fix with limited success in that I believe I have managed to stop the file growing by switching off LMS and UNS and preventing them from restarting.
Due to the nature of the systems (being remote) I have looked for alternate options to replacing the Default file.
On approx 35 systems, I have run a suggested fix to trim the host file from the registry digest which at 20 of the systems has reduced the Default file to less than 50mb. On 10 of the systems it has reduced the file to less than 1mb and thus proving this was the sole cause of the issue.
However, on some of the systems the triming of the registry has either only reduced the Default file or not changed it's size at all.
I therefore am led to believe that something else is also or has also been written to the registry which is hived into the default file on bootup.
Does anyone know what this could be? I have a few thousand systems to resolve so would appreciate any suggestions.
Thanks in advance.
This old thread is closed.
"t SAETECH supplied that batch file and also the registry changes required in his post."