Got a call from a client who was having issues with the SQL instance on their SCCM server – and investigation showed that the SQL service was crashing due to various memory errors (event log and SQL logs) – but the descriptions weren’t overly helpful.
The SQL exception.log shows errors such as
09/12/19 12:23:58 spid 125 Exception 0xc0000005 EXCEPTION_ACCESS_VIOLATION writing address 000001E1F29E3390 at 0x000001E1F29E3390
After a bit of investigation, i noticed that the “system” task in task manager was constantly utilising between 20-40% CPU. The “system” task has no associated command line in task manager, so tracking it down required the use of the ever-helpful sysinternal tools – in this case, process explorer.
Once opening process explorer, you can go to the properties of the “system” process and view all its threads – and most importantly, sort by CPU usage.
In this case, i could see that Vmmemctl.sys was using the vast majority of the CPU time within this process.
A quick google lead me to this https://kb.vmware.com/s/article/2138677
While i wasn’t getting blue screens, i was definitely getting memory errors – so this lined up.
Checking the installed programs, i could then see that VMWare tools 10.2.5 was installed, but so was 9.1.
Removed VMWare tools 9.1 from the server and the CPU use immediately dropped – and the memory issues, at least so far, are not longer occurring.
Surprisingly, this didn’t seem to require a reboot after the VMWare tools 9.1 uninstall.
I guess the moral of this story (post) is – keeping your VMWare tools version up to date is wise….. but don’t forget to uninstall old versions as well.