VMWare guest server CPU and memory issues

Got a call from a client who was having issues with the SQL instance on their SCCM server – and investigation showed that the SQL service was crashing due to various memory errors (event log and SQL logs) – but the descriptions weren’t overly helpful.

The SQL exception.log shows errors such as

09/12/19 12:23:58 spid 125 Exception 0xc0000005 EXCEPTION_ACCESS_VIOLATION writing address 000001E1F29E3390 at 0x000001E1F29E3390


After a bit of investigation, i noticed that the “system” task in task manager was constantly utilising between 20-40% CPU. The “system” task has no associated command line in task manager, so tracking it down required the use of the ever-helpful sysinternal tools – in this case, process explorer.

Once opening process explorer, you can go to the properties of the “system” process and view all its threads – and most importantly, sort by CPU usage.

In this case, i could see that Vmmemctl.sys was using the vast majority of the CPU time within this process.

A quick google lead me to this https://kb.vmware.com/s/article/2138677

While i wasn’t getting blue screens, i was definitely getting memory errors – so this lined up.

Checking the installed programs, i could then see that VMWare tools 10.2.5 was installed, but so was 9.1.

Removed VMWare tools 9.1 from the server and the CPU use immediately dropped – and the memory issues, at least so far, are not longer occurring.

Surprisingly, this didn’t seem to require a reboot after the VMWare tools 9.1 uninstall.

I guess the moral of this story (post) is – keeping your VMWare tools version up to date is wise….. but don’t forget to uninstall old versions as well.

Always on VPN – technical follow up

As a follow up to my article a few days ago on Always on VPN vs DA – http://www.hayesjupe.com/always-on-vpn-and-da-a-comparison/ – an employee of mine was having a test with some spare time today and came up with the following findings.

  • Configured and tested the VPN server using L2TP/IPSec + PSK, User/Pass using MS-CHAP-V2
  • Attempted to export the VPN profile using the Microsoft script MakeProfile.ps1 (https://docs.microsoft.com/en-us/windows-server/remote/remote-access/vpn/always-on-vpn/deploy/vpn-deploy-client-vpn-connections#bkmk_fullscript)
    • Doesn’t work if you’re using Folder Redirection, as it tries to write to C:\User\UserID\Desktop instead of using %desktop%
    • Adjusted the script to just write to C:\Temp and it works fine
  • Ran the generated VPN_Profile.ps1 and it comes back with “A general error occurred that is not covered by a more specific error code”. After doing some troubleshooting and googling, worked out that the MakeProfile.ps1 has “<AlwaysOn>true</AlwaysOn>” in it, when it actually needs to be “<AlwaysOn>True</AlwaysOn>” (upper-case T). Thanks Microsoft.
  • Finally got it imported. Attempted to connect and received an error that the destination address didn’t exist.
    • Checked the XML, the “Servers” item was populated correctly
    • Checked the VPN connection in Windows, the “Server” item wasn’t populated. Awesome.
  • Populated the Server field manually, tried to connect, failed.
    • The export also didn’t bring across the PSK
    • Populated the PSK, works.

To sum up:

  • Microsoft’s MakeProfile.ps1 is helpful, but isn’t even remotely reliable for exporting all of the settings
  • No idea why the server isn’t be populated. It’s in the XML, it just doesn’t populate it
  • There doesn’t seem to be a way of using PSK instead of certs – the XML doesn’t seem to have any options for specifying a PSK (that I’ve been able to find)


So let me revise my earlier “its very much a v1 product” to “its very much a v0.1 product”

Always on VPN and DA – a comparison

Date: 04/09/2018

For a while now, Microsoft salespeople have been telling customers that Direct Access is old technology and that Windows 10 always on VPN is the way to go.

There are a number of points which are widely acknowledged when it comes to DA vs Always on VPN, most of which can be found here


What is the Difference Between DirectAccess and Always On VPN?


As usual, the Microsoft article, while technically correct, is not really very helpful. The ones from Richard Hicks are far more helpful, but still misses a few key points in my opinion.

With that in mind, lets have a bit more of a look at comparing these technologies.


Windows version support

Direct Access is supported on Windows 7, 8, 8.1 and 10 enterprise editions only. The target machines must also be domain joined.

Always on VPN is only supported with Windows 10 (1607 and newer), however, any edition of windows 10 (standard etc.) and the target machines can be domain joined or in a workgroup, or part of Azure AD.

Depending on where your organization is with its Win 10 migration, always on VPN may not be an option for you (but probably will be in the future at some point). Likewise, DA might not be an option if you do not have enterprise licensing or non-domain joined machines.



Direct Access requires a direct access server (or multiple if it is going to be made highly available), certificates, a public IP/DNS entry and port 443 opened at the firewall.

Always on VPN can connect to multiple vendors back ends such as Cisco, Fortinet, F5 etc.

If you are already locked in to a certain vendors VPN – always on VPN being able to leverage that is great.


Future Development

Direct Access is in Windows Server 2016 and we expect will be in windows Server 2019 and is fully supported, however, there are many places around the internet that state it is no longer under active development – and considering there have been no improvements in 2016 (over 2012 R2) – that does (unfortunately) seem to be the case.

Always on VPN comes across to me very much like a v1 product at the moment, we can assume that it will get further development… machine-based sessions got added in Windows 10 1709, but its hard to know with Microsoft. The current attitude seems to be to attack anything that isn’t azure or O365, even their own on-premise products – and always on VPN would tend to sit in the “on-premise” category at the moment….



The first version of DA (in UAG) was a support nightmare, but it became quite servicable in 2012 R2 and 2016. It can be a steep learning curve for people that haven’t dealt with DA before, however it does actually have quite good support tools, even though they are a little disparate. There is a reasonable-ish level of community support for DA

Always on VPN as stated above very much seems to be a v1, Microsoft doco is generally unusable, TechNet no longer exists, so guides written by 3rd party bloggers etc remain important – and due to relative newness of the solution, there aren’t many guides out there – but there are a few.



The DA management interface is pretty good and powershell commands (both client and server) are pretty complete.

Always on VPN – xml files and powershell…. very v1. Cant configure it via the three MS endpoint delivery systems natively – Intune, SCCM or group policy…. so yer, this is not up to scratch.



DA performance has never been great, the solution utilises multiple encapsulations and has to convert from IPV4 to IPV6 in most instances. Over a reasonable internet connection, its fine for the vast majority of things, but don’t try and transfer large files etc. I tend to think of it as a connection technology – the focus is on ease of connection, not performance.

Always on VPN, depending on what type of VPN you are using, is not going to have as many encapsulations and is going to be utilizing a tried and tested VPN technology. The result of this is that performance should be substantially better than DA.


Network detection

DA utilizes NLS (network location server) to detect if the client is on the internal network or not. The issue with this is that if your NLS does down, the client wont be able to connect to the network at all. This was one of the things I would hoping would be addressed in a future version of DA, the ability to configure multiple NLS’s in a “OR” configuration.

Always on VPN utilizes the DNS suffix of the network connection to determine if Always on should be utilized or not. This is definitely a step forward – it would be nice in the future to be able to define multiple DNS suffixes.


Machine vs user based VPN

DA can bring up the infrastructure tunnel prior to the full DA connection. This is incredibly good for machines which are permanently off the corporate network, as they can still apply computer policy prior to logon, update via internal WSUS or SCCM services (and AV definitions) without logon etc.

Always on VPN is a user based solution – so the user must logon before the VPN tunnel is established….  so no go for the machine updating policy, getting updates etc when not logged on. Before you say “big deal”, you try getting a user without admin rights to update group policy… now try that times all the roaming users in your org.

As of Windows 10 1709 – device based VPN sessions are now also available and require the machine to be domain-joined running enterprise or education editions….  which is fine, as that’s where device based connections are needed – when the machine is domain joined. The tunnel is IKEv2 by default, but can be configured for SSTP.


Locking down access

DA has no options for locking down access. Just none – they either have DA or they do not. Theoretically you could prevent access to certain locations/devices at a layer 3 level… but that’s a lot of effort. You could also create multiple DA services, but again, its a lot of effort…. it would have been nice to see more development in this area.

Always on VPN has many more options when it comes to locking down the connections – and different configurations can be deployed to different users. The current options are pretty good.


Split and force tunneling

DA supports both, but is difficult to configure and effectively unsupported when used in force tunneling mode

Always on VPN supports force tunneling and given its different architecture to DA, should less of an issue… but I haven’t actually tried it as yet.



DA manage-out functionality is great, its not hard to configure, but it is “harder than it needs to be” – but much of this complexity is due to the IPv4/6 transition technologies.

Always on VPN does not have the IPv4/6 transition issues and simply allows you to configure DNS registration in the xml via – <RegisterDNS>true</RegisterDNS> – which in turn effectively enables manage out – much easier.


Summing it up

As per many things, its whatever suits your environment and needs best.

If DA ticks boxes for your environment as it is right now – its still a good technology, particularly if you wont be off of Windows 7 for a few years yet – but don’t expect it to get any improvements – and expect to have to migrate off it at some point (but, assuming its in Server 2019, not for quite a while)

Always on VPN clearly has some pretty good stuff around access control and back end support. The windows 10 (1709+ for the most options at time of writing) requirement will become less of any issue over the next few years, but it really needs to get its deployment methods sorted out – with native support in Intune and SCCM….. and ideally, group policy (but I think the last one is a long shot)

SMB 1 no longer installed by default in Win 10 1710


As per the link above, SMB 1 will no longer be installed by default in Win 10 1710 (which, given the release date, I’m guess that’s what it will be called among techs, rather than the exceedingly shitty “fall creators update” name  – because calling two different versions “creators update” is logical) or the next version of Server 2016 (whatever that ends up being called).

Considering the recent-ish SMB1 targeted attacks, this isn’t surprising – and is a good move in my opinion. Issue is of course, the companies likely to hit by SMB1 (or other old-school attacks) are likely to not be up to date with their patching and even less likely to be up to date with OS versions – so it wont help secure the more vulnerable networks out there….



Licensing mode for the Remote Desktop Session Host is not configured

Had a situation recently when building a 2012 R2 RDS farm that the message

“Licensing mode for the Remote Desktop Session Host is not configured

kept appearing, even though the licensing server was activated etc. and the server was configured to use it.

Thankfully, this site had the answer


$obj = gwmi -namespace “Root/CIMV2/TerminalServices” Win32_TerminalServiceSetting
$obj. SetSpecifiedLicenseServerList(“licserver.domain.local”)

HKLM\SYSTEM\CurrentControlSet\Control\Terminal Server\RCM\Licensing Core\LicensingMode
Change the DWORD to 2 for Per Device or 4 for Per User


Update 6/09/2016

An employee mentioned to me that setting the license server and licensing mode via group policy also seems to get around this bug

Computer Policy\Computer Configuration\Administrative Templates\Windows Components\Remote Desktop Services\Remote Desktop Session Host


XP/2003 clients cannot run logon scripts from 2012 R2 U1 servers

Recently we had a client who diligently updated their 2012 R2 DC’s to U1 as part of their normal patching cycle – and found afterwards that their 2003 servers were experiencing some “odd” behaviour.

After having a look, sure enough, networking was fine, RPC was fine, all services were fine, but when trying to connect to netlogon, the error: “The specified network name is no longer available” was thrown.

A short amount of searching later, turned up this article:

Our experience of the issue was slightly different – in that it did not occur when the 2012 R2 DC’s were first upgraded, only when U1 was applied, in addition, the 2003 clients could connect to the file server which was running 2012 R2 (not u1) without any issues, but had trouble to any file shares on 2012 R2 u1 servers. It is also worth noting that all of these servers had been upgraded from 2008 R2/2012 – none were fresh builds.

The registry change documented in the article worked and all was good with the world. In this instance, the client only had a small number of servers still running 2003 – and this has given them a bit of a hurry up to sort out the apps on these servers.

*edit 24/11/2015 * – The original article seems to have become unavailable… so the changes are:

  • Install the feature “SMB 1.0/CIFS file sharing support”
  • Navigate HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\LanmanServer
  • Edit DependOnService and change
  • SamSS Srv2
  • to
  • SamSS Srv Srv2
  • Restart