VMWare guest server CPU and memory issues

Posted on by Hayes Jupe

Got a call from a client who was having issues with the SQL instance on their SCCM server – and investigation showed that the SQL service was crashing due to various memory errors (event log and SQL logs) – but the descriptions weren’t overly helpful.

The SQL exception.log shows errors such as

09/12/19 12:23:58 spid 125 Exception 0xc0000005 EXCEPTION_ACCESS_VIOLATION writing address 000001E1F29E3390 at 0x000001E1F29E3390

 

After a bit of investigation, i noticed that the “system” task in task manager was constantly utilising between 20-40% CPU. The “system” task has no associated command line in task manager, so tracking it down required the use of the ever-helpful sysinternal tools – in this case, process explorer.

Once opening process explorer, you can go to the properties of the “system” process and view all its threads – and most importantly, sort by CPU usage.

In this case, i could see that Vmmemctl.sys was using the vast majority of the CPU time within this process.

A quick google lead me to this https://kb.vmware.com/s/article/2138677

While i wasn’t getting blue screens, i was definitely getting memory errors – so this lined up.

Checking the installed programs, i could then see that VMWare tools 10.2.5 was installed, but so was 9.1.

Removed VMWare tools 9.1 from the server and the CPU use immediately dropped – and the memory issues, at least so far, are not longer occurring.

Surprisingly, this didn’t seem to require a reboot after the VMWare tools 9.1 uninstall.

I guess the moral of this story (post) is – keeping your VMWare tools version up to date is wise….. but don’t forget to uninstall old versions as well.

SCCM 1906 release

Posted on by Hayes Jupe

SCCM 1906 released! – https://docs.microsoft.com/en-us/sccm/core/plan-design/changes/whats-new-in-version-1906

If you want it right now – you can opt-in via the fast ring script – https://go.microsoft.com/fwlink/?linkid=2099733

Another belting release from the SCCM team – while some releases have different focuses over time – generally releases have tended to have something which makes day-to-day admin life a bit easier for someone…. and sometimes, big things, such as passive site servers, that just structurally make the product substantially better.

Anyway – out of this release comes a couple of items that i think are particularly of note to me (other may be interested in different parts – depending on your setup):

  • Site maintenance UI is friendlier
  • Use your distribution point as an in-network cache server for Delivery Optimization
  • Support for Windows Virtual Desktop
  • OneTrace – a preview of a tool which could be dubbed the next-gen of CMTrace…
  • Improvements to co-management auto-enrollment
  • Retry the install of pre-approved applications
  • Task sequence debugger – not there yet – but great that its being thought about – can see this being exceedingly handy as this matures
  • The Disable BitLocker task sequence step has a new restart counter
  • Additional options for WSUS maintenance – these are brilliant. Hopefully this release will also fix bug id 4808740 – i haven’t been able to test this as yet.
  • New Windows 10, version 1903 and later product category – this enables admins everywhere to further reduce the number of updates stored by WSUS – which leads to improvements with server and client performance
  • Role-based access for folders – finally!
  • Administration service support for security nodes – this is potentially a big change – keen to test this out in a bigger environment
  • Collections tab in devices node – gradually chipping away at right click tools functionality
  • Multiselect and delete packages – finally!

 

 

 

 

Microsoft NCSI – prompt for proxy authentication

Posted on by Hayes Jupe

NCSI has been around for a long time now.

 

It can be disabled by using the policy at Computer Configuration\Computer Configuration\Administrative Templates\System\Internet Communication Management \ Turn off Windows Network Connectivity Status active tests

however, disabling it has impacts on technologies such as direct access.

Recently a client was getting prompted for auth form their proxy, for all connections, wired, wireless and 4G.

Msftncsi.com had been added as un-authenticated location for proxy access, but it was still occurring on Windows 10 1809.

Googling this found a few sites talking about proxy issues, disabling NCSI or re-directing this. I did not want to disable or re-direct, and the proxy issues didnt seem to fit our situation.

I ended up going down the wireshark path and discovered that www.msftconnecttest.com is now the DNS name used for NCSI resolution.

Added this to the list of sites which do no required auth – and all is good with the world again.

MYOB – finding the current library root for server edition

Posted on by Hayes Jupe

MYOB – other small business owners may be familiar with this software…. its not good software and its expensive…. but it is one of the options out there for small business.

Anyway – upgrade MYOB server edition

When performing this upgrade, it automatically resets the Library location back to the default – which is not helpful…. and there is no way in the GUI to determine the library location (that i can see)

So – in order to ascertain your library location prior to upgrade, check the following registry key

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\MYOB\HuxleyServer\LibraryRoot (assuming an x64 server, remove the “wow6432Node” if you are somehow still running a 32 bit server OS in 2019)

HP CM1312 – using the network scanner with Windows 10

Posted on by Hayes Jupe

I still have an older MFP – the HP CM1312nfi

The printer is detected and installs fine on Windows 10, the scanner however is not detected.

The Microsoft support articles are junk… “Windows will automatically find your devices” – with no assistance on what to do if Windows doesn’t find your device.

The HP site links to the positively ancient product install here – https://support.hp.com/us-en/drivers/selfservice/hp-color-laserjet-cm1312-multifunction-printer-series/3558902/model/3558903 

No great surprise – this installer fails to detect the printer, let alone the scanner.

The “HP Smart” Windows 10 app also doesn’t detect the scanner.

To get this scanner working

  1. Download the product driver install package here https://support.hp.com/us-en/drivers/selfservice/hp-color-laserjet-cm1312-multifunction-printer-series/3558902/model/3558903
  2. Extract the installer to a directory using 7zip
  3. Locate hppasc11.inf within the extraction directory – right click | install
  4. Give it a few seconds – all done.

I suspect HP would you prefer you purchased a newer model…. but its a printer/scanner that still works…. they just haven’t made it easy.

Draytek – central AP management – not so great

Posted on by Hayes Jupe

I grabbed a couple of Draytek Vigor AP902 access points to work in conjunction with my Draytek Vigor 2925 router and Draytek Vigor 910c access point.

I dont mind the Draytek gear in general – for a home setup it is sufficient… however i did find i needed better wireless range thought that 3 AP’s with roaming would be a good fit…. and Drayteks central management swayed me towards getting an all-draytek solution. (as if you can manage 3 AP’s centrally, why wouldn’t you?!)

In short – it was/is a disappointment. The central AP management interface, accessed via the vigor 2925 is quite basic and there is no feedback indicating if applying a configuration to AP’s has been successful or not (for example).

The biggest issue however is that my PSK was applied incorrectly to all access points – this became apparent when all devices lost connectivity, despite the PSK being “the same”

When logging in directly to one of the access points to troubleshoot, i found the PSK in clear text, missing the last 5 characters. This was both good and bad…. bad as it was displayed in clear text, but good as it made the issue very clear.

From there, i simply updated the PSK on each individual AP, each with a 2.4 and 5.0GHz SSID, so 6 updates – and all was OK. Then disabled central management on the 2925.

If someone else is thinking of going for an all draytek solution for the same reasons…. my impression at the moment is – don’t do it.

I’ve emailed draytek support – and will update this post with their reply.

 

*** Update 07/01/2019 ***

I got a reply from Australian Draytek support – initially i got told “it works fine” – when i replied asking if a WPA2 key with certain characters in it, it does look like that a WPA2 key with “&” in it cannot be managed via the Vigor 2925 central management

Active Directory 2019 and Exchange 2019 – what’s new

Posted on by Hayes Jupe

The short answer is – not much.

Exchange 2019 was released a few weeks back, but was effectively un-usable, as Exchange 2019 requires Windows Server 2019…. and Windows server 2019 got pulled from release (like Windows 10 1809) due to some issues.

Windows Server 2019 was re-released a few days ago, which allowed nerds everywhere (including me) to put Server 2019 and Exchange 2019 into a test environment.

The most striking thing that is immediately noticeable is that everything looks the same…. The install process, the GUI, the management, all looks the same as it did in 2016. To me, this is a good thing – while Microsoft of the past seemed to believe that moving functions between areas was good – some consistency is nice to have too.

 

Active Directory

First appearances indicate there is nothing new in AD 2019, the installation process and management is exactly the same as 2016.

While installing, there is not even an option to set the forest and domain functional level to “2019” – only 2016.

A quick look at the schema version indicates it has increased and quick google finds this article

https://blogs.technet.microsoft.com/389thoughts/2018/08/21/whats-new-in-active-directory-2019-nothing/

So, while there is something new in the schema, its an incredibly small update….. and there are no new features or functionality of any type to focus on.

 

Exchange 2019

Exchange 2019 is a bit the same as AD, everything appears to be the same as Exchange 2016, from the install process to the management interface.

A google comes up with this

Should you upgrade to Exchange Server 2019?

So there are some changes and feature updates – but these updates may not have an impact/matter to your organization.

 

I found these two releases interesting overall as

  • AD is the core of many enterprise networks
  • Exchange is a core business application

To see a new release of both of these products with very minimal improvements I think demonstrates where all Microsoft’s development effort is going (which, to be fair, we already knew)

 

Always on VPN – technical follow up

Posted on by Hayes Jupe

As a follow up to my article a few days ago on Always on VPN vs DA – http://www.hayesjupe.com/always-on-vpn-and-da-a-comparison/ – an employee of mine was having a test with some spare time today and came up with the following findings.

  • Configured and tested the VPN server using L2TP/IPSec + PSK, User/Pass using MS-CHAP-V2
  • Attempted to export the VPN profile using the Microsoft script MakeProfile.ps1 (https://docs.microsoft.com/en-us/windows-server/remote/remote-access/vpn/always-on-vpn/deploy/vpn-deploy-client-vpn-connections#bkmk_fullscript)
    • Doesn’t work if you’re using Folder Redirection, as it tries to write to C:\User\UserID\Desktop instead of using %desktop%
    • Adjusted the script to just write to C:\Temp and it works fine
  • Ran the generated VPN_Profile.ps1 and it comes back with “A general error occurred that is not covered by a more specific error code”. After doing some troubleshooting and googling, worked out that the MakeProfile.ps1 has “<AlwaysOn>true</AlwaysOn>” in it, when it actually needs to be “<AlwaysOn>True</AlwaysOn>” (upper-case T). Thanks Microsoft.
  • Finally got it imported. Attempted to connect and received an error that the destination address didn’t exist.
    • Checked the XML, the “Servers” item was populated correctly
    • Checked the VPN connection in Windows, the “Server” item wasn’t populated. Awesome.
  • Populated the Server field manually, tried to connect, failed.
    • The export also didn’t bring across the PSK
    • Populated the PSK, works.

To sum up:

  • Microsoft’s MakeProfile.ps1 is helpful, but isn’t even remotely reliable for exporting all of the settings
  • No idea why the server isn’t be populated. It’s in the XML, it just doesn’t populate it
  • There doesn’t seem to be a way of using PSK instead of certs – the XML doesn’t seem to have any options for specifying a PSK (that I’ve been able to find)

 

So let me revise my earlier “its very much a v1 product” to “its very much a v0.1 product”

Always on VPN and DA – a comparison

Posted on by Hayes Jupe

Date: 04/09/2018

For a while now, Microsoft salespeople have been telling customers that Direct Access is old technology and that Windows 10 always on VPN is the way to go.

There are a number of points which are widely acknowledged when it comes to DA vs Always on VPN, most of which can be found here

https://docs.microsoft.com/en-us/windows-server/remote/remote-access/vpn/vpn-map-da

What is the Difference Between DirectAccess and Always On VPN?

https://directaccess.richardhicks.com/2017/12/04/3-important-advantages-of-always-on-vpn-over-directaccess/

As usual, the Microsoft article, while technically correct, is not really very helpful. The ones from Richard Hicks are far more helpful, but still misses a few key points in my opinion.

With that in mind, lets have a bit more of a look at comparing these technologies.

 

Windows version support

Direct Access is supported on Windows 7, 8, 8.1 and 10 enterprise editions only. The target machines must also be domain joined.

Always on VPN is only supported with Windows 10 (1607 and newer), however, any edition of windows 10 (standard etc.) and the target machines can be domain joined or in a workgroup, or part of Azure AD.

Depending on where your organization is with its Win 10 migration, always on VPN may not be an option for you (but probably will be in the future at some point). Likewise, DA might not be an option if you do not have enterprise licensing or non-domain joined machines.

 

Backend

Direct Access requires a direct access server (or multiple if it is going to be made highly available), certificates, a public IP/DNS entry and port 443 opened at the firewall.

Always on VPN can connect to multiple vendors back ends such as Cisco, Fortinet, F5 etc.

If you are already locked in to a certain vendors VPN – always on VPN being able to leverage that is great.

 

Future Development

Direct Access is in Windows Server 2016 and we expect will be in windows Server 2019 and is fully supported, however, there are many places around the internet that state it is no longer under active development – and considering there have been no improvements in 2016 (over 2012 R2) – that does (unfortunately) seem to be the case.

Always on VPN comes across to me very much like a v1 product at the moment, we can assume that it will get further development… machine-based sessions got added in Windows 10 1709, but its hard to know with Microsoft. The current attitude seems to be to attack anything that isn’t azure or O365, even their own on-premise products – and always on VPN would tend to sit in the “on-premise” category at the moment….

 

Supportability

The first version of DA (in UAG) was a support nightmare, but it became quite servicable in 2012 R2 and 2016. It can be a steep learning curve for people that haven’t dealt with DA before, however it does actually have quite good support tools, even though they are a little disparate. There is a reasonable-ish level of community support for DA

Always on VPN as stated above very much seems to be a v1, Microsoft doco is generally unusable, TechNet no longer exists, so guides written by 3rd party bloggers etc remain important – and due to relative newness of the solution, there aren’t many guides out there – but there are a few.

 

Management

The DA management interface is pretty good and powershell commands (both client and server) are pretty complete.

Always on VPN – xml files and powershell…. very v1. Cant configure it via the three MS endpoint delivery systems natively – Intune, SCCM or group policy…. so yer, this is not up to scratch.

 

Performance

DA performance has never been great, the solution utilises multiple encapsulations and has to convert from IPV4 to IPV6 in most instances. Over a reasonable internet connection, its fine for the vast majority of things, but don’t try and transfer large files etc. I tend to think of it as a connection technology – the focus is on ease of connection, not performance.

Always on VPN, depending on what type of VPN you are using, is not going to have as many encapsulations and is going to be utilizing a tried and tested VPN technology. The result of this is that performance should be substantially better than DA.

 

Network detection

DA utilizes NLS (network location server) to detect if the client is on the internal network or not. The issue with this is that if your NLS does down, the client wont be able to connect to the network at all. This was one of the things I would hoping would be addressed in a future version of DA, the ability to configure multiple NLS’s in a “OR” configuration.

Always on VPN utilizes the DNS suffix of the network connection to determine if Always on should be utilized or not. This is definitely a step forward – it would be nice in the future to be able to define multiple DNS suffixes.

 

Machine vs user based VPN

DA can bring up the infrastructure tunnel prior to the full DA connection. This is incredibly good for machines which are permanently off the corporate network, as they can still apply computer policy prior to logon, update via internal WSUS or SCCM services (and AV definitions) without logon etc.

Always on VPN is a user based solution – so the user must logon before the VPN tunnel is established….  so no go for the machine updating policy, getting updates etc when not logged on. Before you say “big deal”, you try getting a user without admin rights to update group policy… now try that times all the roaming users in your org.

As of Windows 10 1709 – device based VPN sessions are now also available and require the machine to be domain-joined running enterprise or education editions….  which is fine, as that’s where device based connections are needed – when the machine is domain joined. The tunnel is IKEv2 by default, but can be configured for SSTP.

 

Locking down access

DA has no options for locking down access. Just none – they either have DA or they do not. Theoretically you could prevent access to certain locations/devices at a layer 3 level… but that’s a lot of effort. You could also create multiple DA services, but again, its a lot of effort…. it would have been nice to see more development in this area.

Always on VPN has many more options when it comes to locking down the connections – and different configurations can be deployed to different users. The current options are pretty good.

 

Split and force tunneling

DA supports both, but is difficult to configure and effectively unsupported when used in force tunneling mode

Always on VPN supports force tunneling and given its different architecture to DA, should less of an issue… but I haven’t actually tried it as yet.

 

Manage-out

DA manage-out functionality is great, its not hard to configure, but it is “harder than it needs to be” – but much of this complexity is due to the IPv4/6 transition technologies.

Always on VPN does not have the IPv4/6 transition issues and simply allows you to configure DNS registration in the xml via – <RegisterDNS>true</RegisterDNS> – which in turn effectively enables manage out – much easier.

 

Summing it up

As per many things, its whatever suits your environment and needs best.

If DA ticks boxes for your environment as it is right now – its still a good technology, particularly if you wont be off of Windows 7 for a few years yet – but don’t expect it to get any improvements – and expect to have to migrate off it at some point (but, assuming its in Server 2019, not for quite a while)

Always on VPN clearly has some pretty good stuff around access control and back end support. The windows 10 (1709+ for the most options at time of writing) requirement will become less of any issue over the next few years, but it really needs to get its deployment methods sorted out – with native support in Intune and SCCM….. and ideally, group policy (but I think the last one is a long shot)