AlwaysON VPN breaks after root certificate update

Scenario

  • After updating the internal CA root certificate, AlwaysOn VPN stops working with an error (at the user end) of “A Certificate could not be found that can be used with this Extensible Authentication Protocol
  • In this case, we were using an Enterprise integrated CA and renewed the root using the same signing keys – which should ease the process – at least for all windows clients
  • AOVPN is configured to use PEAP for authentication

 

Troubleshooting

  • Initially, 4 out of the 6 AOVPN servers had not received the new root cert from a GPupdate yet – so i forced that, restarted the service, but no difference
  • We discovered that the issue only occured on devices which had the updated trusted root cert in trusted root store. Additionally, for those that had updated, if we deleted the updated trusted root cert, AOVPN would connect again
  • We quickly found this article by the doyen of DirectAccess and AOVPN – https://directaccess.richardhicks.com/2020/10/19/always-on-vpn-ipsec-root-certificate-configuration-issue/  
    • While its a good article – it ended up not being our issue and actually led our down the wrong path a little
    • At the same time, for someone that wasn’t overly familiar with AOVPN (This was implemented by someone else and i’ve not had much to do with AOVPN) it was great, because i could look at the scripts and suss out some of the relevant powershell commandlets
  • After checking and re-checking every setting under the sun, a colleague could connect again after updating the client end
  • Once she worked that out, we then clarified and replicated the change on a different machine to be sure – and confirmed it was all good

 

Resolution

  • On a client machine, we updated the AOVPN configuration to include (i.e. tick the new as well as the old root cert) the updated root cert in 3 places under
    • <AOVPN connection name> / Properties / Security / Properties
    • <AOVPN connection name> / Properties / Security / Properties /Configure
    • <AOVPN connection name> / Properties / Security / Properties /Configure / Advanced
  • Confirm that the AOVPN connection is working
  • Export the profile using the script from https://directaccess.richardhicks.com/tag/profilexml/
  • Look at the xml – you should now see the thumbprints of both the “old” and “new” root certificate listed in multiple sections
  • Copy the section <EAPHostConfig> from its open xml tag to its close xml tag and insert into the “EAP xml” part of intune AOVPN configuration

Always on VPN – technical follow up

As a follow up to my article a few days ago on Always on VPN vs DA – http://www.hayesjupe.com/always-on-vpn-and-da-a-comparison/ – an employee of mine was having a test with some spare time today and came up with the following findings.

  • Configured and tested the VPN server using L2TP/IPSec + PSK, User/Pass using MS-CHAP-V2
  • Attempted to export the VPN profile using the Microsoft script MakeProfile.ps1 (https://docs.microsoft.com/en-us/windows-server/remote/remote-access/vpn/always-on-vpn/deploy/vpn-deploy-client-vpn-connections#bkmk_fullscript)
    • Doesn’t work if you’re using Folder Redirection, as it tries to write to C:\User\UserID\Desktop instead of using %desktop%
    • Adjusted the script to just write to C:\Temp and it works fine
  • Ran the generated VPN_Profile.ps1 and it comes back with “A general error occurred that is not covered by a more specific error code”. After doing some troubleshooting and googling, worked out that the MakeProfile.ps1 has “<AlwaysOn>true</AlwaysOn>” in it, when it actually needs to be “<AlwaysOn>True</AlwaysOn>” (upper-case T). Thanks Microsoft.
  • Finally got it imported. Attempted to connect and received an error that the destination address didn’t exist.
    • Checked the XML, the “Servers” item was populated correctly
    • Checked the VPN connection in Windows, the “Server” item wasn’t populated. Awesome.
  • Populated the Server field manually, tried to connect, failed.
    • The export also didn’t bring across the PSK
    • Populated the PSK, works.

To sum up:

  • Microsoft’s MakeProfile.ps1 is helpful, but isn’t even remotely reliable for exporting all of the settings
  • No idea why the server isn’t be populated. It’s in the XML, it just doesn’t populate it
  • There doesn’t seem to be a way of using PSK instead of certs – the XML doesn’t seem to have any options for specifying a PSK (that I’ve been able to find)

 

So let me revise my earlier “its very much a v1 product” to “its very much a v0.1 product”

Always on VPN and DA – a comparison

Date: 04/09/2018

For a while now, Microsoft salespeople have been telling customers that Direct Access is old technology and that Windows 10 always on VPN is the way to go.

There are a number of points which are widely acknowledged when it comes to DA vs Always on VPN, most of which can be found here

https://docs.microsoft.com/en-us/windows-server/remote/remote-access/vpn/vpn-map-da

What is the Difference Between DirectAccess and Always On VPN?

https://directaccess.richardhicks.com/2017/12/04/3-important-advantages-of-always-on-vpn-over-directaccess/

As usual, the Microsoft article, while technically correct, is not really very helpful. The ones from Richard Hicks are far more helpful, but still misses a few key points in my opinion.

With that in mind, lets have a bit more of a look at comparing these technologies.

 

Windows version support

Direct Access is supported on Windows 7, 8, 8.1 and 10 enterprise editions only. The target machines must also be domain joined.

Always on VPN is only supported with Windows 10 (1607 and newer), however, any edition of windows 10 (standard etc.) and the target machines can be domain joined or in a workgroup, or part of Azure AD.

Depending on where your organization is with its Win 10 migration, always on VPN may not be an option for you (but probably will be in the future at some point). Likewise, DA might not be an option if you do not have enterprise licensing or non-domain joined machines.

 

Backend

Direct Access requires a direct access server (or multiple if it is going to be made highly available), certificates, a public IP/DNS entry and port 443 opened at the firewall.

Always on VPN can connect to multiple vendors back ends such as Cisco, Fortinet, F5 etc.

If you are already locked in to a certain vendors VPN – always on VPN being able to leverage that is great.

 

Future Development

Direct Access is in Windows Server 2016 and we expect will be in windows Server 2019 and is fully supported, however, there are many places around the internet that state it is no longer under active development – and considering there have been no improvements in 2016 (over 2012 R2) – that does (unfortunately) seem to be the case.

Always on VPN comes across to me very much like a v1 product at the moment, we can assume that it will get further development… machine-based sessions got added in Windows 10 1709, but its hard to know with Microsoft. The current attitude seems to be to attack anything that isn’t azure or O365, even their own on-premise products – and always on VPN would tend to sit in the “on-premise” category at the moment….

 

Supportability

The first version of DA (in UAG) was a support nightmare, but it became quite servicable in 2012 R2 and 2016. It can be a steep learning curve for people that haven’t dealt with DA before, however it does actually have quite good support tools, even though they are a little disparate. There is a reasonable-ish level of community support for DA

Always on VPN as stated above very much seems to be a v1, Microsoft doco is generally unusable, TechNet no longer exists, so guides written by 3rd party bloggers etc remain important – and due to relative newness of the solution, there aren’t many guides out there – but there are a few.

 

Management

The DA management interface is pretty good and powershell commands (both client and server) are pretty complete.

Always on VPN – xml files and powershell…. very v1. Cant configure it via the three MS endpoint delivery systems natively – Intune, SCCM or group policy…. so yer, this is not up to scratch.

 

Performance

DA performance has never been great, the solution utilises multiple encapsulations and has to convert from IPV4 to IPV6 in most instances. Over a reasonable internet connection, its fine for the vast majority of things, but don’t try and transfer large files etc. I tend to think of it as a connection technology – the focus is on ease of connection, not performance.

Always on VPN, depending on what type of VPN you are using, is not going to have as many encapsulations and is going to be utilizing a tried and tested VPN technology. The result of this is that performance should be substantially better than DA.

 

Network detection

DA utilizes NLS (network location server) to detect if the client is on the internal network or not. The issue with this is that if your NLS does down, the client wont be able to connect to the network at all. This was one of the things I would hoping would be addressed in a future version of DA, the ability to configure multiple NLS’s in a “OR” configuration.

Always on VPN utilizes the DNS suffix of the network connection to determine if Always on should be utilized or not. This is definitely a step forward – it would be nice in the future to be able to define multiple DNS suffixes.

 

Machine vs user based VPN

DA can bring up the infrastructure tunnel prior to the full DA connection. This is incredibly good for machines which are permanently off the corporate network, as they can still apply computer policy prior to logon, update via internal WSUS or SCCM services (and AV definitions) without logon etc.

Always on VPN is a user based solution – so the user must logon before the VPN tunnel is established….  so no go for the machine updating policy, getting updates etc when not logged on. Before you say “big deal”, you try getting a user without admin rights to update group policy… now try that times all the roaming users in your org.

As of Windows 10 1709 – device based VPN sessions are now also available and require the machine to be domain-joined running enterprise or education editions….  which is fine, as that’s where device based connections are needed – when the machine is domain joined. The tunnel is IKEv2 by default, but can be configured for SSTP.

 

Locking down access

DA has no options for locking down access. Just none – they either have DA or they do not. Theoretically you could prevent access to certain locations/devices at a layer 3 level… but that’s a lot of effort. You could also create multiple DA services, but again, its a lot of effort…. it would have been nice to see more development in this area.

Always on VPN has many more options when it comes to locking down the connections – and different configurations can be deployed to different users. The current options are pretty good.

 

Split and force tunneling

DA supports both, but is difficult to configure and effectively unsupported when used in force tunneling mode

Always on VPN supports force tunneling and given its different architecture to DA, should less of an issue… but I haven’t actually tried it as yet.

 

Manage-out

DA manage-out functionality is great, its not hard to configure, but it is “harder than it needs to be” – but much of this complexity is due to the IPv4/6 transition technologies.

Always on VPN does not have the IPv4/6 transition issues and simply allows you to configure DNS registration in the xml via – <RegisterDNS>true</RegisterDNS> – which in turn effectively enables manage out – much easier.

 

Summing it up

As per many things, its whatever suits your environment and needs best.

If DA ticks boxes for your environment as it is right now – its still a good technology, particularly if you wont be off of Windows 7 for a few years yet – but don’t expect it to get any improvements – and expect to have to migrate off it at some point (but, assuming its in Server 2019, not for quite a while)

Always on VPN clearly has some pretty good stuff around access control and back end support. The windows 10 (1709+ for the most options at time of writing) requirement will become less of any issue over the next few years, but it really needs to get its deployment methods sorted out – with native support in Intune and SCCM….. and ideally, group policy (but I think the last one is a long shot)

The disappointment of lack of Direct Access development

<rant mode: on>

Direct Access (or DA) is awesome. Much like TMG before it, it fits into a segment of the market that nothing else covers in quite the same way.

DA is fully supported in 2016, but has had no new features added. I read somewhere that while it is still supported, it is no longer under active development (but I have no credible references to back that up).

DA could go from awesome (where it is now) to super-awesome (yes, that is my technical term for it) by:

  • Allowing network control based on group at the server side (i.e. if member of this group, users are only allowed to 10.10.10.x subnet etc.)
  • Allowing more control on the client side (i.e. a group policy to optionally allow the user to enable/disable multiple optional DA entries <or just the one>)
  • Allowing creation of a DA “package” that could be sent to non-domain machines to still allow DA connections (in conjunction with the above)

Outside of that, we also had a client recently pass on that their Microsoft TAM was ragging on DA, claiming that its out-dated technology… I can only assume because “everything should be in the cloud”. Organisations aren’t going (and technically cannot in many cases) to move everything to the cloud overnight….. even if they did, clients still need to be able to get onto the corporate network – and some things companies may not wish to make some apps/data available publically – even with MFA/certs etc.

 

Anyway, this is my plea…. MS, don’t fuck up with DA like you did with TMG. Its a good product, develop it.

<rant mode: off>

DA, force tunnelling and www.msftncsi.com

At a client – 2012 R2 IPHTTPS DA and win 10 1511…..

Client wants force tunnelling so all traffic is routed back through the internal network and the web filter etc….

Issue is, www.msftncsi.com cannot be contacted as soon as force tunnelling is enabled.

Tried a couple of basic things, such as adding msftncsi.com into the excluded DNS suffixes, no dice.

Found this article from a poor guy with the same issue – http://wmug.co.uk/wmug/b/mattwhite/archive/2014/12/18/directaccess-force-tunneling-and-a-corporate-proxy

Tried the suggestions in the article – and while the situation got better – it was still too unpredicatble for production use.

Force tunnelling isn’t something that we have asked to switch on at many clients – in fact, i think this is the first time…. its disappointing to see, what comes across a quite simple issue. Unless im missing something, MS could simply hard code clients to ignore force tunnelling only for www.msftncsi.com and make the issue go away…. which does open potential for users with local admin to use the hosts file to obtain “direct” internet access – but hey – maybe im looking at it too simplistically myself.

I think the take away here is “don’t use force tunnelling”