AlwaysON VPN breaks after root certificate update

Posted on by Hayes Jupe

Scenario

  • After updating the internal CA root certificate, AlwaysOn VPN stops working with an error (at the user end) of “A Certificate could not be found that can be used with this Extensible Authentication Protocol“
  • In this case, we were using an Enterprise integrated CA and renewed the root using the same signing keys – which should ease the process – at least for all windows clients
  • AOVPN is configured to use PEAP for authentication

 

Troubleshooting

  • Initially, 4 out of the 6 AOVPN servers had not received the new root cert from a GPupdate yet – so i forced that, restarted the service, but no difference
  • We discovered that the issue only occured on devices which had the updated trusted root cert in trusted root store. Additionally, for those that had updated, if we deleted the updated trusted root cert, AOVPN would connect again
  • We quickly found this article by the doyen of DirectAccess and AOVPN – https://directaccess.richardhicks.com/2020/10/19/always-on-vpn-ipsec-root-certificate-configuration-issue/  
    • While its a good article – it ended up not being our issue and actually led our down the wrong path a little
    • At the same time, for someone that wasn’t overly familiar with AOVPN (This was implemented by someone else and i’ve not had much to do with AOVPN) it was great, because i could look at the scripts and suss out some of the relevant powershell commandlets
  • After checking and re-checking every setting under the sun, a colleague could connect again after updating the client end
  • Once she worked that out, we then clarified and replicated the change on a different machine to be sure – and confirmed it was all good

 

Resolution

  • On a client machine, we updated the AOVPN configuration to include (i.e. tick the new as well as the old root cert) the updated root cert in 3 places under
    • <AOVPN connection name> / Properties / Security / Properties
    • <AOVPN connection name> / Properties / Security / Properties /Configure
    • <AOVPN connection name> / Properties / Security / Properties /Configure / Advanced
  • Confirm that the AOVPN connection is working
  • Export the profile using the script from https://directaccess.richardhicks.com/tag/profilexml/
  • Look at the xml – you should now see the thumbprints of both the “old” and “new” root certificate listed in multiple sections
  • Copy the section <EAPHostConfig> from its open xml tag to its close xml tag and insert into the “EAP xml” part of intune AOVPN configuration

Documenting AD ACL’s

Posted on by Hayes Jupe

A while ago i joined an organisation whose MS estate was in need of a significant amount of love, time and effort. Getting them off of 2012 R2 DC’s and onto 2022 DC’s, upgrading the forest/domain functional levels and getting replication times down were the obvious first jobs… but once they were done – there was so many other things to do – it was hard to know what to go with first. So… i made a start on all of it at once – knowing that it would probably take all year to get the AD into a semblance of decent condition.

The more i looked, the more i found… one thing that was/is particularly disturbing is that the DS ACL’s have been fucked with at the top level – and flowed down to all descendant objects for some admin accounts, service accounts etc…. stuff that clearly doesnt need, or has never needed that level of access….

Before changing anything, the goal is to document the permissions – as is a spaghetti of inherited, non-inherited and multi-nested groups applied at many different levels…. resulting in one severe head-fuck for anyone trying to do anything effective with permissions delegation.

First of all i tried

https://powershellisfun.com/2022/08/22/report-on-active-directory-container-permissions-using-powershell/

A decent solution – which works perfectly in my test environment, but in the prod environment with thousands of OU’s and a stupid level of excessive custom permissions, uses approx 4GB of memory before dying consistently. So while this is definitely a good script – it just doesn’t work in this prod environment…. and that’s because of how fucked the environment is, not because the script is bad.

I moved on and found

https://github.com/canix1/ADACLScanner

Which seems to be an exceedingly nice (powershell based) AD ACL solution…. an optional GUI, plenty of configuration options and great output options – a really good solution.

For me – i needed to tick “inherited permissions”… as it is important for me to demonstrate how incredibly stupid (in case you haven’t noticed, I’m still flabbergasted that someone would do this….) it is to allocate permissions at the top level of a domain – along with having complete documentation.

 

Well done & thanks to the author – Robin Granberg – for creating a genuinely awesome tool.

 

Now comes the hard bit – removing the permissions without breaking anything.

Removing folders with a trailing space on NTFS volumes

Posted on by Hayes Jupe

At the moment im cleaning up a very poorly designed and implemented file server structure.

and before you say it – a large amount of data has been moved into teams/sharepoint/onedrive etc already – but the storage costs were getting excessive – so there is still plenty of data on prem.

One of the issues ive run into while cleaning up un-user DFS-R replicas is folders that have spaces at the end of the name, such as “D:\Sales\December ” for example – which NTFS does not support…. but seems to be something Mac users do regularly (for unknown reasons)

These folders cannot be deleted via the GUI.

Open an elevated command prompt and

rmdir /q “\\?\D:\Sales\December “

AADConnect – get Sync’ed and excluded OU’s via powershell

Posted on by Hayes Jupe

AADConnect has a JSON file and the ability to export – and there are also various AADConnect documenters out there… but sometimes you just want to get a core piece of info without having to start the GUI of wade through many pages of JSON.

Get-ADSyncConnector | select Name

Note the name of your “internal” domain as the connector (the one that doesn’t have “AAD” at the end)

(Get-ADSyncConnector -name <ConnectorName>).Partitions.ConnectorPartitionScope.ContainerInclusionList

(Get-ADSyncConnector -name <ConnectorName>).Partitions.ConnectorPartitionScope.ContainerExclusionList

Otherwise healthy DC failing DFS-R, pointing to DC that no longer exists

Posted on by Hayes Jupe

Today i had a DC that was otherwise healthy, but reporting error 4612 and 5012 in the DFS Replication log, specifically:

The DFS Replication service failed to communicate with partner <decommissioned DC name> for replication group Domain System Volume. The partner did not recognize the connection or the replication group configuration.

My first port of call was to open ADSIEdit.msc and check

CN=Topology,CN=Domain System Volume,CN=DFSR-GlobalSettings,CN=System,DC=domain,DC=com,DC=au

but the dead server was not in there.

After some googling, i found a reference to

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\DFSR\Parameters\SysVols\Seeding SysVols\DomainName\Parent Computer

and sure enough – this was referencing the now decommissioned DC – no idea how it happened. The new/old DC’s were both online together for over a month – it should not have been still seeding…. but obviously something went wrong.

Updated the name to a DC that existed, restarted the DFS-R service, waited about 15 seconds – all is now good.

Moving off on HostGator

Posted on by Hayes Jupe

In some truly bizarre circumstances, my previous wordpress host – hostgator, suspended my account a few days after paying for another 3 years.

I suspect it had something to do with their “support” directing me to turn off automatic billing, actually cancelling the account instead of only turning off billing.

While frustrating – it would have been ok…. but the response by Hostgator support was beyond poor.

1) They claimed to have emailed me about the cancellation. Since i use a hotmail account, i can’t see the SMTP logs or spam logs to verify if they actually did or not…. but i can say that all their billing emails reach that account fine – but none of the messages they claim to have sent from support have ever arrived… and their response to this was “check your spam folder”… excellent point… never thought of that. Just insultingly basic.

2) They were unable to give me a reason for the cancellation… as above, i suspect it was an unintended consequence of turning off auto-renew… but surely they would be able to see that.

3) They were seemingly happy to cancel my account, but not refund my money… while repeatedly claiming they have refunded my money – and to “check with my bank” – whatever the fuck that means…. while I’m looking at my online banking and can see the money going out – but no money coming back in.

4) They have emailed me a site backup – which will arrive in 24 hours (and never did)… they will also get someone from their accounts team to email me within 24 hours about the refund (never happened). I managed to eventually get a cpanel backup (rather than the preference of a wordpress export) out of them via the online chat, which, fortunately only had corruption for stuff i didn’t need anymore…. and for the refund i have taken the approach of lodging a dispute via my bank – as its pretty clear to me that its intentional theft on hostgators part.

 

Anyhoo – long story short – i would not recommend hostgator.

Error: SWbemObjectEx: Invalid index when trying to update a NIC using SConfig on server core

Posted on by Hayes Jupe

When using SConfig on a server core install, i was getting the following error

had similar issues when trying to configure the NIC using powershell.

Thanks very much to Mike and his post @ https://mikeconjoice.wordpress.com/2017/01/24/windows-server-core-error-swbemobjectex-invalid-index/

for pointing out that it was because IPv6 was not bound to the adapter.

Using the following powershell worked for me

Enable-NetAdapterBinding -Name Ethernet –ComponentID ms_tcpip6

 

the other important thing here is that unbinding IPv6 from adapters is a relatively common and completely silly practice. It frequently causes issues and doesn’t even achieve the goal of properly disabling IPv6 on the machine.

If you want to disable IPv6 – do it properly – via the registry as per

https://learn.microsoft.com/en-us/troubleshoot/windows-server/networking/configure-ipv6-in-windows

Location: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip6\Parameters\
Name: DisabledComponents
Type: REG_DWORD
Min Value: 0x00 (default value)
Max Value: 0xFF (IPv6 disabled)

Panasonic air con and some stellar support

Posted on by Hayes Jupe

Email chain with panasonic support below…. asking for the totally unreasonable option to setting a timer for auto-off within the “comfort cloud” app. Long version below – short version, Panasonic seem to have some mind-blowingly unhelpful people working for them…. go with a different brand.

My initial request via web form

Support type: ContactUs(Feedback or suggestion)

Product details:

Product type: Air Conditioning and Ventilation

Product category: Split Systems – Wall Mounted

Product model: CS-Z42TKR / CU-Z42TKR

We have 2 x CS-Z42XKRW units (didn’t appear to be listed in the model drop-down) … and we would like to be able to set a timer within the comfort cloud app – but we cannot find this setting. Just to be clear, we don’t want a weekly schedule… we want to be able to turn the unit on – and then say “auto-turn off in 2 hours”… our existing panasonic ducted unit can do it (which granted is not controlled by comfort cloud) – but these new units don’t seem to be able to do it. What are we missing ?

Their 1st response ignoring i specifically mentioned i was not after a weekly schedule

Dear Hayes,

Thank you for your email.

As per the attached user manual page 16, the application only has a weekly for timer.

My attempt to try and get something… anything out of them.

So… no plans to add this? Anywhere I can request it ? Anything that actually helps at all ?

The Microsoft-support-level-bad response

Hi Hayes,

The comfort cloud app gives you the opportunity to switch your air conditioner off and on at you leisure from anywhere, anytime. It does not have a 2hr timer to automatically turn off your system.

The suggestion is that you set an alarm on your phone to prompt yourself to turn off the air conditioning system from your phone after 2 hrs. if that is your requirement.

So… there you have it…. there is no way to request a multi-billion $ company to add a basic function to their app….. and the response just indicates that the product is, like many things currently, effectively unsupported.

Issues with mailbox migration, multiple identities where tenant wide retention policies in use

Posted on by Hayes Jupe

This is a somewhat niche issue – which is why im documententing it here.

Scenario

  • Exchange migration to exchange online which i came into approx 75% of the way through – so i don’t have any history on why some things have happened (and there is no useful doco)
  • Tenant wide retention policies are in place for all data (legislative requirement im led to believe for this client)
  • Identity sync via AADConnect
  • Some mailboxes cannot be moved. Powershell error message from new-moverequest indicates that the identity is not unique

Investigation

  • Start off by looking at the AAD Object sync with
    • Connect-MSOLService
    • (Get-MsolUser -UserPrincipalName identity@goes.here.com).errors.errordetail.objecterrors.errorrecord| fl ErrorCode
    • The output, will likely look something like this:
      • The value “<guid-value>” of property “ArchiveGuid” is used by another recipient object. Please specify a unique value.
  • Next up, we want to have a look at the potential duplicate objects
    • Connect-ExchangeOnline
    • Get-recipient -identity <identity> -includesoftdeletedrecipients
      • This will likely show you 2 (or more) mail user objects
    • To confirm the soft-deleted mailuser object you can use
      • Get-MailUser -resultsize unlimited -SoftDeletedMailUser -Identity <identity> | fl *guid*
      • Notice the ArchiveGUID returned is the same as the ArchiveGUID from the Get-MSOLuser error retrieved earlier in the investigation
    • If you then try and run the obvious next step
      • Get-MailUser -Identity <identity> -SoftDeletedMailUser | Remove-MailUser
      • You will get an error similar to
        • Remove-MailUser: The operation couldn’t be performed because object ‘Soft Delete d Objects\<identity>’ couldn’t be found on ‘SYBPR01A004DC01.AUSPR01A004 .PROD.OUTLOOK.COM’

Now, i know what your thinking “just exclude the mailbox from the retention policy” – and there within lies the issue…. there is no mailbox, only a mail user object, but with an archive mailbox that has been retained by the retention policy after the primary mailbox has been removed. It is then, to my knowledge, impossible to exclude that archive mailbox from retention – as its associated with a mailuser – not a mailbox.

As to how these identities got into this state…. absolutely no idea. I wasn’t around for the earlier parts of the project – but given some other things i’ve seen at the client, standardisation and documentation appear to be frowned upon (which is why i’m getting out ASAP)

 

Solution

The unfortunate solution is to log a call with O365 support.

I included all of the above information in my original support request and was still asked to run a “get-mailbox”… i included all the info again (and again, and again over a teams call showed them the exact same errors and data that i sent them) – and eventually they got the point (took approx 15 business days) and sent it to an internal team, who deleted the objects

Unfortunately i cant post the case number for reference (as it would potentially identify the client) – but maybe pointing MS support to this article might speed the process for others (?). Ideally, there would be a way around this, without engaging support – but there is not as far as I’m aware as of June 2023.

Issue with manually created EXO inbound connector in hybrid environment

Posted on by Hayes Jupe

Working at a client whom are approx 75% of the way through their migration to exchange online – and there are some odd things im running into – so here’s one of them.

The scenario and issue

  • Exchange hybrid setup, with servers on prem and EXO active. Active mailboxes in both.
  • Mail flow from on prem to EXO shows the following:
    • Outbound SMTP logs shows the message being handed off correctly to EXO
    • Message tracking in EXO shows 3 copies of the message, all of which, when looking into the details are bounces
    • When looking in security.microsoft.com, the messages have been flagged as phishing attempts… with seemingly no way to flag them as not phishing attempts
  • The connectors on-prem looked ok, and after, double, triple and ninieteenth-thousandth checking, they were solid
  • The connectors in EXO were manually created (for reasons i don’t know that pre-date me) and the HCW created connectors had been disabled. No idea why.
  • The connectors in EXO looked fine and validated without any issue
  • After circling around for ages, i compared the disabled HCW connector with the active connect with “get-inboundconnector | fl”
  • This is when i noticed that the HCW created connector had IP’s in the “EFSkipIPs” property

The Fix

  • EFSkipIPs can be configured as per the powershell doco here
  • The EFSkipIPs property looks like it defines IP’s that should be excluded from enhanced filtering. Since the HCW automatically populates this field – most of us will never have to use this…. but if some bright spark decides that the HCW isn’t good enough for them (for whatever reason), then this becomes important.
  • Because i had the previous, disabled connector, created by the HCW – i already knew the IP’s i needed to add.  If you don’t have this, you will need get your the Public IP that is presented to EXO. This could be obtained with something such as www.whatsmyip.com
  • The multi-valued property… well, it would have been nice on the doco page if an example was included… so since there isn’t one in the official doc – here is an example below:

Set-inboundConnector -Identity “OrgToEXO” -EFSkipIPs @{Add=”xx.xx.xx.xx”, “xy.xy.xy.xy”}

  • After that, i needed to wait approx 15 minutes (not sure on the exact time, but it didn’t work straight away) – and bingo-bango – no more mail flow issue