AADConnect – Proxy Address in conflict

Had an interesting one recently with a customer that has created cloud accounts for use during COVID-19 with approx 50 users. Each of these accounts were assigned a license and the users used teams, onenote, onedrive etc…. but not exchange online mailbox – as they already have an on-premise mailbox.

After this, we became involved and implemented AADConnect and exchange hybrid.

After removing the exchange license and clearing the mailbox info as per https://techcommunity.microsoft.com/t5/exchange-team-blog/permanently-clear-previous-mailbox-info/ba-p/607619

some users, not all, were not sync’ing correctly in AAD connect.

we can see the objectID that is conflicting

We can then confirm which object it is in O365

and then search for the conflict – which in this case, was a proxy address.

 

i had previously thought that ProxyAddresses was an exchange related attribute….. and given i had already removed all exchange information from the account… it shouldn’t be there….

At this stage the battle became how to get rid of it – since set-MSOLuser does not seem to allow it.

I found this – https://byronwright.blogspot.com/2017/09/remove-proxy-address-from-office-365.html – but in my instance, the conflicting address is the “correct” address, which keeps getting re-added automatically if you remove it.

Renaming the account temporarily doesn’t work – as the proxy address (as a user account attribute for an unknown reason) seems to update to match the UPN when changed.

 

This article had a good suggestion – https://www.petenetlive.com/KB/Article/0001588 but resulted in an error

Failed to apply fix – User with conflicting attribute is soft deleted in Azure Active Directory. Ensure the user is hard deleted before retrying.<a href=”https://go.microsoft.com/fwlink/?linkid=2007018″ target=”_blank”>Read More</a>

and we cant hard delete the user – they have data…

 

So – what are we left with? Hard matching. A process i was aware of, had never had to do (until now) – but thought was going to be overkill for this situation….. but apparently i was wrong.

There is a script here to help – https://gallery.technet.microsoft.com/office/immutableid-hard-match-in-d3518b08 or, if, like me, you prefer to step through it (when there is a small number of affected users)

import-module ActiveDirectory

$user = “SameAccountName” (not UPN)

$guid = [guid]((Get-ADUser -Identity $user).objectGuid)

$immutableId = [System.Convert]::ToBase64String($guid.ToByteArray())

Set-MsolUser -UserPrincipalName <UPN> -ImmutableId <Base64String>

 

this should work…. but didnt, instead i got the error

Set-MsolUser : Uniqueness violation. Property: SourceAnchor.
At line:1 char:1
+ Set-MsolUser -UserPrincipalName benjamin.primus@rtwsa.com -ImmutableI …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (:) [Set-MsolUser], MicrosoftOnlineException
+ FullyQualifiedErrorId : Microsoft.Online.Administration.Automation.UniquenessValidationException,Microsoft.Online.Adminis
tration.Automation.SetUser

 

Then it clicked – this guy must have had an account previously…. run

get-msoluser -all -returndeletedusers

and sure enough – there it was….. with the immutableID assigned to that account…. so i then ran

Get-MsolUser -ReturnDeletedUsers -UserPrincipalName <UPN> | Remove-MsolUser -RemoveFromRecycleBin -Force

Followed by setting the immutableID again – which went through this time.

I then waited for AADConnect to sync again – which went through without error…. “yay” – i thought…. but then the user no longer showed up in the O365 GUI, and powershell showed

Errors : {Microsoft.Online.Administration.ValidationError,
Microsoft.Online.Administration.ValidationError,
Microsoft.Online.Administration.ValidationError,
Microsoft.Online.Administration.ValidationError…}

i used https://support.microsoft.com/en-us/help/2741233/you-see-validation-errors-for-users-in-the-office-365-portal-or-in-the

to troubleshoot this, in particular

$errors = (Get-MsolUser -UserPrincipalName “<User_ID>”).Errors

Get-MsolUser -HasErrorsOnly | select DisplayName,UserPrincipalName,@{Name=”Error”;Expression={($_.errors[0].ErrorDetail.objecterrors.errorrecord.ErrorDescription)}} | Export-csv c:\temp\validationerrors.csv

In this users case, there was still something hanging around from an old mailbox

The execution of cmdlet Set-SyncMailUser failed. The value “8dceb397-5af1-4142-8689-24bffeb83f92” of property “ArchiveGuid” is used by another recipient object. Please specify a unique value.

 

This lead me to https://docs.microsoft.com/en-us/archive/blogs/exovoice/how-to-fix-office365-user-provisioning-issues-that-are-generated-by-faulty-exchange-attributes

Run this to get the conflicting GUID

(Get-MsolUser -UserPrincipalName affecteduser@domain.com).errors.errordetail.objecterrors.errorrecord| fl

Then run this – with the inserted GUID from the above step

Get-Recipient -IncludeSoftDeletedRecipients ‘ExchangeGUID value’|ft RecipientType,PrimarySmtpAddress,*WhenSoftDeleted*

in my case – it was a soft deleted mailbox, so i then run

Remove-MailUser ‘ExchangeGUID value’ -PermanentlyDelete

After this – it was finally all good on next AADConnect Sync.

This whole thing was one huge pain in the arse…. but, at the same time, i learnt a lot…. I’ve previously always setup AADConnect first – and therefore not had cloud accounts that need to be merged etc. The whole process is way fucking harder than it needs to be…. which is very Microsoft…. but, i feel like this will come in useful in future!

Powershell fails to download from gallery – Wanring: Unable to download from URI xxx

Recently had this issue…. i was able to get to the specified URI via the browser with no issue.

Came across this post – https://techcommunity.microsoft.com/t5/windows-powershell/failed-downloading-az-and-other-modules-for-powershell/m-p/1292985 and this command fixed it for me

[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

Apparently support for older protocols was dropped in April 2020

While i agree with the move – its just a bit frustrating that, by the nature of the change, the site cant communicate the new requirement back.

SCCM – picking random clients for a collection

There are plenty of ways to randomly add systems to a collection in SCCM, a common query being something similar to

select SMS_R_System.Name from  SMS_R_System where SMS_R_System.SMSUniqueIdentifier like “%0” OR SMS_R_System.SMSUniqueIdentifier like “%1”

 

In a recent instance, i needed to ensure that systems were randomly chosen for a product roll out across a 1500-ish servers. The client was specific about staging the deployment and wanted to selection process of what went when to be truly random… fair enough.

SCCM cant really do do random when limited to a number…. the query above for example is likely to retrieve approx 20% of your total client count…. where-as i needed a random 50.

I ended up using the following powershell to accomplish the task.

 

Function Connect-ConfigMgr {
Param(
)
write-host -ForegroundColor Magenta “Connect-ConfigMgr”
Try {
Get-WMIObject -ComputerName $CMSiteServer -Namespace “root\SMS” -Class “SMS_ProviderLocation” -ErrorAction Stop | foreach-object{
if ($_.ProviderForLocalSite -eq $true){$Script:SiteCode=$_.sitecode}
}
if ($SiteCode -eq “”) {
throw (“Sitecode of ConfigMgr Site at ” + $CMSiteServer + ” could not be determined.”)
Exit 1
}
}
Catch {
$ErrorMessage = $_.Exception.Message
$text = “Error, could not connect to Site server $CMSiteServer, check spelling, network connectivity, permissions etc.”
$text += “`nError message: $ErrorMessage”
Write-Error $text
Exit 1
}
#$CMNamespace = “Root\SMS\$($SiteCode)”

write-host -ForegroundColor Green “ConfigMgr Site Code: $SiteCode”
write-host -ForegroundColor Green “ConfigMgr Site Server: $CMSiteServer”
#write-host “ConfigMgr Name Space: $CMNamespace”

#Import ConfigMgr Module
if (!(Get-Module -Name ConfigurationManager)) {
write-host “PowerShell module not loaded.”
If (Test-Path -Path “$($ENV:SMS_ADMIN_UI_PATH)\..\ConfigurationManager.psd1” -PathType Leaf) {
write-host “Attempting loa PS module from: $($ENV:SMS_ADMIN_UI_PATH)\..\ConfigurationManager.psd1”
Import-Module “$($ENV:SMS_ADMIN_UI_PATH)\..\ConfigurationManager.psd1” | Out-Null
}
ElseIf (Test-Path “$($ScriptPath)\ConfigurationManager.psd1” -PathType Leaf) {
write-host “Attempting loa PS module from: $($ScriptPath)\ConfigurationManager.psd1”
Import-Module “$($ScriptPath)\ConfigurationManager.psd1” | Out-Null
}
Else {
$text = “Error, unable to load PowerShell module ConfigurationManager.psd1, file not found.”
Write-Error $text
Exit 1
}
}
if (!(Get-Module -Name ConfigurationManager)) {
$text = “Error loading PowerShell module ConfigurationManager.psd1.”
Write-Error $text
Exit 1
}

#Set the current location to be the site code.
Set-Location “$SiteCode`:”
write-host -ForegroundColor Green “————————————————————————”
write-host “”
}### End Connect-ConfigMgr ###

Function GetRandomCollectionMembers {
Param(
)
write-host -ForegroundColor Magenta “Getting collection members”
$CollMembers = Get-CMCollectionMember -CollectionName $SourcecollectionName

#Randomising and selecting first x members
write-host -ForegroundColor Magenta “Selecting $NumberofMembers members randomly”
$CollMembers = $CollMembers | Sort-Object {Get-Random}
$CollMembers = $CollMembers | Select -First $NumberofMembers

Foreach ($CollMember in $CollMembers) {
$CollectionName = $CollMember.Name
write-host -ForegroundColor Green “Adding $CollectionName to $DestinationCollectionName”
Get-CMCollection -Name $DestinationCollectionName | Add-CMDeviceCollectionDirectMembershipRule -ResourceId $CollMember.ResourceID
}

}

##############Start of Main Script ####################

#Begin {

## Get script path and name
$ScriptPath = [System.IO.Path]::GetDirectoryName($MyInvocation.MyCommand.Definition)
$ScriptName = [System.IO.Path]::GetFileNameWithoutExtension($MyInvocation.MyCommand.Definition)
$PowerShellVersion = [int]$PSVersionTable.PSVersion.Major

#Save current path, to return after CM operations
$path = Split-Path -parent $MyInvocation.MyCommand.Definition

#Set variables here
$CMSiteServer = ‘sccm01.adexis.com.au’
$Script:SiteCode = $null
$SourcecollectionName = “All Systems”
$DestinationCollectionName = “RandomSystems”
$NumberofMembers = “5”

Connect-ConfigMgr
GetRandomCollectionMembers

#} #End Begin

Monitor flicker with AMD Radeon RX590

A month or two back, i grabbed a new media/VR PC – and had a great deal of flicker on the screen. After some googling, i found that “Virtual super resolution” seemed to be a common cause – and disabling this stopped the flicker.

Fast forward to today – and a monitor on my main work PC was flickering once every 5-10 seconds. I had recently swapped over my 2nd monitor from DVI to DP – but there was a 4 day delay between that and when the flicker started occurring. I had also updated the monitor driver for the primary display.

After some looking at the settings – i found that “virtual super resolution” had been enabled on the HDMI monitor – and given previous experience, i turned this off…. and voila – no flicker. Since my primary monitor is already a pretty good res – i have no need to “render at resolution higher than the displays native pixel grid”.

There didn’t appear to be a driver update between the time it wasn’t and was happening…. so my only guess is that updating the monitor driver caused the AMD settings software to re-evaluate and turn this setting on – which is frustratingly unnecessary.

Still – its a somewhat obscure setting – so – this post is to remind me for next time and help anyone else that might run into the same issue!

O365 ATP recommended config analyser

Came across this after following a post on linked in – checked it out – and its pretty bloody good.

Run the tool – it will spit out a report with recommendations for O365 Advanced threat protection…

The recommendations it spits out are of a pretty good quality

https://www.linkedin.com/pulse/reviewing-your-office-atp-configuration-cam-murray/

its not perfect (these things rarely are) – for example, after setting up DKIM and verifying, i find that it still reports an issue with DKIM – but it isn’t clear what the issue is – but it remains a good checklist enabler.

Exchange 2010 to 2016 mail flow stops with “421 4.4 2 connection dropped due to socket error”

Had a client ring today with a mail flow issue.

They are most of the way through their migration to 2016, but mail flow stopped with the error “421 4.4 2 connection dropped due to socket error” on the exchange 2010 side – when trying to relay to exchange 2016. This occurred after patching – but i’m not 100% on which patches they applied – and didn’t really have the time to find out.

 

Long story short – enabled TLS 1.2 on the Exchange 2010 (on a 2008 R2 OS) as per https://support.quovadisglobal.com/kb/a433/how-to-enable-tls-1_2-on-windows-server-2008-r2.aspx

then restarted the transport service – and mail stated flowing again.

While this is known for Exchange 2019 on Server 2019, where TLS 1.2 is the default – i wasn’t aware this was being retro-fitted……  not a bad thing…. and its only going to catch out the people that are lagging behind… still, considering how many people are lagging behind – this quick post might help!

SCCM antimalware client policies – do they merge ?

There seems to be a great deal of mis-information about this floating around the web…. despite articles like this that lay it out quite well.

When you create an SCCM antimalware policy, the settings do merge.

You do not have to create bucketloads of policies and apply/re-apply the same settings over and over and over again.

This insane practice makes anti-malware policies much harder to manage and track.

 

Testing

In order to prove this for yourself, do the following:

  • add a process into the default anti-malware policy – give it an obvious name such as “DefaultPolicyExclusion.exe”
  • Now create a new policy, add an exclusion called “NewPolicyexclusion.exe”
  • Apply the new policy to a machine and update policy on that machine
  • Check the exclusion list (settings | Updates and Security | windows security | Virus and threat protection | manage settings | Add or remove exclusions)
  • You can now see that both exclusions are listed

 

You can also run the following from a command prompt to see which policies are applied

reg query HKLM\SOFTWARE\Microsoft\CCM\EPAgent\LastAppliedPolicy /f 2 /d

Notice that the exclusions are applied from both “Default client antimalware policy” and whatever you called your new policy.

 

Why this matters

We have clients that (correctly) create new antimalware policies for different types of servers…. e.g. to exclude sqlservr.exe on SQL servers – which makes complete sense.

Unfortunately, they then proceed to define every other setting that is available….. which means that if we want to change real time settings (for example) – we need to go into every single policy to do so. A little painful but not to base with 10 policies, but just unnecessarily time consuming when there are hundreds.

 

What about conflicting settings ?

This is where priority comes into play.

Its very simple… in the even of a conflicting setting (e.g. Real time scanning enabled in one policy and disabled in another) then the one with the lowest priority will win. This is referred to as “Order” in some parts of the console and Priority in other parts of the console. (As of SCCM 1910)

Notice that the default policy has a priority of 10000, which is not changeable – and therefore will always be the lowest priority settings.

 

A better way

Just like your SCCM client policies, the best way to handle anti-malware policies is to set the defaults and then only make additional policies where you need to deviate from those defaults.

Set all the default settings for all machines across the fleet in the “Default Client antimalware Policy”, then create additional policies where required…. but only set the specific settings that deviate from the default settings – not every setting!

 

Another approach, for slightly larger organisations is to:

  • Leave the default policy alone
  • Create a Default workstation policy – which the desktop guys control
  • Create a Default server policy – which the server guys control
  • And so on (for different area’s of the business if there is a management demarcation)
  • Deploy appropriately

The downside of this model is that the default policy will keep needing to moved to the lowest priority each time you create a new policy.

 

What should i be excluding ?

https://social.technet.microsoft.com/wiki/contents/articles/953.microsoft-anti-virus-exclusion-list.aspx

 

A practical example

In this example were going to focus on an organisation with x00 servers, 2 of which are:

  • SCCM with SQL co-loctaed
  • Payroll application server with SQL co-located

 

In this case we would

 

Apply the SQL server policy to anything running SQL

Apply the SCCM server policy to anything running SCCM

Apply the payroll server policy to the payroll server

 

This way, when we update the SQL server policy – it flows through to all SQL servers automatically. We dont have to update the SCCM policy and the payroll server policy.

A rare nice thing to say about NBN

The NBN (National Broadband Network) is an Australian nation building project which was announced in 2009 and was meant to be Australia’s savior from our terrible internet services.

Unfortunately, it became a political football instead – and has been fucked more times than all the porn stars in the world put together…. to the point where it is just a well-known running joke for all Aussies – even completely non-technical people are aware of how completely useless this project is.

However – while i, along with most other Aussies, still think the NBN has been a colossal failure – the people they have been sending out to fix the issues, once you actually get them to come on site – have been very good.

Let me be clear – it is still way harder than it needs to be to “convince” NBN there is an issue…. “try a different modem”, “how long is the cable”… all the same backwards, pointless, brain-dead questions that were being asked in the ADSL days. Especially when the issue is always, always, ALWAYS, always at the fucking pillar! (for FTTN)

But…  each time i have had to log a call due to connection drop outs, once i go through the pointless rigmarole and get someone to come out… the person has been really quite helpful… explaining where they found the fault (always at the node / pillar), what they did to fix it (attaching/re-attaching the cables), the tests they run, the speeds they got, the level of noise on the line etc.

Apparently i am on a retrofitted pillar and due to the way they are cabled, every time work is done, i (along with others) have a fair chance of having my copper knocked around…. yay!

so while the NBN as technical solution has been bastardised to all hell for political purposes…. well done to the NBN techs – who im sure are putting up with a lot of pointless sh*t from the type of people that think politicians are important, makes it even more impressive that you are able to deal with this mutated monstrosity of shit technical solution.

Surface Pro X and Pro 7 – SCCM deployment

A few years back – an (ex) employee posted on the Adexis blog about the fun of Surface laptops and deployment in SCCM

Microsoft Surface Laptops and SCCM OS Deployment

Personally, i’ve deployed a lot of Surface Pro 3’s and 4’s…. but not so much after that…. until now.

During the week i was called into a client to find out why their new Surface Pro devices were not PXE booting.

Once i attended site, had a look at the pxe log – and saw unsupported architecture… strange…. both the boot images are distributed and boot roms are correct…

Upon further investigation and google – i found that the devices were Surface Pro X’s which are ARM based and not supported for deployment via SCCM or MDT.

https://docs.microsoft.com/en-us/surface/surface-pro-arm-app-management

the TLDR of the article is that the Surface Pro X is not an enterprise device…. the client managed to get devices replaced with Surface Pro 7’s – and they actually got a refund, as the 7 was slightly less expensive than the X….

 

Onto the surface pro 7’s….

These can be detected in the task sequence via a MWI query of

SELECT * FROM Win32_ComputerSystem WHERE Model = “Surface Pro 7″

Or by using the custom model variable from global conditions….

I prefer having one application with multiple deployments, each with a global condition of model…. this keeps the driver database less cluttered and is, for want of a better term, “neat”.

Issue with this is that the Surface Pro 7 drivers install fine from the MSI, but kill the task sequence…. given the time restraints on this project, i didnt really have time to check… so i extracted the drivers from the MSI, imported them as drivers and allowed auto-apply to do its job.

 

TL;DR version of the whole article

  • Surface Pro X – no you cant deploy it via SCCM and yes it is shitty/confusing naming from MS
  • Surface Pro 7 – installing drivers via MSI will kill your task sequence – extract the drivers and install them as part of auto apply or apply drivers instead

 

Resetting the local admin password on a more locked down server

A client recently had an issue where they had lost the administrator password on their offline root CA.

The well known method of copying cmd.exe over utilman.exe was not working

When trying to reset the administrator password, it appeared to work, but on reboot, the new password was not accepted.

To that end i utilise “net” to add a temporary admin account via

net user <username> /add
net localgroup administrators <username> /add
in addition, the server had been configured to not allow any other user but administrator to show up at the console….. this i hadn’t seen before.
After a bit of poking around, i found that the following registry keys were the culprit.
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Authentication\LogonUI]
“LastLoggedOnProvider”=””
“LastLoggedOnSAMUser”=””
“LastLoggedOnUser”=””
once those keys were set, i could logon as my temporary admin account.