Removing your last exchange on-premise servers after migrating to exchange online

One question that comes up constantly from clients is “Once we move all mailboxes to exchange online, we can remove all our exchange servers, right ?”

 

To which we answer “no – for these reasons….” but its something that has a great deal of trouble sticking…. as the same question gets asked over and over again… sometimes by the same client, sometimes even by the same people.

I stumbled across this today – an article written a few months back which nicely lays out the possible scenario’s

https://docs.microsoft.com/en-us/exchange/decommission-on-premises-exchange

well done to the authors for laying it out plainly!

We don’t really work with any places that will be in scenario 1 anytime soon… but scenario 2 and 3 are spot on… with 3 being the most common scenario for our clients.

Adding F8 command prompt to legacy boot wims

Adding and managing legacy boot wims into SCCM is a pain….. the removal of GUI options for non-current versions makes things more painful than they need to be…. and the instances of newer versions of ADK (such as ADK 2004) being incompatible with certain things, seem to be increasing.

Adding command prompt to a legacy boot wim is substantially less than intuitive – so im leaving this here as a quick reference for myself!

WBEMTEST

connect to \root\sms\site_<sitecode>

Open instance

sms_bootimagepackage.packageID=”<boot image package ID>” (quotes are required)

Set “EnableLabShell” to TRUE, save property, then save object

Re-open the boot image and verify the property is set

SCCM reports show “The DefaultValue expression for the report parameter ‘UserTokenSIDS’ contains an error: The user name or password is incorrect. (rsRuntimeErrorInExpression)”

Had a rare fresh SCCM install recently with a client that has more of a security focus than many of our other clients.

Had SCCM 2002 (current at time of writing), Server 2019, SQL 2017 and Reporting services 2017.

Reporting services is using the default, “virtual service account”, which, under the covers, is “NT Service\SQLReportingServices”

 

After everything was installed, i tried to run a report and was greeted with

The DefaultValue expression for the report parameter ‘UserTokenSIDS’ contains an error: The user name or password is incorrect. (rsRuntimeErrorInExpression)

 

A bit of research showed that

  • The account logged on successfully when the reporting services service started and the security event log showed a successful logon – so i knew the username/password was correct
  • When trying to run a report, the security log showed event ID 4625 with the account “NT Service\SQLReportingServices” and a status and sub-status of 0xC000006D and 0xC0000064 respectively. These can be looked up at https://docs.microsoft.com/en-us/windows/security/threat-protection/auditing/event-4625
  • C:\Windows\ServiceProfiles\SQLServerReportingServices\AppData\Local\Temp\SCCMReporting.log showed “The username or password is incorrect”

Given that i knew the username and password was correct – i was a bit baffled by this for a while…. i spent a bit of time looking for how the auth is passed between the user session and SSRS. Googling presented a couple of suggestions, but they were the type of “answers” that just didn’t look like they fit the issue.

Fortunately, one of the other guys on a related project suggested adding the SCCM computer account to “Windows Authorization Access Group” – and it then proceeded to work.

Once finding that out, i searched and quickly came back with https://www.enhansoft.com/windows-authorization-access-group-ssrs-and-sccm/  which shows its been an issue for a while – one i just hadn’t run into before – and apparently my google searches were a bit too specific – and the error message others were getting was slightly different.

AADConnect – Proxy Address in conflict

Had an interesting one recently with a customer that has created cloud accounts for use during COVID-19 with approx 50 users. Each of these accounts were assigned a license and the users used teams, onenote, onedrive etc…. but not exchange online mailbox – as they already have an on-premise mailbox.

After this, we became involved and implemented AADConnect and exchange hybrid.

After removing the exchange license and clearing the mailbox info as per https://techcommunity.microsoft.com/t5/exchange-team-blog/permanently-clear-previous-mailbox-info/ba-p/607619

some users, not all, were not sync’ing correctly in AAD connect.

we can see the objectID that is conflicting

We can then confirm which object it is in O365

and then search for the conflict – which in this case, was a proxy address.

 

i had previously thought that ProxyAddresses was an exchange related attribute….. and given i had already removed all exchange information from the account… it shouldn’t be there….

At this stage the battle became how to get rid of it – since set-MSOLuser does not seem to allow it.

I found this – https://byronwright.blogspot.com/2017/09/remove-proxy-address-from-office-365.html – but in my instance, the conflicting address is the “correct” address, which keeps getting re-added automatically if you remove it.

Renaming the account temporarily doesn’t work – as the proxy address (as a user account attribute for an unknown reason) seems to update to match the UPN when changed.

 

This article had a good suggestion – https://www.petenetlive.com/KB/Article/0001588 but resulted in an error

Failed to apply fix – User with conflicting attribute is soft deleted in Azure Active Directory. Ensure the user is hard deleted before retrying.<a href=”https://go.microsoft.com/fwlink/?linkid=2007018″ target=”_blank”>Read More</a>

and we cant hard delete the user – they have data…

 

So – what are we left with? Hard matching. A process i was aware of, had never had to do (until now) – but thought was going to be overkill for this situation….. but apparently i was wrong.

There is a script here to help – https://gallery.technet.microsoft.com/office/immutableid-hard-match-in-d3518b08 or, if, like me, you prefer to step through it (when there is a small number of affected users)

import-module ActiveDirectory

$user = “SameAccountName” (not UPN)

$guid = [guid]((Get-ADUser -Identity $user).objectGuid)

$immutableId = [System.Convert]::ToBase64String($guid.ToByteArray())

Set-MsolUser -UserPrincipalName <UPN> -ImmutableId <Base64String>

 

this should work…. but didnt, instead i got the error

Set-MsolUser : Uniqueness violation. Property: SourceAnchor.
At line:1 char:1
+ Set-MsolUser -UserPrincipalName benjamin.primus@rtwsa.com -ImmutableI …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (:) [Set-MsolUser], MicrosoftOnlineException
+ FullyQualifiedErrorId : Microsoft.Online.Administration.Automation.UniquenessValidationException,Microsoft.Online.Adminis
tration.Automation.SetUser

 

Then it clicked – this guy must have had an account previously…. run

get-msoluser -all -returndeletedusers

and sure enough – there it was….. with the immutableID assigned to that account…. so i then ran

Get-MsolUser -ReturnDeletedUsers -UserPrincipalName <UPN> | Remove-MsolUser -RemoveFromRecycleBin -Force

Followed by setting the immutableID again – which went through this time.

I then waited for AADConnect to sync again – which went through without error…. “yay” – i thought…. but then the user no longer showed up in the O365 GUI, and powershell showed

Errors : {Microsoft.Online.Administration.ValidationError,
Microsoft.Online.Administration.ValidationError,
Microsoft.Online.Administration.ValidationError,
Microsoft.Online.Administration.ValidationError…}

i used https://support.microsoft.com/en-us/help/2741233/you-see-validation-errors-for-users-in-the-office-365-portal-or-in-the

to troubleshoot this, in particular

$errors = (Get-MsolUser -UserPrincipalName “<User_ID>”).Errors

Get-MsolUser -HasErrorsOnly | select DisplayName,UserPrincipalName,@{Name=”Error”;Expression={($_.errors[0].ErrorDetail.objecterrors.errorrecord.ErrorDescription)}} | Export-csv c:\temp\validationerrors.csv

In this users case, there was still something hanging around from an old mailbox

The execution of cmdlet Set-SyncMailUser failed. The value “8dceb397-5af1-4142-8689-24bffeb83f92” of property “ArchiveGuid” is used by another recipient object. Please specify a unique value.

 

This lead me to https://docs.microsoft.com/en-us/archive/blogs/exovoice/how-to-fix-office365-user-provisioning-issues-that-are-generated-by-faulty-exchange-attributes

Run this to get the conflicting GUID

(Get-MsolUser -UserPrincipalName affecteduser@domain.com).errors.errordetail.objecterrors.errorrecord| fl

Then run this – with the inserted GUID from the above step

Get-Recipient -IncludeSoftDeletedRecipients ‘ExchangeGUID value’|ft RecipientType,PrimarySmtpAddress,*WhenSoftDeleted*

in my case – it was a soft deleted mailbox, so i then run

Remove-MailUser ‘ExchangeGUID value’ -PermanentlyDelete

After this – it was finally all good on next AADConnect Sync.

This whole thing was one huge pain in the arse…. but, at the same time, i learnt a lot…. I’ve previously always setup AADConnect first – and therefore not had cloud accounts that need to be merged etc. The whole process is way fucking harder than it needs to be…. which is very Microsoft…. but, i feel like this will come in useful in future!

Powershell fails to download from gallery – Wanring: Unable to download from URI xxx

Recently had this issue…. i was able to get to the specified URI via the browser with no issue.

Came across this post – https://techcommunity.microsoft.com/t5/windows-powershell/failed-downloading-az-and-other-modules-for-powershell/m-p/1292985 and this command fixed it for me

[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

Apparently support for older protocols was dropped in April 2020

While i agree with the move – its just a bit frustrating that, by the nature of the change, the site cant communicate the new requirement back.

SCCM – picking random clients for a collection

There are plenty of ways to randomly add systems to a collection in SCCM, a common query being something similar to

select SMS_R_System.Name from  SMS_R_System where SMS_R_System.SMSUniqueIdentifier like “%0” OR SMS_R_System.SMSUniqueIdentifier like “%1”

 

In a recent instance, i needed to ensure that systems were randomly chosen for a product roll out across a 1500-ish servers. The client was specific about staging the deployment and wanted to selection process of what went when to be truly random… fair enough.

SCCM cant really do do random when limited to a number…. the query above for example is likely to retrieve approx 20% of your total client count…. where-as i needed a random 50.

I ended up using the following powershell to accomplish the task.

 

Function Connect-ConfigMgr {
Param(
)
write-host -ForegroundColor Magenta “Connect-ConfigMgr”
Try {
Get-WMIObject -ComputerName $CMSiteServer -Namespace “root\SMS” -Class “SMS_ProviderLocation” -ErrorAction Stop | foreach-object{
if ($_.ProviderForLocalSite -eq $true){$Script:SiteCode=$_.sitecode}
}
if ($SiteCode -eq “”) {
throw (“Sitecode of ConfigMgr Site at ” + $CMSiteServer + ” could not be determined.”)
Exit 1
}
}
Catch {
$ErrorMessage = $_.Exception.Message
$text = “Error, could not connect to Site server $CMSiteServer, check spelling, network connectivity, permissions etc.”
$text += “`nError message: $ErrorMessage”
Write-Error $text
Exit 1
}
#$CMNamespace = “Root\SMS\$($SiteCode)”

write-host -ForegroundColor Green “ConfigMgr Site Code: $SiteCode”
write-host -ForegroundColor Green “ConfigMgr Site Server: $CMSiteServer”
#write-host “ConfigMgr Name Space: $CMNamespace”

#Import ConfigMgr Module
if (!(Get-Module -Name ConfigurationManager)) {
write-host “PowerShell module not loaded.”
If (Test-Path -Path “$($ENV:SMS_ADMIN_UI_PATH)\..\ConfigurationManager.psd1” -PathType Leaf) {
write-host “Attempting loa PS module from: $($ENV:SMS_ADMIN_UI_PATH)\..\ConfigurationManager.psd1”
Import-Module “$($ENV:SMS_ADMIN_UI_PATH)\..\ConfigurationManager.psd1” | Out-Null
}
ElseIf (Test-Path “$($ScriptPath)\ConfigurationManager.psd1” -PathType Leaf) {
write-host “Attempting loa PS module from: $($ScriptPath)\ConfigurationManager.psd1”
Import-Module “$($ScriptPath)\ConfigurationManager.psd1” | Out-Null
}
Else {
$text = “Error, unable to load PowerShell module ConfigurationManager.psd1, file not found.”
Write-Error $text
Exit 1
}
}
if (!(Get-Module -Name ConfigurationManager)) {
$text = “Error loading PowerShell module ConfigurationManager.psd1.”
Write-Error $text
Exit 1
}

#Set the current location to be the site code.
Set-Location “$SiteCode`:”
write-host -ForegroundColor Green “————————————————————————”
write-host “”
}### End Connect-ConfigMgr ###

Function GetRandomCollectionMembers {
Param(
)
write-host -ForegroundColor Magenta “Getting collection members”
$CollMembers = Get-CMCollectionMember -CollectionName $SourcecollectionName

#Randomising and selecting first x members
write-host -ForegroundColor Magenta “Selecting $NumberofMembers members randomly”
$CollMembers = $CollMembers | Sort-Object {Get-Random}
$CollMembers = $CollMembers | Select -First $NumberofMembers

Foreach ($CollMember in $CollMembers) {
$CollectionName = $CollMember.Name
write-host -ForegroundColor Green “Adding $CollectionName to $DestinationCollectionName”
Get-CMCollection -Name $DestinationCollectionName | Add-CMDeviceCollectionDirectMembershipRule -ResourceId $CollMember.ResourceID
}

}

##############Start of Main Script ####################

#Begin {

## Get script path and name
$ScriptPath = [System.IO.Path]::GetDirectoryName($MyInvocation.MyCommand.Definition)
$ScriptName = [System.IO.Path]::GetFileNameWithoutExtension($MyInvocation.MyCommand.Definition)
$PowerShellVersion = [int]$PSVersionTable.PSVersion.Major

#Save current path, to return after CM operations
$path = Split-Path -parent $MyInvocation.MyCommand.Definition

#Set variables here
$CMSiteServer = ‘sccm01.adexis.com.au’
$Script:SiteCode = $null
$SourcecollectionName = “All Systems”
$DestinationCollectionName = “RandomSystems”
$NumberofMembers = “5”

Connect-ConfigMgr
GetRandomCollectionMembers

#} #End Begin

Monitor flicker with AMD Radeon RX590

A month or two back, i grabbed a new media/VR PC – and had a great deal of flicker on the screen. After some googling, i found that “Virtual super resolution” seemed to be a common cause – and disabling this stopped the flicker.

Fast forward to today – and a monitor on my main work PC was flickering once every 5-10 seconds. I had recently swapped over my 2nd monitor from DVI to DP – but there was a 4 day delay between that and when the flicker started occurring. I had also updated the monitor driver for the primary display.

After some looking at the settings – i found that “virtual super resolution” had been enabled on the HDMI monitor – and given previous experience, i turned this off…. and voila – no flicker. Since my primary monitor is already a pretty good res – i have no need to “render at resolution higher than the displays native pixel grid”.

There didn’t appear to be a driver update between the time it wasn’t and was happening…. so my only guess is that updating the monitor driver caused the AMD settings software to re-evaluate and turn this setting on – which is frustratingly unnecessary.

Still – its a somewhat obscure setting – so – this post is to remind me for next time and help anyone else that might run into the same issue!

O365 ATP recommended config analyser

Came across this after following a post on linked in – checked it out – and its pretty bloody good.

Run the tool – it will spit out a report with recommendations for O365 Advanced threat protection…

The recommendations it spits out are of a pretty good quality

https://www.linkedin.com/pulse/reviewing-your-office-atp-configuration-cam-murray/

its not perfect (these things rarely are) – for example, after setting up DKIM and verifying, i find that it still reports an issue with DKIM – but it isn’t clear what the issue is – but it remains a good checklist enabler.

Exchange 2010 to 2016 mail flow stops with “421 4.4 2 connection dropped due to socket error”

Had a client ring today with a mail flow issue.

They are most of the way through their migration to 2016, but mail flow stopped with the error “421 4.4 2 connection dropped due to socket error” on the exchange 2010 side – when trying to relay to exchange 2016. This occurred after patching – but i’m not 100% on which patches they applied – and didn’t really have the time to find out.

 

Long story short – enabled TLS 1.2 on the Exchange 2010 (on a 2008 R2 OS) as per https://support.quovadisglobal.com/kb/a433/how-to-enable-tls-1_2-on-windows-server-2008-r2.aspx

then restarted the transport service – and mail stated flowing again.

While this is known for Exchange 2019 on Server 2019, where TLS 1.2 is the default – i wasn’t aware this was being retro-fitted……  not a bad thing…. and its only going to catch out the people that are lagging behind… still, considering how many people are lagging behind – this quick post might help!

SCCM antimalware client policies – do they merge ?

There seems to be a great deal of mis-information about this floating around the web…. despite articles like this that lay it out quite well.

When you create an SCCM antimalware policy, the settings do merge.

You do not have to create bucketloads of policies and apply/re-apply the same settings over and over and over again.

This insane practice makes anti-malware policies much harder to manage and track.

 

Testing

In order to prove this for yourself, do the following:

  • add a process into the default anti-malware policy – give it an obvious name such as “DefaultPolicyExclusion.exe”
  • Now create a new policy, add an exclusion called “NewPolicyexclusion.exe”
  • Apply the new policy to a machine and update policy on that machine
  • Check the exclusion list (settings | Updates and Security | windows security | Virus and threat protection | manage settings | Add or remove exclusions)
  • You can now see that both exclusions are listed

 

You can also run the following from a command prompt to see which policies are applied

reg query HKLM\SOFTWARE\Microsoft\CCM\EPAgent\LastAppliedPolicy /f 2 /d

Notice that the exclusions are applied from both “Default client antimalware policy” and whatever you called your new policy.

 

Why this matters

We have clients that (correctly) create new antimalware policies for different types of servers…. e.g. to exclude sqlservr.exe on SQL servers – which makes complete sense.

Unfortunately, they then proceed to define every other setting that is available….. which means that if we want to change real time settings (for example) – we need to go into every single policy to do so. A little painful but not to base with 10 policies, but just unnecessarily time consuming when there are hundreds.

 

What about conflicting settings ?

This is where priority comes into play.

Its very simple… in the even of a conflicting setting (e.g. Real time scanning enabled in one policy and disabled in another) then the one with the lowest priority will win. This is referred to as “Order” in some parts of the console and Priority in other parts of the console. (As of SCCM 1910)

Notice that the default policy has a priority of 10000, which is not changeable – and therefore will always be the lowest priority settings.

 

A better way

Just like your SCCM client policies, the best way to handle anti-malware policies is to set the defaults and then only make additional policies where you need to deviate from those defaults.

Set all the default settings for all machines across the fleet in the “Default Client antimalware Policy”, then create additional policies where required…. but only set the specific settings that deviate from the default settings – not every setting!

 

Another approach, for slightly larger organisations is to:

  • Leave the default policy alone
  • Create a Default workstation policy – which the desktop guys control
  • Create a Default server policy – which the server guys control
  • And so on (for different area’s of the business if there is a management demarcation)
  • Deploy appropriately

The downside of this model is that the default policy will keep needing to moved to the lowest priority each time you create a new policy.

 

What should i be excluding ?

https://social.technet.microsoft.com/wiki/contents/articles/953.microsoft-anti-virus-exclusion-list.aspx

 

A practical example

In this example were going to focus on an organisation with x00 servers, 2 of which are:

  • SCCM with SQL co-loctaed
  • Payroll application server with SQL co-located

 

In this case we would

 

Apply the SQL server policy to anything running SQL

Apply the SCCM server policy to anything running SCCM

Apply the payroll server policy to the payroll server

 

This way, when we update the SQL server policy – it flows through to all SQL servers automatically. We dont have to update the SCCM policy and the payroll server policy.