Removing your last exchange on-premise servers after migrating to exchange online

One question that comes up constantly from clients is “Once we move all mailboxes to exchange online, we can remove all our exchange servers, right ?”

 

To which we answer “no – for these reasons….” but its something that has a great deal of trouble sticking…. as the same question gets asked over and over again… sometimes by the same client, sometimes even by the same people.

I stumbled across this today – an article written a few months back which nicely lays out the possible scenario’s

https://docs.microsoft.com/en-us/exchange/decommission-on-premises-exchange

well done to the authors for laying it out plainly!

We don’t really work with any places that will be in scenario 1 anytime soon… but scenario 2 and 3 are spot on… with 3 being the most common scenario for our clients.

Adding F8 command prompt to legacy boot wims

Adding and managing legacy boot wims into SCCM is a pain….. the removal of GUI options for non-current versions makes things more painful than they need to be…. and the instances of newer versions of ADK (such as ADK 2004) being incompatible with certain things, seem to be increasing.

Adding command prompt to a legacy boot wim is substantially less than intuitive – so im leaving this here as a quick reference for myself!

WBEMTEST

connect to \root\sms\site_<sitecode>

Open instance

sms_bootimagepackage.packageID=”<boot image package ID>” (quotes are required)

Set “EnableLabShell” to TRUE, save property, then save object

Re-open the boot image and verify the property is set

SCCM reports show “The DefaultValue expression for the report parameter ‘UserTokenSIDS’ contains an error: The user name or password is incorrect. (rsRuntimeErrorInExpression)”

Had a rare fresh SCCM install recently with a client that has more of a security focus than many of our other clients.

Had SCCM 2002 (current at time of writing), Server 2019, SQL 2017 and Reporting services 2017.

Reporting services is using the default, “virtual service account”, which, under the covers, is “NT Service\SQLReportingServices”

 

After everything was installed, i tried to run a report and was greeted with

The DefaultValue expression for the report parameter ‘UserTokenSIDS’ contains an error: The user name or password is incorrect. (rsRuntimeErrorInExpression)

 

A bit of research showed that

  • The account logged on successfully when the reporting services service started and the security event log showed a successful logon – so i knew the username/password was correct
  • When trying to run a report, the security log showed event ID 4625 with the account “NT Service\SQLReportingServices” and a status and sub-status of 0xC000006D and 0xC0000064 respectively. These can be looked up at https://docs.microsoft.com/en-us/windows/security/threat-protection/auditing/event-4625
  • C:\Windows\ServiceProfiles\SQLServerReportingServices\AppData\Local\Temp\SCCMReporting.log showed “The username or password is incorrect”

Given that i knew the username and password was correct – i was a bit baffled by this for a while…. i spent a bit of time looking for how the auth is passed between the user session and SSRS. Googling presented a couple of suggestions, but they were the type of “answers” that just didn’t look like they fit the issue.

Fortunately, one of the other guys on a related project suggested adding the SCCM computer account to “Windows Authorization Access Group” – and it then proceeded to work.

Once finding that out, i searched and quickly came back with https://www.enhansoft.com/windows-authorization-access-group-ssrs-and-sccm/  which shows its been an issue for a while – one i just hadn’t run into before – and apparently my google searches were a bit too specific – and the error message others were getting was slightly different.

AADConnect – Proxy Address in conflict

Had an interesting one recently with a customer that has created cloud accounts for use during COVID-19 with approx 50 users. Each of these accounts were assigned a license and the users used teams, onenote, onedrive etc…. but not exchange online mailbox – as they already have an on-premise mailbox.

After this, we became involved and implemented AADConnect and exchange hybrid.

After removing the exchange license and clearing the mailbox info as per https://techcommunity.microsoft.com/t5/exchange-team-blog/permanently-clear-previous-mailbox-info/ba-p/607619

some users, not all, were not sync’ing correctly in AAD connect.

we can see the objectID that is conflicting

We can then confirm which object it is in O365

and then search for the conflict – which in this case, was a proxy address.

 

i had previously thought that ProxyAddresses was an exchange related attribute….. and given i had already removed all exchange information from the account… it shouldn’t be there….

At this stage the battle became how to get rid of it – since set-MSOLuser does not seem to allow it.

I found this – https://byronwright.blogspot.com/2017/09/remove-proxy-address-from-office-365.html – but in my instance, the conflicting address is the “correct” address, which keeps getting re-added automatically if you remove it.

Renaming the account temporarily doesn’t work – as the proxy address (as a user account attribute for an unknown reason) seems to update to match the UPN when changed.

 

This article had a good suggestion – https://www.petenetlive.com/KB/Article/0001588 but resulted in an error

Failed to apply fix – User with conflicting attribute is soft deleted in Azure Active Directory. Ensure the user is hard deleted before retrying.<a href=”https://go.microsoft.com/fwlink/?linkid=2007018″ target=”_blank”>Read More</a>

and we cant hard delete the user – they have data…

 

So – what are we left with? Hard matching. A process i was aware of, had never had to do (until now) – but thought was going to be overkill for this situation….. but apparently i was wrong.

There is a script here to help – https://gallery.technet.microsoft.com/office/immutableid-hard-match-in-d3518b08 or, if, like me, you prefer to step through it (when there is a small number of affected users)

import-module ActiveDirectory

$user = “SameAccountName” (not UPN)

$guid = [guid]((Get-ADUser -Identity $user).objectGuid)

$immutableId = [System.Convert]::ToBase64String($guid.ToByteArray())

Set-MsolUser -UserPrincipalName <UPN> -ImmutableId <Base64String>

 

this should work…. but didnt, instead i got the error

Set-MsolUser : Uniqueness violation. Property: SourceAnchor.
At line:1 char:1
+ Set-MsolUser -UserPrincipalName benjamin.primus@rtwsa.com -ImmutableI …
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (:) [Set-MsolUser], MicrosoftOnlineException
+ FullyQualifiedErrorId : Microsoft.Online.Administration.Automation.UniquenessValidationException,Microsoft.Online.Adminis
tration.Automation.SetUser

 

Then it clicked – this guy must have had an account previously…. run

get-msoluser -all -returndeletedusers

and sure enough – there it was….. with the immutableID assigned to that account…. so i then ran

Get-MsolUser -ReturnDeletedUsers -UserPrincipalName <UPN> | Remove-MsolUser -RemoveFromRecycleBin -Force

Followed by setting the immutableID again – which went through this time.

I then waited for AADConnect to sync again – which went through without error…. “yay” – i thought…. but then the user no longer showed up in the O365 GUI, and powershell showed

Errors : {Microsoft.Online.Administration.ValidationError,
Microsoft.Online.Administration.ValidationError,
Microsoft.Online.Administration.ValidationError,
Microsoft.Online.Administration.ValidationError…}

i used https://support.microsoft.com/en-us/help/2741233/you-see-validation-errors-for-users-in-the-office-365-portal-or-in-the

to troubleshoot this, in particular

$errors = (Get-MsolUser -UserPrincipalName “<User_ID>”).Errors

Get-MsolUser -HasErrorsOnly | select DisplayName,UserPrincipalName,@{Name=”Error”;Expression={($_.errors[0].ErrorDetail.objecterrors.errorrecord.ErrorDescription)}} | Export-csv c:\temp\validationerrors.csv

In this users case, there was still something hanging around from an old mailbox

The execution of cmdlet Set-SyncMailUser failed. The value “8dceb397-5af1-4142-8689-24bffeb83f92” of property “ArchiveGuid” is used by another recipient object. Please specify a unique value.

 

This lead me to https://docs.microsoft.com/en-us/archive/blogs/exovoice/how-to-fix-office365-user-provisioning-issues-that-are-generated-by-faulty-exchange-attributes

Run this to get the conflicting GUID

(Get-MsolUser -UserPrincipalName affecteduser@domain.com).errors.errordetail.objecterrors.errorrecord| fl

Then run this – with the inserted GUID from the above step

Get-Recipient -IncludeSoftDeletedRecipients ‘ExchangeGUID value’|ft RecipientType,PrimarySmtpAddress,*WhenSoftDeleted*

in my case – it was a soft deleted mailbox, so i then run

Remove-MailUser ‘ExchangeGUID value’ -PermanentlyDelete

After this – it was finally all good on next AADConnect Sync.

This whole thing was one huge pain in the arse…. but, at the same time, i learnt a lot…. I’ve previously always setup AADConnect first – and therefore not had cloud accounts that need to be merged etc. The whole process is way fucking harder than it needs to be…. which is very Microsoft…. but, i feel like this will come in useful in future!

Powershell fails to download from gallery – Wanring: Unable to download from URI xxx

Recently had this issue…. i was able to get to the specified URI via the browser with no issue.

Came across this post – https://techcommunity.microsoft.com/t5/windows-powershell/failed-downloading-az-and-other-modules-for-powershell/m-p/1292985 and this command fixed it for me

[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

Apparently support for older protocols was dropped in April 2020

While i agree with the move – its just a bit frustrating that, by the nature of the change, the site cant communicate the new requirement back.

SCCM – picking random clients for a collection

There are plenty of ways to randomly add systems to a collection in SCCM, a common query being something similar to

select SMS_R_System.Name from  SMS_R_System where SMS_R_System.SMSUniqueIdentifier like “%0” OR SMS_R_System.SMSUniqueIdentifier like “%1”

 

In a recent instance, i needed to ensure that systems were randomly chosen for a product roll out across a 1500-ish servers. The client was specific about staging the deployment and wanted to selection process of what went when to be truly random… fair enough.

SCCM cant really do do random when limited to a number…. the query above for example is likely to retrieve approx 20% of your total client count…. where-as i needed a random 50.

I ended up using the following powershell to accomplish the task.

 

Function Connect-ConfigMgr {
Param(
)
write-host -ForegroundColor Magenta “Connect-ConfigMgr”
Try {
Get-WMIObject -ComputerName $CMSiteServer -Namespace “root\SMS” -Class “SMS_ProviderLocation” -ErrorAction Stop | foreach-object{
if ($_.ProviderForLocalSite -eq $true){$Script:SiteCode=$_.sitecode}
}
if ($SiteCode -eq “”) {
throw (“Sitecode of ConfigMgr Site at ” + $CMSiteServer + ” could not be determined.”)
Exit 1
}
}
Catch {
$ErrorMessage = $_.Exception.Message
$text = “Error, could not connect to Site server $CMSiteServer, check spelling, network connectivity, permissions etc.”
$text += “`nError message: $ErrorMessage”
Write-Error $text
Exit 1
}
#$CMNamespace = “Root\SMS\$($SiteCode)”

write-host -ForegroundColor Green “ConfigMgr Site Code: $SiteCode”
write-host -ForegroundColor Green “ConfigMgr Site Server: $CMSiteServer”
#write-host “ConfigMgr Name Space: $CMNamespace”

#Import ConfigMgr Module
if (!(Get-Module -Name ConfigurationManager)) {
write-host “PowerShell module not loaded.”
If (Test-Path -Path “$($ENV:SMS_ADMIN_UI_PATH)\..\ConfigurationManager.psd1” -PathType Leaf) {
write-host “Attempting loa PS module from: $($ENV:SMS_ADMIN_UI_PATH)\..\ConfigurationManager.psd1”
Import-Module “$($ENV:SMS_ADMIN_UI_PATH)\..\ConfigurationManager.psd1” | Out-Null
}
ElseIf (Test-Path “$($ScriptPath)\ConfigurationManager.psd1” -PathType Leaf) {
write-host “Attempting loa PS module from: $($ScriptPath)\ConfigurationManager.psd1”
Import-Module “$($ScriptPath)\ConfigurationManager.psd1” | Out-Null
}
Else {
$text = “Error, unable to load PowerShell module ConfigurationManager.psd1, file not found.”
Write-Error $text
Exit 1
}
}
if (!(Get-Module -Name ConfigurationManager)) {
$text = “Error loading PowerShell module ConfigurationManager.psd1.”
Write-Error $text
Exit 1
}

#Set the current location to be the site code.
Set-Location “$SiteCode`:”
write-host -ForegroundColor Green “————————————————————————”
write-host “”
}### End Connect-ConfigMgr ###

Function GetRandomCollectionMembers {
Param(
)
write-host -ForegroundColor Magenta “Getting collection members”
$CollMembers = Get-CMCollectionMember -CollectionName $SourcecollectionName

#Randomising and selecting first x members
write-host -ForegroundColor Magenta “Selecting $NumberofMembers members randomly”
$CollMembers = $CollMembers | Sort-Object {Get-Random}
$CollMembers = $CollMembers | Select -First $NumberofMembers

Foreach ($CollMember in $CollMembers) {
$CollectionName = $CollMember.Name
write-host -ForegroundColor Green “Adding $CollectionName to $DestinationCollectionName”
Get-CMCollection -Name $DestinationCollectionName | Add-CMDeviceCollectionDirectMembershipRule -ResourceId $CollMember.ResourceID
}

}

##############Start of Main Script ####################

#Begin {

## Get script path and name
$ScriptPath = [System.IO.Path]::GetDirectoryName($MyInvocation.MyCommand.Definition)
$ScriptName = [System.IO.Path]::GetFileNameWithoutExtension($MyInvocation.MyCommand.Definition)
$PowerShellVersion = [int]$PSVersionTable.PSVersion.Major

#Save current path, to return after CM operations
$path = Split-Path -parent $MyInvocation.MyCommand.Definition

#Set variables here
$CMSiteServer = ‘sccm01.adexis.com.au’
$Script:SiteCode = $null
$SourcecollectionName = “All Systems”
$DestinationCollectionName = “RandomSystems”
$NumberofMembers = “5”

Connect-ConfigMgr
GetRandomCollectionMembers

#} #End Begin

Monitor flicker with AMD Radeon RX590

A month or two back, i grabbed a new media/VR PC – and had a great deal of flicker on the screen. After some googling, i found that “Virtual super resolution” seemed to be a common cause – and disabling this stopped the flicker.

Fast forward to today – and a monitor on my main work PC was flickering once every 5-10 seconds. I had recently swapped over my 2nd monitor from DVI to DP – but there was a 4 day delay between that and when the flicker started occurring. I had also updated the monitor driver for the primary display.

After some looking at the settings – i found that “virtual super resolution” had been enabled on the HDMI monitor – and given previous experience, i turned this off…. and voila – no flicker. Since my primary monitor is already a pretty good res – i have no need to “render at resolution higher than the displays native pixel grid”.

There didn’t appear to be a driver update between the time it wasn’t and was happening…. so my only guess is that updating the monitor driver caused the AMD settings software to re-evaluate and turn this setting on – which is frustratingly unnecessary.

Still – its a somewhat obscure setting – so – this post is to remind me for next time and help anyone else that might run into the same issue!

O365 ATP recommended config analyser

Came across this after following a post on linked in – checked it out – and its pretty bloody good.

Run the tool – it will spit out a report with recommendations for O365 Advanced threat protection…

The recommendations it spits out are of a pretty good quality

https://www.linkedin.com/pulse/reviewing-your-office-atp-configuration-cam-murray/

its not perfect (these things rarely are) – for example, after setting up DKIM and verifying, i find that it still reports an issue with DKIM – but it isn’t clear what the issue is – but it remains a good checklist enabler.

Exchange 2010 to 2016 mail flow stops with “421 4.4 2 connection dropped due to socket error”

Had a client ring today with a mail flow issue.

They are most of the way through their migration to 2016, but mail flow stopped with the error “421 4.4 2 connection dropped due to socket error” on the exchange 2010 side – when trying to relay to exchange 2016. This occurred after patching – but i’m not 100% on which patches they applied – and didn’t really have the time to find out.

 

Long story short – enabled TLS 1.2 on the Exchange 2010 (on a 2008 R2 OS) as per https://support.quovadisglobal.com/kb/a433/how-to-enable-tls-1_2-on-windows-server-2008-r2.aspx

then restarted the transport service – and mail stated flowing again.

While this is known for Exchange 2019 on Server 2019, where TLS 1.2 is the default – i wasn’t aware this was being retro-fitted……  not a bad thing…. and its only going to catch out the people that are lagging behind… still, considering how many people are lagging behind – this quick post might help!