Category: Exchange

Exchange – Can’t remove the domain ‘xy.com’ because it’s referenced in an email address policy by the proxy address template ‘smtp:@xy.com’.

Posted on by Hayes Jupe

The current company im working for is a large (for Aus) media company and as such, because basically everyone that works here is into marketing and sales, we just have a ridiculous number of domains.

As part of the cleanup process here – i bought up the huge number of domains we had and that maybe some of them weren’t current – and got back confirmation that many weren’t (approx 120)

The first job was removing them from exchange accepted domains, but i kept getting an error

Can’t remove the domain ‘xy.com’ because it’s referenced in an email address policy by the proxy address template ‘smtp:@xy.com’.

Even though the domains were not anywhere in the existing email address policies.

As usual, ADSIEdit to the rescue

navigate to

CN=Recipient Policies,CN=<Your Org Name>,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=xy,DC=com

Find the recipient policy which contains an attribute called “disabledGatewayProxy”

Remove each of the domain values from that attribute

Wait a few minutes for replication

then go back to exchange and remove the accepted domain.

Removing EXO litigation holds and making your mailbox functional again

Posted on by Hayes Jupe

Recently i had a situation where a user with a shared mailbox which had a ludicrous number of items. He recently started in the role and wanted to “start fresh” – but items could not be deleted.

A quick investigation found that litigation hold had been enabled for the mailbox (in addition to our org wide retention policies). It was unclear why the litigation hold had been enabled – and the reasons were lost due to staff turnover.

Finding and fixing the issue included:

  • get-mailbox <identity> | fl *hold*
    • This command will show the status of all holds on that specific mailbox. In my case, i could see that “LitigationHoldDate” and “DelayHoldApplied” were populated
    • In order to remove these i ran
      • get-mailbox <identity> | Set-Mailbox -LitigationHoldEnabled $false
      • get-mailbox <identity> | Set-Mailbox -RemoveDelayHoldApplied
  • After running these steps – i was still not able to delete items from the mailbox, so i ran
    • get-mailboxFolderStatistics <identity> -FolderScope RecoverableItems | FL Name,FolderAndSubfolderSize,ItemsInFolderAndSubfolders
    • and could see that the recoverable items and purges were both at 100GB – meaning that the quota, which also applied to delete items was full – so i could not yet delete anything more
    • In order to speed up the process of the managed folder assistant doing its job, run
      • Start-ManagedFolderAssistant -Identity <identity> -FullCrawl -HoldCleanup
    • After some time, if you re-run the get-mailboxFolderStatistics command, you should see the recoverable items and purges start to come down
  • Since this mailbox was full and is receiving a very high volume of new mail – the 100GB limit was going to be hit again very quickly – so in order to mitigate that for this initial cleanup, i then set
    • Set-Mailbox -Identity <identity> -RetainDeletedItemsFor 1
    • This will only detain deleted items for 1 day in the dumpster before purging
    • I will set this back to 30 days once the initial cleanup is complete and the mailbox is back to “normal” operation.
  • If the mailbox also has a retention policy in addition to the litigation hold
    • get-mailbox -Identity <identity> -Archive | fl *holds*
      • Take note of the ID of the in place hole
    • Connect-IPPSSession
    • Get-RetentionCompliancePolicy | fl Name,ExchangeObjectId
      • Match up the id of the retention policy to a name
    • Set-RetentionCompliancePolicy -Identity <NameOfPolicy> -AddExchangeLocationException <useridentity>
    • Get-Mailbox <identity> | FL DelayHoldApplied,DelayReleaseHoldApplied
    • Set-Mailbox <identity> -RemoveDelayHoldApplied
    • Set-Mailbox <identity> -RemoveDelayReleaseHoldApplied
    • Start-ManagedFolderAssistant -Identity <identity> -FullCrawl -HoldCleanup
    • <wait>
    • To check the status, use
      • get-mailboxFolderStatistics <identity> -FolderScope RecoverableItems | FL Name,FolderAndSubfolderSize,ItemsInFolderAndSubfolders

I’ve read some articles that say that “Start-ManagedFolderAssistant” needs to be run twice to pick up the new settings – but its all a bit of black magic because we cant see what is happening at the backend…. i tend to run it two or three times – as if those articles are right, im speeding things up, if they are wrong, there is no visible harm.

Issues with mailbox migration, multiple identities where tenant wide retention policies in use

Posted on by Hayes Jupe

This is a somewhat niche issue – which is why im documententing it here.

Scenario

  • Exchange migration to exchange online which i came into approx 75% of the way through – so i don’t have any history on why some things have happened (and there is no useful doco)
  • Tenant wide retention policies are in place for all data (legislative requirement im led to believe for this client)
  • Identity sync via AADConnect
  • Some mailboxes cannot be moved. Powershell error message from new-moverequest indicates that the identity is not unique

Investigation

  • Start off by looking at the AAD Object sync with
    • Connect-MSOLService
    • (Get-MsolUser -UserPrincipalName identity@goes.here.com).errors.errordetail.objecterrors.errorrecord| fl ErrorCode
    • The output, will likely look something like this:
      • The value “<guid-value>” of property “ArchiveGuid” is used by another recipient object. Please specify a unique value.
  • Next up, we want to have a look at the potential duplicate objects
    • Connect-ExchangeOnline
    • Get-recipient -identity <identity> -includesoftdeletedrecipients
      • This will likely show you 2 (or more) mail user objects
    • To confirm the soft-deleted mailuser object you can use
      • Get-MailUser -resultsize unlimited -SoftDeletedMailUser -Identity <identity> | fl *guid*
      • Notice the ArchiveGUID returned is the same as the ArchiveGUID from the Get-MSOLuser error retrieved earlier in the investigation
    • If you then try and run the obvious next step
      • Get-MailUser -Identity <identity> -SoftDeletedMailUser | Remove-MailUser
      • You will get an error similar to
        • Remove-MailUser: The operation couldn’t be performed because object ‘Soft Delete d Objects\<identity>’ couldn’t be found on ‘SYBPR01A004DC01.AUSPR01A004 .PROD.OUTLOOK.COM’

Now, i know what your thinking “just exclude the mailbox from the retention policy” – and there within lies the issue…. there is no mailbox, only a mail user object, but with an archive mailbox that has been retained by the retention policy after the primary mailbox has been removed. It is then, to my knowledge, impossible to exclude that archive mailbox from retention – as its associated with a mailuser – not a mailbox.

As to how these identities got into this state…. absolutely no idea. I wasn’t around for the earlier parts of the project – but given some other things i’ve seen at the client, standardisation and documentation appear to be frowned upon (which is why i’m getting out ASAP)

 

Solution

The unfortunate solution is to log a call with O365 support.

I included all of the above information in my original support request and was still asked to run a “get-mailbox”… i included all the info again (and again, and again over a teams call showed them the exact same errors and data that i sent them) – and eventually they got the point (took approx 15 business days) and sent it to an internal team, who deleted the objects

Unfortunately i cant post the case number for reference (as it would potentially identify the client) – but maybe pointing MS support to this article might speed the process for others (?). Ideally, there would be a way around this, without engaging support – but there is not as far as I’m aware as of June 2023.

Issue with manually created EXO inbound connector in hybrid environment

Posted on by Hayes Jupe

Working at a client whom are approx 75% of the way through their migration to exchange online – and there are some odd things im running into – so here’s one of them.

The scenario and issue

  • Exchange hybrid setup, with servers on prem and EXO active. Active mailboxes in both.
  • Mail flow from on prem to EXO shows the following:
    • Outbound SMTP logs shows the message being handed off correctly to EXO
    • Message tracking in EXO shows 3 copies of the message, all of which, when looking into the details are bounces
    • When looking in security.microsoft.com, the messages have been flagged as phishing attempts… with seemingly no way to flag them as not phishing attempts
  • The connectors on-prem looked ok, and after, double, triple and ninieteenth-thousandth checking, they were solid
  • The connectors in EXO were manually created (for reasons i don’t know that pre-date me) and the HCW created connectors had been disabled. No idea why.
  • The connectors in EXO looked fine and validated without any issue
  • After circling around for ages, i compared the disabled HCW connector with the active connect with “get-inboundconnector | fl”
  • This is when i noticed that the HCW created connector had IP’s in the “EFSkipIPs” property

The Fix

  • EFSkipIPs can be configured as per the powershell doco here
  • The EFSkipIPs property looks like it defines IP’s that should be excluded from enhanced filtering. Since the HCW automatically populates this field – most of us will never have to use this…. but if some bright spark decides that the HCW isn’t good enough for them (for whatever reason), then this becomes important.
  • Because i had the previous, disabled connector, created by the HCW – i already knew the IP’s i needed to add.  If you don’t have this, you will need get your the Public IP that is presented to EXO. This could be obtained with something such as www.whatsmyip.com
  • The multi-valued property… well, it would have been nice on the doco page if an example was included… so since there isn’t one in the official doc – here is an example below:

Set-inboundConnector -Identity “OrgToEXO” -EFSkipIPs @{Add=”xx.xx.xx.xx”, “xy.xy.xy.xy”}

  • After that, i needed to wait approx 15 minutes (not sure on the exact time, but it didn’t work straight away) – and bingo-bango – no more mail flow issue

Removing your last exchange on-premise servers after migrating to exchange online

Posted on by Hayes Jupe

One question that comes up constantly from clients is “Once we move all mailboxes to exchange online, we can remove all our exchange servers, right ?”

 

To which we answer “no – for these reasons….” but its something that has a great deal of trouble sticking…. as the same question gets asked over and over again… sometimes by the same client, sometimes even by the same people.

I stumbled across this today – an article written a few months back which nicely lays out the possible scenario’s

https://docs.microsoft.com/en-us/exchange/decommission-on-premises-exchange

well done to the authors for laying it out plainly!

We don’t really work with any places that will be in scenario 1 anytime soon… but scenario 2 and 3 are spot on… with 3 being the most common scenario for our clients.

O365 ATP recommended config analyser

Posted on by Hayes Jupe

Came across this after following a post on linked in – checked it out – and its pretty bloody good.

Run the tool – it will spit out a report with recommendations for O365 Advanced threat protection…

The recommendations it spits out are of a pretty good quality

https://www.linkedin.com/pulse/reviewing-your-office-atp-configuration-cam-murray/

its not perfect (these things rarely are) – for example, after setting up DKIM and verifying, i find that it still reports an issue with DKIM – but it isn’t clear what the issue is – but it remains a good checklist enabler.

Exchange 2010 to 2016 mail flow stops with “421 4.4 2 connection dropped due to socket error”

Posted on by Hayes Jupe

Had a client ring today with a mail flow issue.

They are most of the way through their migration to 2016, but mail flow stopped with the error “421 4.4 2 connection dropped due to socket error” on the exchange 2010 side – when trying to relay to exchange 2016. This occurred after patching – but i’m not 100% on which patches they applied – and didn’t really have the time to find out.

 

Long story short – enabled TLS 1.2 on the Exchange 2010 (on a 2008 R2 OS) as per https://support.quovadisglobal.com/kb/a433/how-to-enable-tls-1_2-on-windows-server-2008-r2.aspx

then restarted the transport service – and mail stated flowing again.

While this is known for Exchange 2019 on Server 2019, where TLS 1.2 is the default – i wasn’t aware this was being retro-fitted……  not a bad thing…. and its only going to catch out the people that are lagging behind… still, considering how many people are lagging behind – this quick post might help!

Exchange migration and AdminSDProp

Posted on by Hayes Jupe

I recently did a piece of work for a client – moving from Exchange 2010 to 2016. Nothing too exciting…. but they did have an interesting issue.

Once migrating some of test mailboxes, inheritance in AD had to be enabled for a few admin accounts before they could connect via outlook and activesync – to be expected (yes yes, i know admin accounts shouldn’t have mailboxes, but we all know that some clients still do this – and thats not the focus of this post)

What was interesting, was that on further investigation – every account has AdminCount set to “1” and had inheritance disabled – not something to handle manually..

On further investigation, it was found that via some group nesting, all users were members of print operators.

Groups with AdminCount=1 can be located utilising the powershell

Get-ADGroup -LDAPFilter “(admincount=1)”

The client did not want to immediately reverse this due to potential client impacts – and while i disagreed – excluding a group from AdminSDHolder was not something i had looked into before – so i was interested.

A short amount of googling later – and reading a long list of articles, we decided to exclude “print operators” from AdminSDHolder. Two of the better articles (for reference) around this were:

https://social.technet.microsoft.com/wiki/contents/articles/22331.adminsdholder-protected-groups-and-security-descriptor-propagator.aspx

https://social.technet.microsoft.com/Forums/windows/en-US/ddd8d964-6c8b-42b0-b170-2cacaa283d1c/adminsdholder-remove-groups-server-operators-print-operators-backup-operators?forum=winserverDS

The condensed version of the overall solution is:

  • In order to exclude a group from AdminSDHolder, you can utilise ADSIEdit to modify  the property dsHeuristic under “CN=Directory Services,CN=Windows NT,CN=Services,CN=Configuration,DC=YourDomain,DC=com”
  • The value can be calculated depending what groups you wish to exclude, the 2nd linked technet social post above has a really nice explanation
  • in my case, i needed to it to “0000000001000004” (without the quotes)
  • Once this is done, clear the AdminCount property from the appropriate group (in my case, this was “print operators” + another group within the long-line of nesting this client had)
  • Re-run the powershell – “Get-ADGroup -LDAPFilter “(admincount=1)” to verify the groups no longer show up
  • Once this is done, we need to remove the “adminCount” from each of the affected user accounts and enable inheritance – to do that, you can run the below script

 

$users = Get-ADUser -ldapfilter “(objectclass=user)” -searchbase “<DN of path you wish to use for your search base>”
#$users = Get-ADUser -Identity <username> ‘ Use this for testing on a single user first

#Get domain values
$domain = Get-ADDomain
$domainPdc = $domain.PDCEmulator
$domainDn = $domain.DistinguishedName

#HashTable to be used for the reset
$replaceAttributeHashTable = New-Object HashTable
$replaceAttributeHashTable.Add(“AdminCount”,0)

$isProtected = $false ## allows inheritance
$preserveInheritance = $true ## preserve inheritance rules

ForEach($user in $users)
{
# Binding the users to DS
$ou = [ADSI](“LDAP://” + $user)
$sec = $ou.psbase.objectSecurity

Set-ADUser -identity $user -clear adminCount

if ($sec.get_AreAccessRulesProtected())
{
#Changes AdminCount back to &lt;not set&gt;
Get-ADuser $user.DistinguishedName -Properties “admincount” | Set-ADUser -Remove $replaceAttributeHashTable -Server $domainPdc
#Change security and commit
$sec.SetAccessRuleProtection($isProtected, $preserveInheritance)
$ou.psbase.commitchanges()
}
}

 

References:

https://social.technet.microsoft.com/wiki/contents/articles/22331.adminsdholder-protected-groups-and-security-descriptor-propagator.aspx

https://sdbrett.com/BrettsITBlog/2016/12/discover-clear-admincount-powershell/

https://docs.microsoft.com/en-us/previous-versions/technet-magazine/ee361593(v=msdn.10)?redirectedfrom=MSDN

https://blogs.msdn.microsoft.com/muaddib/2013/12/30/how-to-modify-security-inheritance-on-active-directory-objects-using-powershell/

https://blogs.technet.microsoft.com/chadcox/2018/01/08/adposh-find-and-fix-adminsdholder-orphans-admincount/

http://www.selfadsi.org/extended-ad/ad-permissions-adminsdholder.htm

https://social.technet.microsoft.com/Forums/windows/en-US/ddd8d964-6c8b-42b0-b170-2cacaa283d1c/adminsdholder-remove-groups-server-operators-print-operators-backup-operators?forum=winserverDS

 

Active Directory 2019 and Exchange 2019 – what’s new

Posted on by Hayes Jupe

The short answer is – not much.

Exchange 2019 was released a few weeks back, but was effectively un-usable, as Exchange 2019 requires Windows Server 2019…. and Windows server 2019 got pulled from release (like Windows 10 1809) due to some issues.

Windows Server 2019 was re-released a few days ago, which allowed nerds everywhere (including me) to put Server 2019 and Exchange 2019 into a test environment.

The most striking thing that is immediately noticeable is that everything looks the same…. The install process, the GUI, the management, all looks the same as it did in 2016. To me, this is a good thing – while Microsoft of the past seemed to believe that moving functions between areas was good – some consistency is nice to have too.

 

Active Directory

First appearances indicate there is nothing new in AD 2019, the installation process and management is exactly the same as 2016.

While installing, there is not even an option to set the forest and domain functional level to “2019” – only 2016.

A quick look at the schema version indicates it has increased and quick google finds this article

https://blogs.technet.microsoft.com/389thoughts/2018/08/21/whats-new-in-active-directory-2019-nothing/

So, while there is something new in the schema, its an incredibly small update….. and there are no new features or functionality of any type to focus on.

 

Exchange 2019

Exchange 2019 is a bit the same as AD, everything appears to be the same as Exchange 2016, from the install process to the management interface.

A google comes up with this

https://practical365.com/exchange-server/should-you-upgrade-to-exchange-server-2019/

So there are some changes and feature updates – but these updates may not have an impact/matter to your organization.

 

I found these two releases interesting overall as

  • AD is the core of many enterprise networks
  • Exchange is a core business application

To see a new release of both of these products with very minimal improvements I think demonstrates where all Microsoft’s development effort is going (which, to be fair, we already knew)

 

Exchange hybrid – mailboxes missing on-premise

Posted on by Hayes Jupe

While hybrid exchange environments are awesome for stretching your on premise exchange topology to Office 365, they do introduce a bunch of complexity – primarily around user creation, licensing, and mail flow.

I recently had an issue at a client where they had email bounce-backs from an on premise service destined for a few Exchange Online mailboxes. For some reason, these few mailboxes didn’t appear in the on-premise exchange environment (as remote Office 365 mailboxes), so exchange was unable to route the emails destined for those particular mailboxes.

In general, you should be creating your mailboxes on premise (Enable-RemoteMailbox), then synchronising via AADConnect – that way the on premise environment knows about the mailbox and it can be managed properly. This client was actually doing this, but obviously the process broke somewhere along the way for a few mailboxes.

There’s a bunch of different options on Google about how to get the mailbox to show up on premise – with a lot of them recommending to remove the mailbox and start again (er… how about no!).

I came across this Microsoft article on a very similar issue, but for Shared Mailboxes created purely in Exchange Online. Looking at the process, it looked like a modified version may work for user mailboxes – and it does. Below is a quick and dirty powershell script that can be used to fix a single mailbox:

#Specify who we're working with
$UPN = "end.user@domain.com"
#Local exchange server
$ExServer = "Server1.local"
#365 Domain - for remote routing address
$RoutingDomain = "mydomain.mail.onmicrosoft.com"

#Connect to 365 Exchange - only import select cmdlets so they don't conflict with the Exchange On Premise session
$RemoteSession = New-PSSession -ConfigurationName Microsoft.Exchange `
      -ConnectionUri https://outlook.office365.com/powershell-liveid/ -Credential $(Get-Credential) `
      -Authentication Basic -AllowRedirection
Import-PSSession $RemoteSession -CommandName Get-Mailbox

#Connect to local exchange - only import select cmdlets so they don't conflict with the Exchange Online session
$LocalSession = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri "http://$ExServer/PowerShell/" `
      -Authentication Kerberos -Credential $(Get-Credential)
Import-PSSession $LocalSession -CommandName Enable-RemoteMailbox, Set-RemoteMailbox

#Get the Alias and ExchangeGuid from 365
$Mailbox = Get-Mailbox $UPN
$Alias = $Mailbox.Alias
$ExchangeGUID = $Mailbox.ExchangeGuid

#Create a remote mailbox
Enable-RemoteMailbox $UPN -Alias $Alias -RemoteRoutingAddress "$Alias@$RoutingDomain"
#Set the Remote Mailbox GUID to match the 365 mailbox GUID
Set-RemoteMailbox $Alias -ExchangeGuid $ExchangeGUID

#Remove sessions
Get-PSSession | Remove-PSSession