Issues with mailbox migration, multiple identities where tenant wide retention policies in use

This is a somewhat niche issue – which is why im documententing it here.

Scenario

  • Exchange migration to exchange online which i came into approx 75% of the way through – so i don’t have any history on why some things have happened (and there is no useful doco)
  • Tenant wide retention policies are in place for all data (legislative requirement im led to believe for this client)
  • Identity sync via AADConnect
  • Some mailboxes cannot be moved. Powershell error message from new-moverequest indicates that the identity is not unique

Investigation

  • Start off by looking at the AAD Object sync with
    • Connect-MSOLService
    • (Get-MsolUser -UserPrincipalName identity@goes.here.com).errors.errordetail.objecterrors.errorrecord| fl ErrorCode
    • The output, will likely look something like this:
      • The value “<guid-value>” of property “ArchiveGuid” is used by another recipient object. Please specify a unique value.
  • Next up, we want to have a look at the potential duplicate objects
    • Connect-ExchangeOnline
    • Get-recipient -identity <identity> -includesoftdeletedrecipients
      • This will likely show you 2 (or more) mail user objects
    • To confirm the soft-deleted mailuser object you can use
      • Get-MailUser -resultsize unlimited -SoftDeletedMailUser -Identity <identity> | fl *guid*
      • Notice the ArchiveGUID returned is the same as the ArchiveGUID from the Get-MSOLuser error retrieved earlier in the investigation
    • If you then try and run the obvious next step
      • Get-MailUser -Identity <identity> -SoftDeletedMailUser | Remove-MailUser
      • You will get an error similar to
        • Remove-MailUser: The operation couldn’t be performed because object ‘Soft Delete d Objects\<identity>’ couldn’t be found on ‘SYBPR01A004DC01.AUSPR01A004 .PROD.OUTLOOK.COM’

Now, i know what your thinking “just exclude the mailbox from the retention policy” – and there within lies the issue…. there is no mailbox, only a mail user object, but with an archive mailbox that has been retained by the retention policy after the primary mailbox has been removed. It is then, to my knowledge, impossible to exclude that archive mailbox from retention – as its associated with a mailuser – not a mailbox.

As to how these identities got into this state…. absolutely no idea. I wasn’t around for the earlier parts of the project – but given some other things i’ve seen at the client, standardisation and documentation appear to be frowned upon (which is why i’m getting out ASAP)

 

Solution

The unfortunate solution is to log a call with O365 support.

I included all of the above information in my original support request and was still asked to run a “get-mailbox”… i included all the info again (and again, and again over a teams call showed them the exact same errors and data that i sent them) – and eventually they got the point (took approx 15 business days) and sent it to an internal team, who deleted the objects

Unfortunately i cant post the case number for reference (as it would potentially identify the client) – but maybe pointing MS support to this article might speed the process for others (?). Ideally, there would be a way around this, without engaging support – but there is not as far as I’m aware as of June 2023.

Issue with manually created EXO inbound connector in hybrid environment

Working at a client whom are approx 75% of the way through their migration to exchange online – and there are some odd things im running into – so here’s one of them.

The scenario and issue

  • Exchange hybrid setup, with servers on prem and EXO active. Active mailboxes in both.
  • Mail flow from on prem to EXO shows the following:
    • Outbound SMTP logs shows the message being handed off correctly to EXO
    • Message tracking in EXO shows 3 copies of the message, all of which, when looking into the details are bounces
    • When looking in security.microsoft.com, the messages have been flagged as phishing attempts… with seemingly no way to flag them as not phishing attempts
  • The connectors on-prem looked ok, and after, double, triple and ninieteenth-thousandth checking, they were solid
  • The connectors in EXO were manually created (for reasons i don’t know that pre-date me) and the HCW created connectors had been disabled. No idea why.
  • The connectors in EXO looked fine and validated without any issue
  • After circling around for ages, i compared the disabled HCW connector with the active connect with “get-inboundconnector | fl”
  • This is when i noticed that the HCW created connector had IP’s in the “EFSkipIPs” property

The Fix

  • EFSkipIPs can be configured as per the powershell doco here
  • The EFSkipIPs property looks like it defines IP’s that should be excluded from enhanced filtering. Since the HCW automatically populates this field – most of us will never have to use this…. but if some bright spark decides that the HCW isn’t good enough for them (for whatever reason), then this becomes important.
  • Because i had the previous, disabled connector, created by the HCW – i already knew the IP’s i needed to add.  If you don’t have this, you will need get your the Public IP that is presented to EXO. This could be obtained with something such as www.whatsmyip.com
  • The multi-valued property… well, it would have been nice on the doco page if an example was included… so since there isn’t one in the official doc – here is an example below:

Set-inboundConnector -Identity “OrgToEXO” -EFSkipIPs @{Add=”xx.xx.xx.xx”, “xy.xy.xy.xy”}

  • After that, i needed to wait approx 15 minutes (not sure on the exact time, but it didn’t work straight away) – and bingo-bango – no more mail flow issue

Disabling “link Azure Active Directory accounts to personal Microsoft accounts”

In some type of bizarre alternate reality, linking a corporate Microsoft account to a personal Microsoft account so the user can earn “Microsoft Rewards points for Microsoft Bing searches done in their browser or Windows search box while signed in with their AAD account” is seen as a good option to have – despite the security concerns and fact that almost no-one uses Bing or knows WTF “Microsoft rewards” are.

For the rest of us – fortunately we can disable this insanity using an MS provided script from https://download.microsoft.com/download/2/4/5/245c3b59-a897-4ee1-a24d-e0ead9007603/AccountLinkingDisable.ps1

General FAQ – https://www.microsoft.com/en-us/bing/account-linking-admin-faq

Office 365 deployment – notes for some painful bits

O365 or M365 Apps for enterprise or Microsoft defender for spreadsheets – or whatever its called next week – can be… unpleasant…. at the best of times to deploy and configure…. here’s a couple of things that can help.

 

Use the web-based config.office.com to create your xml, not SCCM

The main reason here, is that when you go to “Office 365 client management” in SCCM, and select “Office 365 installer” and follow the wizard, you will get a bunch of nicely presented options, but for some reason the language option of Language ID=”MatchOS” is not an option that is available in the drop down.

If you head over to config.office.com, Language ID=”MatchOS”  is the first option. This makes deployment much easier where you would like your proofing language to be English (Australian) for example, or where you want to use proofing languages that match the the region where you are deploying to multiple regions.

 

Teams is a per-user install – so if you have performed basic security lockdowns such as disabling the run key – it wont install

This is just all types of absolute shit. When teams (optionally) installs as part of office, it actually installs the “Teams Machine-Wide Installer”, and then as each user logs on, it installs in the users profile utilising the run key.

One work around for this is to create a shortcut in all-users startup with the command C:\Program Files (x86)\Teams Installer\Teams.exe” –checkInstall –source=default as helpfully documented here – https://xenit.se/tech-blog/why-does-teams-not-install-for-my-users/

There is also other possible modifications such as https://sysmansquad.com/2021/01/26/trigger-teams-installer-for-user-after-machine-wide-installer/

 

Now that i have installed teams, how do i pre-configure some of the settings for my users ?

Here comes the fun part – setting defaults as part of an install or using group policy or registry settings in HKLM… well, no… those traditional methods im sure are termed “legacy” somewhere within MS…

Configuring Startup Settings in Microsoft Teams with Windows PowerShell

or

https://www.nubo.eu/Change-The-Display-Language-In-Your-Microsoft-Teams-Client/

Thanks to links above, we have example powershell which we can use to modify the json file that keeps the user preferences… and in order to save them, we just need to terminate the teams process – and roll this out across the userbase. Cool, normal and totally not user intrusive at all.

 

Updating Outlook to use Celsius

Its understandable that Farenheit is the default, given the 4.21% of the worlds population use that unit of measurement.

For the other 95% of the planet – updating outlook to use Celsius is almost as fun…. its another per-user setting stored in a semi-randomly named dat file – as nicely documented in this technet forum post.

https://social.technet.microsoft.com/Forums/en-US/6a30c22d-2224-44eb-a409-12bb85a0b9e0/outlook-2013-change-temperature?forum=officeitpro

Again, the outlook process needs to be terminated for this to occur – completely reasonable. Given that Technet forums will disappear at some stage – the bare-bones of the script is below.

$LocalPath = "$env:USERPROFILE\AppData\Local\Microsoft\Outlook\RoamCache\*"
$LocalFileName = (Get-Item -Path $LocalPath -Filter "Stream_Weather*.dat").FullName
$Content = Get-Content -path $LocalFileName
Stop-Process -Name outlook -Force -ErrorAction SilentlyContinue
Set-Content $LocalFileName $content.Replace("DegreeType"" v=""9-1""", "DegreeType"" v=""9-0""")

 

The other option is to simply disable it via GPO – given its weather is far from accurate (at least for Aus) anyway.

User Config | Admin templates | Outlook 2016 | Outlook options | preferences | calendar options | Disable weather bar > Enabled

 

What about OneDrive ?

A OneDrive is installer is copied with office as well, and exists in the same folder as teams ready for deployment on a per user basis… however, thankfully the OneDrive team seems to be a bit more aware of management and has made some options available for us to install the “sync app” on a per-machine basis – as documented (very well) here – https://docs.microsoft.com/en-us/onedrive/per-machine-installation

i use a seperate application installation with Onedrivesetup.exe /allusers /silent /configure_business:<tenantID> combined with some group policy settings for silent logon

 

Options

So… how can we address this complete and utter shit-show and make it into something semi-usable, manageable and not user-intrusive ?

  • We can re-enable the processing of the run key…. this allows teams to install per-user with the installer-provided method… but goes against pretty much every reputable Windows hardening guide… and it runs every time – which is just inefficient.
  • We could use a self-deleting scheduled task as recommended here.… but what happens when another user comes along to the same machine?
  • We could use ActiveSetup… its a bit better, as it runs pre-user shell (and is therefore less intrusive) and is only run once per user. Unfortunately this is only effective for the teams install…. and not the modification of the json for teams or the .dat for outlook – as they need to be run once in the user context before the config files are created
  • We could use a logon script – but that runs every time – and is really no better than a scheduled task or using the run key or startup folder
  • Use a funky per-user detection rule as per https://docs.microsoft.com/en-us/onedrive/deploy-on-windows and https://go.microsoft.com/fwlink/p/?LinkId=824069 and adapt it for teams

 

None of the options are great… and i’ve found that i end up using something different at each client – because they have different requirements.