Consolidating services into Azure

Recently I had an exceedingly poor experience with my external DNS provider, Namecheap. After they had some mail issues, their 2FA emails weren’t coming through…. I could see they weren’t even hitting O365… but of course, their support refused to acknowledge this – and went down a path of (bizarrely) insistently asking for a scan of government issued ID – very scammer like. This was enough to make me re-evaluate my external services and where they lived – with a specific view to bringing them into Azure…

 

Why bring all the services into O365/Azure ?

  • One provider… and MS are a provider that isn’t disappearing anytime soon. I can’t see us moving away from O365 in the foreseeable future – so if that service is anchored – why not move others towards it ?
  • Azure management interface and scripting are generally pretty good
  • MS support is generally terrible…. But they have never tried to get me to send a government issued photo ID. Community support around Azure/O365 varies greatly – but there are many great blog articles etc around.
  • Cost – MS partners can get Azure credit with some partnership options – some months I use it all – other months I don’t – so it makes sense to use as much of the credit as possible

 

DNS

DNS seemed like the easiest candidate and it was also the service that was about to expire on Namecheap.

I logged a call with O365 support, asking about transferring a DNS zone into O365/Azure… The guy was actually reasonably nice and tried to be helpful – but seemed to have it in his head that DNS was a website or something…. Anyway, the upshot of the conversation was “no, you can’t transfer in… you can only use O365 DNS if you purchased the domain from MS”

After this I went off did some searching and found the incredibly aptly named Azure DNS.

5 minutes later, it was all setup and ready to go

  • Go to the Azure portal
  • Create resource
  • Networking -> DNS Zone
  • Create
    • Select your subscription, resource group and zone name
  • Add your records

 

I tested the service before updating my registrar using

Nslookup <record name> ns1-02.azure-dns.com.

I then waited a few days – as I wanted to see how much the DNS zone would cost without usage (as Azure pricing pages are exceedingly difficult to decipher IMO) – and while this will obviously vary greatly for everyone – for my zone after 5 days (with no traffic mind you) – the cost for that service was a whopping $0.05.

 

Based on that, I updated my registrar to point to the Azure DNS servers, then ran an O365 check – just in case – and all was good.

 

Domain transfer

Given the above conversation, I thought it was unlikely, but quickly found these items via google

https://learn.microsoft.com/en-us/answers/questions/2168/how-can-i-transfer-a-domain-from-godaddy-to-azure

https://jrudlin.github.io/2018/10/27/domain-name-registration-transfer-to-azure-app-service-domains/

 

So it is possible – but is a bit of kludge… additionally, according to the first forum post at least – the ability to “transfer in” in on the MS radar

 

Given my domain registrations for my current domains does not run out until 2024 – I am going to wait until they are closer to expiry – then come back and see if MS have an officially supported method of transferring domain registration into O365/Azure.

 

WordPress

WordPress on Azure went GA in August 2022 – and you can find some details about it here – https://learn.microsoft.com/en-us/azure/app-service/quickstart-wordpress

 

Unfortunately, when going to https://portal.azure.com/#create/WordPress.WordPress – I am immediately presented with “MySQLFlexible server is not available for your selection of subscription and location”… changing location does nothing – so its something to do with my partner subscription…. Wouldn’t want partners to be able explore your product set and become more familiar with the wide range of Azure offerings…. (or write blog posts on how to use their products) – can’t have that! Geez MS licensing people make some whacky fucking decisions.

 

Static Websites

Last up was my company website, which is a static HTML website. After some google, I found there were a few methods, such as using an Azure storage account – but that seemed to have some limitations around certificate assignment and host headers (from reading other posts). The other main option appeared to be  Azure static web apps – which was a more complete offering, but also more complex. It required linkage to a GitHub or Azure DevOps account and asked me a bunch of questions that I had NFI about. Remember, I’m an infra nerd… so once it goes past PowerShell (or VBScript.. or JSON if I have to) – its all quantum realm magic to me.

Anyway, after some reading and making a few mistakes, the rough process is:

  • Create a GitHub account (I went GitHub – since I already had an account and some code in there)
  • Create a project in GitHub
  • Upload the static html/css site to the GitHub project
    • For whatever reason, only about 90% of the files uploaded first try – but there were no errors. I only found out some files were missing when I tried to use the published website. I’m a newbie to Github – so maybe I did something wrong – but its worth looking out for
  • Go to the Azure portal
  • Create resource
  • Search for “Static Web app”
  • Create
    • Select your subscription, resource group and name
    • Select your hosting plan…. Free is obviously a good place to start – you can always upgrade it later
    • Deployment details – I selected “GitHub”
    • Authorise the connection between the static web app and GitHub
  • The site will now be ready via the Azure URL – which is great for testing to make sure everything is correct
    • My site was ready fairly quickly – but a number of the images didn’t display.
    • I posted on a forum about this and eventually found that files within the Static web app are case sensitive… so my html referred to background.jpg… when the file was named Background.jpg…. I got rid of the capitalisation once I realised, and all was good.
  • Once everything is correct
    • Add your custom domain
      • Azure static web app -> custom domains
      • Add – custom domain on Azure DNS
      • Select your DNS zone from the drop down
      • In the domain name box, you must enter the FQDN… e.g. www.company.com, not just “www” (give that you select the zone in the other drop down – this is confusing)
      • Now – as per this bug – https://github.com/Azure/static-web-apps/issues/202 – I found I got the error “Failed to add custom domain to SWA with error message”… but the CName entry was still actually added… this was a start… but since it did not show up in “custom domains” – the site still did not work without that host header.
      • Due to this, I simply added it as a “custom domain” (even though the DNS was/is hosted in Azure DNS) – and it took a minute to validate, but worked fine

 

In summary

  • Azure DNS – easy
  • Azure static web sites easy-ish… but wasn’t clear that it was case sensitive and the adding of a custom domain seems very buggy
  • Not being able to transfer to MS as a domain registrar is a bizarre omission
  • Microsoft licensing people still make decisions by rolling a D20 inside a Zorba ball when drunk – this is unlikely to change in my lifetime
  • Run the fuck away from NameCheap

Server and SQL upgrades – lessons learned

Im just finishing up on a project where i was upgrading a bunch of servers from 2012 R2 to 2019 or 2022 (depending on what the associated app supported), including a bunch of SQL clusters.

I’ve always been SQL adjacent – working wit/upgrading/installing SQL for other products to utilise… so i have some incidental knowledge – but its not my core skill set.

Things of note from the upgrades were:

 

When performing an in-place OS upgrade – upgrade speed can be significantly increased if you remove old user profiles

Some of the servers i was upgrading had hundreds of profiles on them that had not been used for a year or more….. all servers had at least 20 “Account unknown” profiles 

 

SQL Error Logging

The best way to find the error log if any upgrade goes wrong is to look in the registry at

KEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\<instance/version>\MSSQLServer\Parameters\

You can then copy/paste the path to the error log and get some helpful errors out

 

SSISDB is the bane of SQL cluster upgrades

SQL 2014 and below don’t support replicating SSISDB via AAG, so before you service pack, this DB must be removed from the AAG replication and the passive nodes have the DB deleted.

SQL 2016 and above support replicating SSISDB – so service packs can be applied without having to remove SSISDB from anywhere

All SQL upgrades (e.g. SQL 2014 or 2016 to SQL 2019) do not allow SSISDB to be part of an AAG – so SSISDB must be removed from the replication group and have the copy on the passive nodes deleted first.

If you forget this, you will likely see an error message similar to 

Script level upgrade for database ‘master’ failed because upgrade step ‘SSIS_hotfix_install.sql’ encountered error 15151, state 1, severity 16

 

Starting SQL to fix issues

So – you have run into an issue with the upgrade, as, for example, SSISDB was still replicated….. but now you cant start the SQL service to delete it

This is where /T902 comes in handy

  • Get the short name of your SQL service (from services.msc)
  • open a elevated command prompt
  • net start MSSQL$Instancename /T902

You can then do what you need to the SQL configuration.

 

Reporting services

Reporting services in 2017 and above is not a straight upgrade from 2016 and below. There’s plenty of articles around the web on the upgrade process – but…..

 

During inventory, make sure your discover SSISDB and Reporting services instances

In hindsight, one of the things i would have focused on more in my pre-upgrade inventory script was to identify SSISDB and reporting services instances.

Many of these in the recent project were present but not actually needed/in-use and could just be uninstalled.

 

Cluster rolling upgrades

This is well documented – but just to make it nice and short (the MS doco makes it seem harder than it is)

  • Ensure SQL AAG and cluster resource active node is node “X”
  • Ensure failover is set to manual
  • Verify SQL AAG is healthy and all databases are sync’ed
  • Service pack the current version of SQL – so i will support server 2019
  • Node Y – Upgrade 2012R2 to 2016 – Check node is still able to join cluster
  • Node Z – Upgrade 2012R2 to 2016 – Check node is still able to join cluster
  • Node X – Failover SQL AAG and cluster core resources to another node (e.g. Node Z)
  • Node X – Upgrade 2012R2 to 2016 – Check node is still able to join cluster
  • Upgrade cluster functional level
  • Node X – Upgrade 2016 to 2019 – Check node is still able to join cluster
  • Verify SQL AAG is healthy and all databases are sync’ed
  • Node X – Upgrade SQL 20xx to SQL 2019 with current CU
  • Node X – Failover SQL AAG and cluster core resources back to node Z
    • Once you do this – you will not be able to fail over to other nodes until they are also upgraded. Replication will also stop to “lower” version nodes – don’t freak out when you see this (like i did on my first upgrade!)
  • Node Y – Upgrade 2016 to 2019 – Check node is still able to join cluster
  • Node Y – Upgrade SQL 20xx to SQL 2019 with current CU
  • Node Z – Upgrade 2016 to 2019 – Check node is still able to join cluster
  • Node Z – Upgrade SQL 20xx to SQL 2019 with current CU
  • Upgrade cluster functional level
  • On each database on Node Y an Node Z, you will need to go into SQL management studio and select “resume data movement” – this tells SQL to try again – which will now work – as the same version of SQL is in use across the cluster

 

Microsoft Edge works best with the latest Windows Updates

When installing, seemingly randomly i will get the following in the application event log and msi log for CrEdge

Microsoft Edge works best with the latest Windows Updates. Once you download updates and restart your device, rerun the installer.

This is particularly frustrating as

  • The device has all current Windows updates applied
  • The install works on thousands of other machines – but just has a smattering where it doesn’t with this error
  • The error is in no way actually helpful… it doesn’t specify what updates i am supposedly missing… so doesn’t actually help with troubleshooting in anyway. Not quite as bad as “the task failed successfully” – but not far off.

 

Fortunately, Dr google provided some assistance

Microsoft Edge install issues on some computers
byu/jasonin951 inMicrosoftEdge

 

Microsoft Edge works best with the latest Windows Updates Error
byu/xxx59712 inedge

 

The answer, for me was setting the following reg key in the task sequence prior to Edge installing

reg add HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\EdgeUpdate /v Allowsxs /t REG_DWORD /d 1

 

The idea of preventing edge installs without providing an actual reason – genuinely bizarre behaviour by MS here.

 

Windows Server 2019 networking fails on Hyper-V 2012 (non R2) host once CU 10-2022 applied

Yes, i know… no-one should be running Server 2012 anymore…. but due to this clients ludicrous outsourcing agreement, upgrading servers is simply too expensive – so they still have some 2012 (non-R2) Hyper-V hosts.

Anyway – quite a specific issue

In short – A server 2019 guest sitting on a Hyper-V 2012 (non-R2) host will have issues with networking once Server 2019 CU 10-2022 is applied.

There is no workaround that I’m aware of – and the solution in this case was simply to move to a different host.

Disabling “link Azure Active Directory accounts to personal Microsoft accounts”

In some type of bizarre alternate reality, linking a corporate Microsoft account to a personal Microsoft account so the user can earn “Microsoft Rewards points for Microsoft Bing searches done in their browser or Windows search box while signed in with their AAD account” is seen as a good option to have – despite the security concerns and fact that almost no-one uses Bing or knows WTF “Microsoft rewards” are.

For the rest of us – fortunately we can disable this insanity using an MS provided script from https://download.microsoft.com/download/2/4/5/245c3b59-a897-4ee1-a24d-e0ead9007603/AccountLinkingDisable.ps1

General FAQ – https://www.microsoft.com/en-us/bing/account-linking-admin-faq

Ctrl + Alt + Del in nested RDP/VM sessions

If your anything like me – the amount of environments that you are now accessing via a variety of nested citrix/RDP/VPN/Hyper-V Console/JumpBox’s  etc – well…. its now 100% of what you deal with.

 

One of the pains of nested sessions (apart from the clipboard) is pressing Ctrl + Alt + Del for tasks such as changing your password.

A common method around this is to use the on-screen keyboard

  • launch “osk.exe” from a command line
  • press Ctrl + alt – and then use your mouse to press “del”

However, i’ve always found that little painful – and a few years ago i found a direct command line, then lost it again – and have now found it again! So i am now recording it here – for future me – and anyone else

C:\Windows\explorer.exe shell:::{2559a1f2-21d7-11d4-bdaf-00c04f60b9f0}

RDS Farm HA and Microsoft OLE DB Driver for SQL Server/SQL native client

This post is written in August 2022 – and may or may not current by the time you read this!

 

Recently i was was refreshing a larg-ish RDP farm to newer OS’s/version ready for an upgrade of the core business application.

As part of this, new Server 2019-based RD Brokers were to be setup – and setting up the HA proved to be more challenging than it has been in the past.

And before you say it – the vendor of the business application presented by RDS doesn’t support AVD/WVD – so “just put it in the cloud” is not an option.

 

Most of the articles around the web talk about using the SQL native client – which is now deprecated. Then in turn, recommends the OLE DB Driver for SQL.

Trying to use the OLE DB driver resulted in may lost hours for myself, the clients best tech and their SQL guru.

The most current MS document that we could find, talks about using ODBC 13 – which according to these tables, is not supported on anything above SQL 2017 and Windows Server 2012…. so not exactly current.

Following a few links – we can see that the current ODBC driver version (at time of writing) is 18.1.1.1 – available from https://docs.microsoft.com/en-us/sql/connect/odbc/download-odbc-driver-for-sql-server?view=sql-server-ver16  which supports current OS’s and SQL versions.

 

Our findings were:

  1. We could not get the OLE DB driver to work… we suspect it was due to to security issues – but do not know for sure – as the HA setup for RDCB does not appear to have any verbose logs that i could find.
  2. The ODBC driver v13 worked fine with a connection string of DRIVER={ODBC Driver 13 for SQL Server};SERVER=tcp:server.com.au,1433;APP=Remote Desktop Services Connection Broker;Trusted_Connection=Yes;Database=RDFarm;Encrypt=yes;TrustServerCertificate=yes;Connection Timeout=30 – however, given that ODBC driver v13 was not supported on the platforms we were deploying with (Server 2019 and SQL 2019) – this made us uncomfortable
  3. The ODBC driver v18 worked fine with a connection string of DRIVER={ODBC Driver 18 for SQL Server};SERVER=tcp:server.com.au,1433;APP=Remote Desktop Services Connection Broker;Trusted_Connection=Yes;Database=RDFarm;Encrypt=yes;TrustServerCertificate=yes;Connection Timeout=30 – This version of the driver has current version OS and SQL support – but was not mentioned in the RDP broker article. Some of you may be saying “so what”…. and the reality is when you have many thousands of users reliant on a core business app 24×7 – little things like that become important. Its incredibly difficult these days to get onto someone within MS support that even knows what your talking about – so saying you have deviated from an official document only makes things even more difficult.
  4. I have lodged a request for modification/clarification to the official doc here – and we’ll see what happens with that.

Changing the default sign in for Azure AD joined machines

It seems that many people have a beef with MS for forcing Windows hello for business onto machines where they are joined to Azure AD – as per https://feedback.azure.com/forums/169401-azure-active-directory/suggestions/17762902-allow-for-deactivating-windows-hello-and-set-up

It can be disabled via intune….but what if you don’t have intune – and cant manage the device until its enrolled with your 3rd party solution – after the user has been forcibly enrolled into WHfB ?

Well, first off, lets cover the Why…. why would you want to do this ?

The main reason is SSO for on premise resources. If you have a hybrid environment and AzureAD joined machines sometimes come into the corporate network or VPN in – you dont want them to get prompted every time they try to access a resource. This can be achieved with WHfB – but is not the simplest thing to setup – https://docs.microsoft.com/en-us/windows/security/identity-protection/hello-for-business/hello-hybrid-aadj-sso-base

AzureAD SSO to on premise resources “just works” with a few caveats – as per this horrendously badly written MS doc – https://docs.microsoft.com/en-us/azure/active-directory/devices/azuread-join-sso – i have queried the statement

Azure AD joined devices have no knowledge about your on-premises AD environment because they aren’t joined to it. However, you can provide additional information about your on-premises AD to these devices with Azure AD Connect.

with MS support – but they don’t seem to see anything wrong with the implication that additional information is required, but not specifying what that additional information is anywhere.

 

Anyhoo – back to the topic at hand

While there is no easy way to un-enroll your users from WHfB, you can set password as the default logon option – which will make your SSO journey much simpler… then you can work in the background on getting WHfB SSO going.

In short, loop through each SID listed under

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Authentication\LogonUI\UserTile

and set the value to {60B78E88-EAD8-445C-9CFD-0B87F74EA6CD}

which is “password”.

Password logons will then become default for each user.

Windows 11 start menu configuration

Following on from the dumpster fire that is Windows 10 start menu and taskbar configuration – The win 11 team have really stepped up and made this even worse… why? because “fuck you thats why” seems to be the best explanation.

Start menu and taskbar config is now 3 different configuration settings

 

Start menu

The start menu is now a json file stored under “%LocalAppdata%\Packages\Microsoft.Windows.StartMenuExperienceHost_cw5n1h2txyewy\LocalState” – and appears to be encrypted to a level well past my understanding – so it not manually editable… (after all, what enterprise admin would want to control their users start menu right?!?!)

in order to customise this for the default user in a TS, we can

  • Customise the start menu as we want it to look
  • copy out “%LocalAppdata%\Packages\Microsoft.Windows.StartMenuExperienceHost_cw5n1h2txyewy\LocalState\start.bin” to our SCCM package source
  • The use a command such as the following in our TS
  • xcopy “start.bin” “C:\Users\Default\AppData\Local\Packages\Microsoft.Windows.StartMenuExperienceHost_cw5n1h2txyewy\LocalState\” /y

 

Start Menu folders area

Since this isnt complex enough – another section has been added to the start menu – called “start folders”

 

 

In order to keep things consistent, this is configured at a machine level via the registry keys at HKLM\SOFTWARE\Microsoft\PolicyManager\current\device\Start\AllowPinnedFolderDocuments, AllowPinnedFolderDownloads etc

Here is a site which already has the reg entries you need listed – https://www.tenforums.com/tutorials/2192-add-remove-folders-start-list-windows-10-a-5.html 

At least these settings are easy to manage – and there are even CSP’s for them – https://docs.microsoft.com/en-us/windows/client-management/mdm/policy-csp-start 

 

TaskBar

The Win 11 taskbar is managed in the same way the Win 10 taskbar is, via an xml

https://docs.microsoft.com/en-us/windows/configuration/customize-taskbar-windows-11

its the same old xml file we used in win 10 – but the “start” section is ignored – and the taskbar section is still used.

the article talks about using group policy or Intune to deploy it – but if we only want it set as the default – and allow users to change it, use the same old way that we did with windows 10 task sequence

Powershell.exe Import-StartLayout -LayoutPath Taskbar.xml -MountPath C:\

SCCM – Config baseline to detect Windows optional features

Detecting and removing Windows 10/11 optional features via SCCM can be a bit of pain via inventory – as the class is not inventoried by default.

One alternative is using a compliance baseline – which provides a quicker turn-around for any remediation you may want to do and wont bloat your database with additional inventory.

Im going to use an example of the Windows 10 XPS printer.

  • Create a configuration item
  • Give the item a name, such as “CI – Detect XPS Printer”
  • Select your platforms – most likely Windows 10 and 11
  • Under Settings
    • Give the Setting a name: “Settings – XPS Printer Disabled”
    • Setting Type: Script
    • Data Type: String
    • Script: (Get-WindowsOptionalFeature -FeatureName Printing-XPSServices-Features -Online).State
    • Remediation script : Disable-WindowsOptionalFeature -FeatureName Printing-XPSServices-Features -Online
  • Compliance rules
    • Name: Compliance – XPS Disabled
    • Selected setting: The setting we made in the step above
    • Value: Disabled
    • Run the remediation script when this setting is noncompliant: Enabled

Then create your baseline which contains this CI and deploy to your desired collection(s).

This can obviously be adapted to work with any optional feature – Use Get-WindowsOptionalFeature -Online to help find the exact name of the feature you are looking for.