(The Curious Case of… used to be part of our bi-weekly newsletter but we decided to make it a regular blog post instead so it can sometimes be more frequent. It covers something interesting one of us encountered when working with a client, doing some testing, or were asked in a random question from the community.)
Jonathan was working with a client recently who experienced a CLR assembly failure after an AG failover and needed to figure out why. They’d been testing their AG disaster recovery strategy and ran into an unexpected problem with their application which relies heavily on SQLCLR and an UNSAFE assembly that calls a web service from inside SQL Server. When they failed over their AG to their DR server, the CLR assembly failed with the following error:
An error occurred in the Microsoft .NET Framework while trying to load assembly id 65546. The server may be running out of resources, or the assembly may not be trusted with PERMISSION_SET = EXTERNAL_ACCESS or UNSAFE. Run the query again, or check documentation to see how to solve the assembly trust issues. For more information about this error: System.IO.FileLoadException: Could not load file or assembly ‘sqlclr_assemblyname, Version=220.127.116.11, Culture=neutral, PublicKeyToken=fa39443c11b12591’ or one of its dependencies. Exception from HRESULT: 0x80FC80F1
To try and bypass this error, they executed the command ALTER DATABASE <DBNAME> SET TRUSTWORTHY ON to enable the trustworthy bit on the DR server. They then tried the steps in KB Article 918040 and changed the database owner for the database on the DR server and then their CLR assembly began to work.
Well, at least it worked until they tried to failover to their original primary replica, and they again began to have problems with their CLR assembly.
Why would that be the case, especially since it originally worked on the primary replica before the DR failover?
It has to do with login SIDs in SQL Server and server-scoped permissions. The database owner is mapped inside the database by the SID of the login on the server. If the SID of the owner internally in the database doesn’t match a SID of a server principal on the server then the owner can’t be established. The dbo SID internally in the database is replicated as a part of the AG, but the server login is not. Also server scoped objects, like the asymmetric key used to sign the CLR assembly, are maintained in master, as is the login associated with that key and the EXTERNAL_ACCESS or UNSAFE ASSEMBLY permission associated with it. So to fix this issue and get rid of the TRUSTWORTHY ON bit setting for the database they had to do the following steps:
- Create the asymmetric key from the assembly DLL on the DR server.
- Change the database owner to match the SID on both servers in sys.server_principals (script the dbo login using sp_help_revlogin to transfer with SID intact to both servers)
- Create login from asymmetric key on DR server and grant UNSAFE ASSEMBLY to match primary replica
- ALTER DATABASE <DBNAME> SET TRUSTWORTHY OFF
- Fail over to test between both sites
Bottom line: it’s *always* a good idea to regularly test your failover strategy as you never know what’s going to fail when you do! Kudos to this client for doing that.