(The Curious Case of… used to be part of our bi-weekly newsletter but we decided to make it a regular blog post instead so it can sometimes be more frequent. It covers something interesting one of us encountered when working with a client, doing some testing, or were asked in a random question from the community.)
Jonathan was working with a client recently who experienced a CLR assembly failure after an AG failover and needed to figure out why. They’d been testing their AG disaster recovery strategy and ran into an unexpected problem with their application which relies heavily on SQLCLR and an UNSAFE assembly that calls a web service from inside SQL Server. When they failed over their AG to their DR server, the CLR assembly failed with the following error:
An error occurred in the Microsoft .NET Framework while trying to load assembly id 65546. The server may be running out of resources, or the assembly may not be trusted with PERMISSION_SET = EXTERNAL_ACCESS or UNSAFE. Run the query again, or check documentation to see how to solve the assembly trust issues. For more information about this error: System.IO.FileLoadException: Could not load file or assembly ‘sqlclr_assemblyname, Version=188.8.131.52, Culture=neutral, PublicKeyToken=fa39443c11b12591’ or one of its dependencies. Exception from HRESULT: 0x80FC80F1
To try and bypass this error, they executed the command ALTER DATABASE <DBNAME> SET TRUSTWORTHY ON to enable the trustworthy bit on the DR server. They then tried the steps in KB Article 918040 and changed the database owner for the database on the DR server and then their CLR assembly began to work.
Well, at least it worked until they tried to failover to their original primary replica, and they again began to have problems with their CLR assembly.
Why would that be the case, especially since it originally worked on the primary replica before the DR failover?
It has to do with login SIDs in SQL Server and server-scoped permissions. The database owner is mapped inside the database by the SID of the login on the server. If the SID of the owner internally in the database doesn’t match a SID of a server principal on the server then the owner can’t be established. The dbo SID internally in the database is replicated as a part of the AG, but the server login is not. Also server scoped objects, like the asymmetric key used to sign the CLR assembly, are maintained in master, as is the login associated with that key and the EXTERNAL_ACCESS or UNSAFE ASSEMBLY permission associated with it. So to fix this issue and get rid of the TRUSTWORTHY ON bit setting for the database they had to do the following steps:
- Create the asymmetric key from the assembly DLL on the DR server.
- Change the database owner to match the SID on both servers in sys.server_principals (script the dbo login using sp_help_revlogin to transfer with SID intact to both servers)
- Create login from asymmetric key on DR server and grant UNSAFE ASSEMBLY to match primary replica
- ALTER DATABASE <DBNAME> SET TRUSTWORTHY OFF
- Fail over to test between both sites
Bottom line: it’s *always* a good idea to regularly test your failover strategy as you never know what’s going to fail when you do! Kudos to this client for doing that.
7 thoughts on “The Curious Case of… the CLR assembly failure after an AG failover”
Rather than promoting sp_help_revlogin you might be better served to use the features in the dbatools.io PowerShell project. Chrissy LeMaire has written a great blog post about the gotchas involved in sp_help_revlogin here: https://blog.netnerds.net/2016/06/its-2016-why-is-sp_help_revlogin-a-thing/
“But you don’t have the logins’ server roles, server permission sets, database roles or database permission sets.”
This is incorrect information. The database roles and permissions are mapped to the SID internally in the database, which is exactly why sp_help_rev_login exists to transfer the SID from source server to target server. If all I want is the SID to be the same on the two servers so the database permissions and ownership align, it is far easier to justify creating sp_help_rev_login, since it comes from Microsoft and is on a KB, to a client than it is to get dbatools.io installed by them.
What if the database was contained database? Would this remove the instance dependency and allow the failover without a glitch?
I am not saying this would be the solution for the particular case in the blog. Just wondering whether it could be a solution in other scenarios for the same issue.
I’ll check with Jon and let you know.
Jon says: EXTERNAL_ACCESS and UNSAFE would violate the containment and not work.
it seems this occures in my environment even SAFE is set on the assembly. DB has trustworthy off. It works till AG failover.
An error occurred in the Microsoft .NET Framework while trying to load assembly id 65542. The server may be running out of resources, or the assembly may not be trusted. Run the query again, or check documentation to see how to solve the assembly trust issues. For more information about this error:
System.IO.FileLoadException: Could not load file or assembly ‘assembly, Version=1.0.7128.39344, Culture=neutral, PublicKeyToken=42bd9c920a567c7f’ or one of its dependencies. An error relating to security occurred. (Exception from HRESULT: 0x8013150A)
Any idea would be appriciated.