A question came up on the Data Platform MVP email list last night asking what the FCB_REPLICA_SYNC spinlock is. I answered the question and then promised to do a quick blog post, as there’s no information online about it that I could find.
In a nutshell, this spinlock is used to synchronize access to the list of pages that are present in a database snapshot, as follows:
- If a page in a database with one or more database snapshots is being updated, check each snapshot’s list to see if the page is already in the snapshot. If yes, nothing to do. If no, copy the pre-change image of the page into the snapshot.
- If a query is reading a page in the context of a database snapshot, check the list of pages to see whether to read from the snapshot or the source database.
This synchronization ensures that the correct copy of a page is read by a query using the snapshot, and that updated pages aren’t copied to the snapshot more than once.
The original question was because the person was seeing trillions of spins for the FCB_REPLICA_SYNC spinlock. That’s perfectly normal if there’s at least one database snapshot, a read workload on the snapshot, and a concurrent heavy update workload on the source database.
For example, using our sample SalesDB database (zip file here), I created this query and set it running:
UPDATE [SalesDB].[dbo].[Sales] SET [Quantity] = [Quantity] + 1;
Then I took my script to capture spinlock metrics over a period of time (see this post), changed it to capture over 20 seconds, and then ran a DBCC CHECKDB on the SalesDB database, which took 18 seconds.
The spinlock metrics returned were:
Spinlock DiffCollisions DiffSpins SpinsPerCollision DiffSleepTime DiffBackoffs
------------------------- -------------- --------- ----------------- ------------- ------------
BUF_HASH 2 500 250 0 0
DBTABLE 5 1250 250 0 0
FCB_REPLICA_SYNC 5716270 1513329500 264 0 154380
LOCK_HASH 12 3500 291 0 1
LOGCACHE_ACCESS 6 387 64 0 3
LOGFLUSHQ 4 75840 18960 0 3
LOGPOOL_HASHBUCKET 15 3750 250 0 0
LOGPOOL_SHAREDCACHEBUFFER 32 8000 250 0 0
LSLIST 8 2000 250 0 0
SOS_SCHEDULER 3 1114 371 0 0
SOS_TASK 1 356 356 0 0
You can see that even for a 20-second test, a single DBCC CHECKDB produced 1.5 billion spins on the FCB_REPLICA_SYNC spinlock.
This is perfectly normal.
One of the dangers of looking at spinlock metrics is that the numbers involved can be so high that it’s easy to convince yourself that there’s some kind of problem, especially as there’s so little information available online about what the spinlocks actually mean. The vast majority of the time, there’s no problem, but it takes a lot of internals knowledge to know what’s going on.
About the only thing I’d be concerned about is if there are multiple concurrent snapshots on a database with heavy update workload, as that will cause synchronous writes to all the snapshots when a page in the source database is first updated, slowing down the workload.
One thing you can always do if you’re interested in what a specific spinlock means is to investigate with Extended Events. There’s a whitepaper I helped review called Diagnosing and Resolving Spinlock Contention on SQL Server that you can download here. In it there’s an Extended Event session that I use to see where spinlocks backoffs occur.
Here’s the session I used for FCB_REPLICA_SYNC (which maps to the type value of 136 in sys.dm_xe_map_values):
-- Drop the session if it exists.
IF EXISTS (SELECT * FROM sys.server_event_sessions WHERE [name] = N'WatchSpinlocks')
DROP EVENT SESSION [WatchSpinlocks] ON SERVER
CREATE EVENT SESSION [WatchSpinlocks] ON SERVER
ADD EVENT [sqlos].[spinlock_backoff]
WHERE [type] = 136) -- FCB_REPLICA_SYNC only
ADD TARGET [package0].[asynchronous_bucketizer] (
SET filtering_event_name = N'sqlos.spinlock_backoff',
source_type = 1, -- source_type = 1 is an action
source = N'package0.callstack') -- bucketize on the callstack
WITH (MAX_MEMORY = 50MB, MAX_DISPATCH_LATENCY = 5 seconds)
-- Start the session
ALTER EVENT SESSION [WatchSpinlocks] ON SERVER STATE = START;
-- TF to allow call stack resolution
DBCC TRACEON (3656, -1);
-- Cause some spinlock backoffs
-- Get the callstacks from the bucketizer target
CAST ([target_data] AS XML)
FROM sys.dm_xe_session_targets [xst]
INNER JOIN sys.dm_xe_sessions [xs]
ON ([xst].[event_session_address] = [xs].[address])
WHERE [xs].[name] = N'WatchSpinlocks';
-- Stop the event session
ALTER EVENT SESSION [WatchSpinlocks] ON SERVER
STATE = STOP;
You’ll need to download the debug symbols for the build you’re using – see here for my instructions on how to do this.
I started the event session and re-ran the test. A sampling of the call stacks is below, with matching explanations.
- Pushing a page into a snapshot just before it gets modified in the source database
- Reading a page from a snapshot (in this case, from one of DBCC CHECKDB‘s parallel threads performing readahead)
- Pulling a page in to a snapshot while crash recovery is running on the new snapshot to make it a transactionally-consistent view of the source database (in this case, it’s a ‘transient’ database snapshot that DBCC CHECKDB has created)
And plenty more similar call stacks.
So there you have it. The FCB_REPLICA_SYNC spinlock is to do with database snapshot reads and writes, and high numbers around it are expected with concurrent updates in the source database and reads in the snapshot.