{"id":2082,"date":"2017-05-31T12:23:12","date_gmt":"2017-05-31T16:23:12","guid":{"rendered":"http:\/\/3.209.169.194\/blogs\/jonathan\/?p=2082"},"modified":"2017-05-31T12:23:12","modified_gmt":"2017-05-31T16:23:12","slug":"sql-server-2016-distributed-replay-errors","status":"publish","type":"post","link":"https:\/\/www.sqlskills.com\/blogs\/jonathan\/sql-server-2016-distributed-replay-errors\/","title":{"rendered":"SQL Server 2016 Distributed Replay Errors"},"content":{"rendered":"

If you\u2019ve tried to install and configure Distributed Replay in SQL Server 2016, I wouldn\u2019t be surprised to hear that you ran into all sorts of problems and probably didn\u2019t end up getting it to work in a multi-client setup and eventually gave up. For whatever reason, Microsoft didn\u2019t make the initial configuration of Distributed Replay in 2012, 2014 or 2016 very user friendly, and the error messages that you get when something isn\u2019t correctly configured are less than helpful.\u00a0 For example:<\/p>\n

2017-05-31 10:05:25:211 Error DReplay\u00a0\u00a0 Unexpected error occurred!<\/p>\n

Critical Error: code=[c8503012], msg=Unexpected error occurred!<\/p>\n

Security violation with invalid remote caller.<\/p>\n

Error Code: 0xC8502002<\/p><\/blockquote>\n

None of these errors help to pinpoint the cause of the problems, all of which are security\/permissions related from what I\u2019ve run into so far, but it\u2019s not easy to figure it out unless you already know a fair bit about Distributed Replay and how it SHOULD<\/strong><\/span> be configured so you can spot where problems might be and try making changes.<\/p>\n

Defaults After Installation<\/h2>\n

For my 2016 environment, I installed a Distributed Replay Controller and two separate Distributed Replay Clients, all of which used Service SIDs and were configured to use the correct controller following the installer information in my 2012 post<\/a>.\u00a0 The only difference, aside from server names, was that I didn\u2019t setup domain service accounts and let the installer setup Service SIDs for the controller and client services.\u00a0 When I start the controller service I get the following in the log using Windows Server 2016 and SQL Server 2016:<\/p>\n

2017-05-31 11:05:29:669 OPERATIONAL\u00a0 [Controller Service]\u00a0 Microsoft SQL Server Distributed Replay Controller – 13.0.1601.5.
\n2017-05-31 11:05:29:669 OPERATIONAL\u00a0 [Controller Service]\u00a0 \u00a9 Microsoft Corporation.
\n2017-05-31 11:05:29:669 OPERATIONAL\u00a0 [Controller Service]\u00a0 All rights reserved.
\n2017-05-31 11:05:29:684 OPERATIONAL\u00a0 [Controller Service]\u00a0 Current edition is: [Enterprise Edition].
\n2017-05-31 11:05:29:684 OPERATIONAL\u00a0 [Controller Service]\u00a0 The number of maximum supported client is 16.
\n2017-05-31 11:05:29:684 OPERATIONAL\u00a0 [Controller Service]\u00a0 Windows service “Microsoft SQL Server Distributed Replay Controller” has started under service account “NT SERVICE\\SQL Server Distributed Replay Controller”. Process ID is 6572.
\n2017-05-31 11:05:29:684 OPERATIONAL\u00a0 [Controller Service]\u00a0 Time Zone: Eastern Standard Time.
\n2017-05-31 11:05:29:684 OPERATIONAL\u00a0 [Common]\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Initializing dump support.
\n2017-05-31 11:05:29:684 OPERATIONAL\u00a0 [Common]\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Failed to get DmpClient. [HRESULT=0x8007007F]<\/p><\/blockquote>\n

The Failed to get DmpClient error seems to be pretty common from Google search results, but isn\u2019t actually a problem.\u00a0 So if I start the clients, I get the following in the logs:<\/p>\n

2017-05-31 11:12:16:672 OPERATIONAL\u00a0 [Client Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Microsoft SQL Server Distributed Replay Client – 13.0.1601.5.
\n2017-05-31 11:12:16:672 OPERATIONAL\u00a0 [Client Service]\u00a0\u00a0\u00a0\u00a0\u00a0 \u00a9 Microsoft Corporation.
\n2017-05-31 11:12:16:672 OPERATIONAL\u00a0 [Client Service]\u00a0\u00a0\u00a0\u00a0\u00a0 All rights reserved.
\n2017-05-31 11:12:16:672 OPERATIONAL\u00a0 [Client Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Current edition is: [Enterprise Edition].
\n2017-05-31 11:12:16:672 OPERATIONAL\u00a0 [Common]\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Initializing dump support.
\n2017-05-31 11:12:16:672 OPERATIONAL\u00a0 [Common]\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Failed to get DmpClient. [HRESULT=0x8007007F]
\n2017-05-31 11:12:16:672 OPERATIONAL\u00a0 [Client Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Windows service “Microsoft SQL Server Distributed Replay Client” has started under service account “NT SERVICE\\SQL Server Distributed Replay Client”. Process ID is 7008.
\n2017-05-31 11:12:16:672 OPERATIONAL\u00a0 [Client Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Time Zone: Eastern Standard Time.
\n2017-05-31 11:12:16:688 OPERATIONAL\u00a0 [Client Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Controller name is “SQL2K16-AG01”.
\n2017-05-31 11:12:16:688 OPERATIONAL\u00a0 [Client Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Working directory is “C:\\Program Files (x86)\\Microsoft SQL Server\\130\\Tools\\DReplayClient\\WorkingDir”.
\n2017-05-31 11:12:16:688 OPERATIONAL\u00a0 [Client Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Result directory is “C:\\Program Files (x86)\\Microsoft SQL Server\\130\\Tools\\DReplayClient\\ResultDir”.
\n2017-05-31 11:12:16:688 OPERATIONAL\u00a0 [Client Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Heartbeat Frequency(ms): 3000
\n2017-05-31 11:12:16:688 OPERATIONAL\u00a0 [Client Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Heartbeats Before Timeout: 3<\/p><\/blockquote>\n

Notice that the last line DOES NOT<\/strong> say it was registered with the controller.\u00a0 It should say Registered with controller \u201cSQL2K16-AG01\u201d if it had successfully registered, but it doesn\u2019t so something isn\u2019t allowing the client to register correctly with the controller.\u00a0 To prove this, if we attempt a replay operation using the controller, it will output the following:<\/p>\n

\"image\"<\/a><\/p>\n

C:\\DRUDemo>dreplay replay -s SQL2K16-AG01 -w “SQL2K16-AG02, SQL2K16-AG03” -f 10 -d “C:\\DRUDemo\\ReplayFiles” -o -c “c:\\DRUDemo\\DReplay.Exe.Replay.config”<\/em><\/p><\/blockquote>\n

2017-05-31 11:14:24:467 Error DReplay\u00a0\u00a0 The client ‘SQL2K16-AG02’ is not a registered distributed replay client. Make sure that the SQL Server Distributed Replay Client services is running on ‘SQL2K16-AG02’, and that the client is registered with controller ‘localhost’.<\/em><\/p><\/blockquote>\n

So this confirms that out-of-the-box 2016 DRU won\u2019t work and permissions changes will be required to make it work properly.<\/p>\n

Configuring Component Services Permissions<\/h2>\n

On the Distributed Replay Controller machine, permissions need to be set in Component Services to allow the Distributed Replay Client Service accounts Launch and Activate permissions remotely on the COM component. The service accounts also need to be in the Distributed COM Users group in Windows. <\/strong> So in Component Services, expand Computers > My Computer > DCOM Config > DReplayController and right-click and open the Properties for the COM Component.<\/p>\n

\"image\"<\/a><\/p>\n

Edit the Launch and Activation Permissions and add the Service Account for the clients, in this case because a Service SID is being used, the computer account from Active Directory for each client machine, and allow Local Launch, Remote Launch, Local Activation and Remote Activation.\u00a0 Then edit the Access permissions and set Local Access and Remote Access for the Service accounts again.<\/p>\n

\"image\"<\/a>\"image\"<\/a><\/p>\n

Now as I mentioned above, the service accounts also need to be in the Distributed COM Users group in Windows. <\/strong>So make sure that the service accounts have been added to that group, and restart the services on the controller and clients. Checking the Client log file should now show:<\/p>\n

2017-05-31 11:20:27:454 OPERATIONAL\u00a0 [Client Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Microsoft SQL Server Distributed Replay Client – 13.0.1601.5.
\n2017-05-31 11:20:27:454 OPERATIONAL\u00a0 [Client Service]\u00a0\u00a0\u00a0\u00a0\u00a0 \u00a9 Microsoft Corporation.
\n2017-05-31 11:20:27:454 OPERATIONAL\u00a0 [Client Service]\u00a0\u00a0\u00a0\u00a0\u00a0 All rights reserved.
\n2017-05-31 11:20:27:454 OPERATIONAL\u00a0 [Client Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Current edition is: [Enterprise Edition].
\n2017-05-31 11:20:27:454 OPERATIONAL\u00a0 [Common]\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Initializing dump support.
\n2017-05-31 11:20:27:454 OPERATIONAL\u00a0 [Common]\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Failed to get DmpClient. [HRESULT=0x8007007F]
\n2017-05-31 11:20:27:454 OPERATIONAL\u00a0 [Client Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Windows service “Microsoft SQL Server Distributed Replay Client” has started under service account “NT SERVICE\\SQL Server Distributed Replay Client”. Process ID is 6172.
\n2017-05-31 11:20:27:454 OPERATIONAL\u00a0 [Client Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Time Zone: Eastern Standard Time.
\n2017-05-31 11:20:27:454 OPERATIONAL\u00a0 [Client Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Controller name is “SQL2K16-AG01”.
\n2017-05-31 11:20:27:454 OPERATIONAL\u00a0 [Client Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Working directory is “C:\\Program Files (x86)\\Microsoft SQL Server\\130\\Tools\\DReplayClient\\WorkingDir”.
\n2017-05-31 11:20:27:454 OPERATIONAL\u00a0 [Client Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Result directory is “C:\\Program Files (x86)\\Microsoft SQL Server\\130\\Tools\\DReplayClient\\ResultDir”.
\n2017-05-31 11:20:27:454 OPERATIONAL\u00a0 [Client Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Heartbeat Frequency(ms): 3000
\n2017-05-31 11:20:27:454 OPERATIONAL\u00a0 [Client Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Heartbeats Before Timeout: 3
\n2017-05-31 11:21:21:798 OPERATIONAL\u00a0 [Client Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Registered with controller “SQL2K16-AG01”.<\/p><\/blockquote>\n

However, when we try a replay operation now, we get:<\/p>\n

C:\\DRUDemo>dreplay replay -s SQL2K16-AG01 -w “SQL2K16-AG02, SQL2K16-AG03” -f 10 -d “C:\\DRUDemo\\ReplayFiles” -o -c “c:\\DRUDemo\\DReplay.Exe.Replay.config”<\/em><\/p>\n

2017-05-31 11:21:33:203 Error DReplay\u00a0\u00a0 Unexpected error occurred!<\/span><\/strong><\/em><\/p><\/blockquote>\n

Yep that\u2019s really helpful, so lets go check the controller log and see what it has:<\/p>\n

2017-05-31 11:21:31:374 OPERATIONAL\u00a0 [Controller Service]\u00a0 Event replay in progress. Detailed options:
\n2017-05-31 11:21:31:374 OPERATIONAL\u00a0 [Controller Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Target DB Server: [SQL2K16-AG01].
\n2017-05-31 11:21:31:374 OPERATIONAL\u00a0 [Controller Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Controller Working Directory: [C:\\DRUDemo\\ReplayFiles].
\n2017-05-31 11:21:31:374 OPERATIONAL\u00a0 [Controller Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Generate Result Trace: [Yes].
\n2017-05-31 11:21:31:374 OPERATIONAL\u00a0 [Controller Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Sequencing Mode: [SYNC].
\n2017-05-31 11:21:31:374 OPERATIONAL\u00a0 [Controller Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Connect Time Scale: [100].
\n2017-05-31 11:21:31:374 OPERATIONAL\u00a0 [Controller Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Think Time Scale: [100].
\n2017-05-31 11:21:31:374 OPERATIONAL\u00a0 [Controller Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Healthmon Polling Interval: [60].
\n2017-05-31 11:21:31:374 OPERATIONAL\u00a0 [Controller Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Query Timeout: [3600].
\n2017-05-31 11:21:31:374 OPERATIONAL\u00a0 [Controller Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Data Provider Type: [ODBC].
\n2017-05-31 11:21:31:374 OPERATIONAL\u00a0 [Controller Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Threads Per Client: [255].
\n2017-05-31 11:21:31:374 OPERATIONAL\u00a0 [Controller Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Record Row Count: [Yes].
\n2017-05-31 11:21:31:374 OPERATIONAL\u00a0 [Controller Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Record Result Set: [No].
\n2017-05-31 11:21:31:374 OPERATIONAL\u00a0 [Controller Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Connection Pooling Enabled: [Yes].
\n2017-05-31 11:21:31:374 OPERATIONAL\u00a0 [Controller Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Stress Scale Granularity: [Connection].
\n2017-05-31 11:21:31:374 OPERATIONAL\u00a0 [Controller Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Replay Clients: [SQL2K16-AG02, SQL2K16-AG03].
\n2017-05-31 11:21:33:203 CRITICAL\u00a0\u00a0\u00a0\u00a0 [Controller Service]\u00a0 **** Critical Error ****<\/strong><\/span>
\n2017-05-31 11:21:33:203 CRITICAL\u00a0\u00a0\u00a0\u00a0 [Controller Service]\u00a0 Machine Name: SQL2K16-AG01
\n2017-05-31 11:21:33:203 CRITICAL\u00a0\u00a0\u00a0\u00a0 [Controller Service]\u00a0 Error Code: 0xC8502002<\/strong><\/span>
\n2017-05-31 11:21:33:203 OPERATIONAL\u00a0 [Controller Service]\u00a0 Event replay completed.
\n2017-05-31 11:21:33:203 OPERATIONAL\u00a0 [Controller Service]\u00a0 Elapsed time: 0 day(s), 0 hour(s), 0 minute(s), 1 second(s).<\/p><\/blockquote>\n

You can try and Google\/Bing that error code, hopefully you already did and it brought you to this blog post.\u00a0 So lets go back and check the client logs again, and we find these added messages:<\/p>\n

2017-05-31 11:21:33:189 CRITICAL\u00a0\u00a0\u00a0\u00a0 [Client Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Security violation with invalid remote caller.
\n2017-05-31 11:21:33:189 CRITICAL\u00a0\u00a0\u00a0\u00a0 [Client Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Caller auth level is 2.
\n2017-05-31 11:21:33:189 CRITICAL\u00a0\u00a0\u00a0\u00a0 [Client Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Caller impersonation level is 1.
\n2017-05-31 11:21:33:189 CRITICAL\u00a0\u00a0\u00a0\u00a0 [Client Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Caller identity is SQLSKILLSDEMOS\\SQL2K16-AG01$.
\n2017-05-31 11:21:33:189 CRITICAL\u00a0\u00a0\u00a0\u00a0 [Client Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Controller account is NT SERVICE\\SQL Server Distributed Replay Controller.<\/p><\/blockquote>\n

So this points to another security issue, but I wasn\u2019t sure how to go about troubleshooting this further using s Service SID, so at this point I changed from Service SIDs to Active Directory User accounts to run the services, DReplayClient for the clients and DReplayController for the controller.\u00a0 I reset all the permissions in Component Services on the controller machine and assigned the DReplayClient account to the Distributed COM Users group on the controller machine and gave it another shot.<\/p>\n

\"image\"<\/a><\/p>\n

C:\\DRUDemo>dreplay replay -s SQL2K16-AG01 -w “SQL2K16-AG02, SQL2K16-AG03” -f 10 -d “C:\\DRUDemo\\ReplayFiles” -o -c “c:\\DRUDemo\\DReplay.Exe.Replay.config”<\/em><\/p>\n

2017-05-31 11:37:51:189 Info DReplay\u00a0\u00a0\u00a0 Dispatching in progress.
\n2017-05-31 11:37:51:189 Info DReplay\u00a0\u00a0\u00a0 0 events have been dispatched.
\n2017-05-31 11:37:58:892 Info DReplay\u00a0\u00a0\u00a0 Dispatching has completed.
\n2017-05-31 11:37:58:892 Info DReplay\u00a0\u00a0\u00a0 0 events dispatched in total.
\n2017-05-31 11:37:58:892 Info DReplay\u00a0\u00a0\u00a0 Elapsed time: 0 day(s), 0 hour(s), 0 minute(s), 0 second(s).
\n2017-05-31 11:37:58:892 Info DReplay\u00a0\u00a0\u00a0 Event replay in progress.
\n2017-05-31 11:37:58:892 Info DReplay\u00a0\u00a0\u00a0 Event replay has completed.
\n2017-05-31 11:37:58:892 Info DReplay\u00a0\u00a0\u00a0 0 events (100 %) have been replayed in total. Pass rate 0.00 %.
\n2017-05-31 11:37:58:892 Info DReplay\u00a0\u00a0\u00a0 Elapsed time: 0 day(s), 0 hour(s), 0 minute(s), 9 second(s).
\n2017-05-31 11:37:58:892 Error DReplay\u00a0\u00a0 Unexpected error occurred!<\/p><\/blockquote>\n

Well at least this time there was slightly more progress, it attempts to begin dispatching events, but ends miserably with another not so helpful error message.\u00a0 Looking at the replay client logs, the following information is output:<\/p>\n

2017-05-31 11:35:56:969 OPERATIONAL\u00a0 [Controller Service]\u00a0 Event replay in progress. Detailed options:
\n2017-05-31 11:35:56:985 OPERATIONAL\u00a0 [Controller Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Target DB Server: [SQL2K16-AG01].
\n2017-05-31 11:35:56:985 OPERATIONAL\u00a0 [Controller Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Controller Working Directory: [C:\\DRUDemo\\ReplayFiles].
\n2017-05-31 11:35:56:985 OPERATIONAL\u00a0 [Controller Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Generate Result Trace: [Yes].
\n2017-05-31 11:35:56:985 OPERATIONAL\u00a0 [Controller Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Sequencing Mode: [SYNC].
\n2017-05-31 11:35:56:985 OPERATIONAL\u00a0 [Controller Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Connect Time Scale: [100].
\n2017-05-31 11:35:56:985 OPERATIONAL\u00a0 [Controller Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Think Time Scale: [100].
\n2017-05-31 11:35:56:985 OPERATIONAL\u00a0 [Controller Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Healthmon Polling Interval: [60].
\n2017-05-31 11:35:56:985 OPERATIONAL\u00a0 [Controller Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Query Timeout: [3600].
\n2017-05-31 11:35:56:985 OPERATIONAL\u00a0 [Controller Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Data Provider Type: [ODBC].
\n2017-05-31 11:35:56:985 OPERATIONAL\u00a0 [Controller Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Threads Per Client: [255].
\n2017-05-31 11:35:56:985 OPERATIONAL\u00a0 [Controller Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Record Row Count: [Yes].
\n2017-05-31 11:35:56:985 OPERATIONAL\u00a0 [Controller Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Record Result Set: [No].
\n2017-05-31 11:35:56:985 OPERATIONAL\u00a0 [Controller Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Connection Pooling Enabled: [Yes].
\n2017-05-31 11:35:56:985 OPERATIONAL\u00a0 [Controller Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Stress Scale Granularity: [Connection].
\n2017-05-31 11:35:56:985 OPERATIONAL\u00a0 [Controller Service]\u00a0\u00a0\u00a0\u00a0\u00a0 Replay Clients: [SQL2K16-AG02, SQL2K16-AG03].
\n2017-05-31 11:35:59:048 OPERATIONAL\u00a0 [Controller Service]\u00a0 Event dispatch in progress.
\n2017-05-31 11:36:05:766 OPERATIONAL\u00a0 [Controller Service]\u00a0 Event replay completed.
\n2017-05-31 11:36:05:766 OPERATIONAL\u00a0 [Controller Service]\u00a0 Elapsed time: 0 day(s), 0 hour(s), 0 minute(s), 8 second(s).<\/p><\/blockquote>\n

Not much help there either, and this is where I would expect that most people would end up giving up because there is nothing really actionable here at all.<\/p>\n

Client Service Account Permissions on Target SQL Server<\/h2>\n

While nothing is documented about changes in Distributed Replay behavior in SQL Server 2016, this last error is different than the behavior of previous versions of Distributed Replay.\u00a0 The problem is that the DReplayClient service account doesn\u2019t have permissions in the target SQL Server to connect.\u00a0 To prove this, here is a trace capture of User Error Message events from the last replay operation attempt:<\/p>\n

\"image\"<\/a><\/p>\n

Each of the Replay clients is attempting to connect to the target server and failing.\u00a0 If we add the DReplayClient login to the target SQL Server and retry the reply, everything checks out and it actually begins to dispatch the events for the replay operation:<\/p>\n

\"image\"<\/a><\/p>\n

C:\\DRUDemo>dreplay replay -s SQL2K16-AG01 -w “SQL2K16-AG02, SQL2K16-AG03” -f 10 -d “C:\\DRUDemo\\ReplayFiles” -o -c “c:\\DRUDemo\\DReplay.Exe.Replay.config”<\/em><\/p>\n

2017-05-31 11:45:14:376 Info DReplay\u00a0\u00a0\u00a0 Dispatching in progress.
\n2017-05-31 11:45:14:376 Info DReplay\u00a0\u00a0\u00a0 0 events have been dispatched.
\n2017-05-31 11:45:24:377 Info DReplay\u00a0\u00a0\u00a0 30753 events have been dispatched.
\n2017-05-31 11:45:34:377 Info DReplay\u00a0\u00a0\u00a0 68262 events have been dispatched.
\n2017-05-31 11:45:44:377 Info DReplay\u00a0\u00a0\u00a0 106677 events have been dispatched.
\n2017-05-31 11:45:54:393 Info DReplay\u00a0\u00a0\u00a0 144226 events have been dispatched.
\n2017-05-31 11:46:04:408 Info DReplay\u00a0\u00a0\u00a0 183595 events have been dispatched.
\n2017-05-31 11:46:14:424 Info DReplay\u00a0\u00a0\u00a0 221378 events have been dispatched.
\n2017-05-31 11:46:24:424 Info DReplay\u00a0\u00a0\u00a0 257754 events have been dispatched.
\n2017-05-31 11:46:34:455 Info DReplay\u00a0\u00a0\u00a0 298436 events have been dispatched.
\n2017-05-31 11:46:44:471 Info DReplay\u00a0\u00a0\u00a0 336026 events have been dispatched.
\n2017-05-31 11:46:54:471 Info DReplay\u00a0\u00a0\u00a0 373717 events have been dispatched.
\n2017-05-31 11:47:04:486 Info DReplay\u00a0\u00a0\u00a0 410378 events have been dispatched.
\n2017-05-31 11:47:14:502 Info DReplay\u00a0\u00a0\u00a0 449949 events have been dispatched.
\n2017-05-31 11:47:24:518 Info DReplay\u00a0\u00a0\u00a0 486431 events have been dispatched.
\n2017-05-31 11:47:34:533 Info DReplay\u00a0\u00a0\u00a0 526228 events have been dispatched.
\n2017-05-31 11:47:44:549 Info DReplay\u00a0\u00a0\u00a0 563484 events have been dispatched.
\n2017-05-31 11:47:48:361 Info DReplay\u00a0\u00a0\u00a0 Dispatching has completed.
\n2017-05-31 11:47:48:361 Info DReplay\u00a0\u00a0\u00a0 573630 events dispatched in total.
\n2017-05-31 11:47:48:361 Info DReplay\u00a0\u00a0\u00a0 Elapsed time: 0 day(s), 0 hour(s), 2 minute(s), 35 second(s).
\n2017-05-31 11:47:48:361 Info DReplay\u00a0\u00a0\u00a0 Event replay in progress.<\/p><\/blockquote>\n

Conclusion<\/h2>\n

Permissions, permissions, permissions\u2026\u00a0 While having some of the worst error messages imaginable, the problems with getting Distributed Replay in SQL Server 2016 configured have so far boiled down to permissions issues and ensuring that the permissions for the service accounts are set correctly on the Controller machine, and on the Target Server for the replay operation should resolve the issues.\u00a0 Don\u2019t forget to configure Firewall rules to allow access to the network appropriately as described in the original 2012 DRU post<\/a> I wrote a few years back.\u00a0\u00a0 Hopefully this post will save someone the trouble of trying to figure this all out blindly.<\/p>\n","protected":false},"excerpt":{"rendered":"

If you\u2019ve tried to install and configure Distributed Replay in SQL Server 2016, I wouldn\u2019t be surprised to hear that you ran into all sorts of problems and probably didn\u2019t end up getting it to work in a multi-client setup and eventually gave up. For whatever reason, Microsoft didn\u2019t make the initial configuration of Distributed […]<\/p>\n","protected":false},"author":4,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[14,15,21,35,53],"tags":[],"class_list":["post-2082","post","type-post","status-publish","format-standard","hentry","category-benchmarking","category-building-a-test-environment","category-distributed-replay","category-security","category-sql-server-2016"],"yoast_head":"\nSQL Server 2016 Distributed Replay Errors - Jonathan Kehayias<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.sqlskills.com\/blogs\/jonathan\/sql-server-2016-distributed-replay-errors\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"SQL Server 2016 Distributed Replay Errors - Jonathan Kehayias\" \/>\n<meta property=\"og:description\" content=\"If you\u2019ve tried to install and configure Distributed Replay in SQL Server 2016, I wouldn\u2019t be surprised to hear that you ran into all sorts of problems and probably didn\u2019t end up getting it to work in a multi-client setup and eventually gave up. For whatever reason, Microsoft didn\u2019t make the initial configuration of Distributed […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.sqlskills.com\/blogs\/jonathan\/sql-server-2016-distributed-replay-errors\/\" \/>\n<meta property=\"og:site_name\" content=\"Jonathan Kehayias\" \/>\n<meta property=\"article:published_time\" content=\"2017-05-31T16:23:12+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.sqlskills.com\/blogs\/jonathan\/wp-content\/uploads\/2017\/05\/image_thumb-6.png\" \/>\n<meta name=\"author\" content=\"Jonathan Kehayias\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Jonathan Kehayias\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"11 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.sqlskills.com\\\/blogs\\\/jonathan\\\/sql-server-2016-distributed-replay-errors\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.sqlskills.com\\\/blogs\\\/jonathan\\\/sql-server-2016-distributed-replay-errors\\\/\"},\"author\":{\"name\":\"Jonathan Kehayias\",\"@id\":\"https:\\\/\\\/www.sqlskills.com\\\/blogs\\\/jonathan\\\/#\\\/schema\\\/person\\\/01c10d94f3648654ef706d5e6305f69c\"},\"headline\":\"SQL Server 2016 Distributed Replay Errors\",\"datePublished\":\"2017-05-31T16:23:12+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.sqlskills.com\\\/blogs\\\/jonathan\\\/sql-server-2016-distributed-replay-errors\\\/\"},\"wordCount\":2285,\"commentCount\":4,\"image\":{\"@id\":\"https:\\\/\\\/www.sqlskills.com\\\/blogs\\\/jonathan\\\/sql-server-2016-distributed-replay-errors\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.sqlskills.com\\\/blogs\\\/jonathan\\\/wp-content\\\/uploads\\\/2017\\\/05\\\/image_thumb-6.png\",\"articleSection\":[\"Benchmarking\",\"Building a Test Environment\",\"Distributed Replay\",\"Security\",\"SQL Server 2016\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/www.sqlskills.com\\\/blogs\\\/jonathan\\\/sql-server-2016-distributed-replay-errors\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.sqlskills.com\\\/blogs\\\/jonathan\\\/sql-server-2016-distributed-replay-errors\\\/\",\"url\":\"https:\\\/\\\/www.sqlskills.com\\\/blogs\\\/jonathan\\\/sql-server-2016-distributed-replay-errors\\\/\",\"name\":\"SQL Server 2016 Distributed Replay Errors - Jonathan Kehayias\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.sqlskills.com\\\/blogs\\\/jonathan\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.sqlskills.com\\\/blogs\\\/jonathan\\\/sql-server-2016-distributed-replay-errors\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.sqlskills.com\\\/blogs\\\/jonathan\\\/sql-server-2016-distributed-replay-errors\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.sqlskills.com\\\/blogs\\\/jonathan\\\/wp-content\\\/uploads\\\/2017\\\/05\\\/image_thumb-6.png\",\"datePublished\":\"2017-05-31T16:23:12+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/www.sqlskills.com\\\/blogs\\\/jonathan\\\/#\\\/schema\\\/person\\\/01c10d94f3648654ef706d5e6305f69c\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.sqlskills.com\\\/blogs\\\/jonathan\\\/sql-server-2016-distributed-replay-errors\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.sqlskills.com\\\/blogs\\\/jonathan\\\/sql-server-2016-distributed-replay-errors\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.sqlskills.com\\\/blogs\\\/jonathan\\\/sql-server-2016-distributed-replay-errors\\\/#primaryimage\",\"url\":\"https:\\\/\\\/www.sqlskills.com\\\/blogs\\\/jonathan\\\/wp-content\\\/uploads\\\/2017\\\/05\\\/image_thumb-6.png\",\"contentUrl\":\"https:\\\/\\\/www.sqlskills.com\\\/blogs\\\/jonathan\\\/wp-content\\\/uploads\\\/2017\\\/05\\\/image_thumb-6.png\",\"width\":644,\"height\":339},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.sqlskills.com\\\/blogs\\\/jonathan\\\/sql-server-2016-distributed-replay-errors\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.sqlskills.com\\\/blogs\\\/jonathan\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Benchmarking\",\"item\":\"https:\\\/\\\/www.sqlskills.com\\\/blogs\\\/jonathan\\\/category\\\/benchmarking\\\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"SQL Server 2016 Distributed Replay Errors\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.sqlskills.com\\\/blogs\\\/jonathan\\\/#website\",\"url\":\"https:\\\/\\\/www.sqlskills.com\\\/blogs\\\/jonathan\\\/\",\"name\":\"Jonathan Kehayias - The Rambling DBA\",\"description\":\"The Rambling DBA\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.sqlskills.com\\\/blogs\\\/jonathan\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.sqlskills.com\\\/blogs\\\/jonathan\\\/#\\\/schema\\\/person\\\/01c10d94f3648654ef706d5e6305f69c\",\"name\":\"Jonathan Kehayias\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/86630e27f5deecc5c393ea57fc7c3b6a068949f4fd6b5309f81de5a276f12855?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/86630e27f5deecc5c393ea57fc7c3b6a068949f4fd6b5309f81de5a276f12855?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/86630e27f5deecc5c393ea57fc7c3b6a068949f4fd6b5309f81de5a276f12855?s=96&d=mm&r=g\",\"caption\":\"Jonathan Kehayias\"},\"sameAs\":[\"http:\\\/\\\/3.209.169.194\\\/blogs\\\/jonathan\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"SQL Server 2016 Distributed Replay Errors - Jonathan Kehayias","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.sqlskills.com\/blogs\/jonathan\/sql-server-2016-distributed-replay-errors\/","og_locale":"en_US","og_type":"article","og_title":"SQL Server 2016 Distributed Replay Errors - Jonathan Kehayias","og_description":"If you\u2019ve tried to install and configure Distributed Replay in SQL Server 2016, I wouldn\u2019t be surprised to hear that you ran into all sorts of problems and probably didn\u2019t end up getting it to work in a multi-client setup and eventually gave up. For whatever reason, Microsoft didn\u2019t make the initial configuration of Distributed […]","og_url":"https:\/\/www.sqlskills.com\/blogs\/jonathan\/sql-server-2016-distributed-replay-errors\/","og_site_name":"Jonathan Kehayias","article_published_time":"2017-05-31T16:23:12+00:00","og_image":[{"url":"https:\/\/www.sqlskills.com\/blogs\/jonathan\/wp-content\/uploads\/2017\/05\/image_thumb-6.png","type":"","width":"","height":""}],"author":"Jonathan Kehayias","twitter_misc":{"Written by":"Jonathan Kehayias","Est. reading time":"11 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.sqlskills.com\/blogs\/jonathan\/sql-server-2016-distributed-replay-errors\/#article","isPartOf":{"@id":"https:\/\/www.sqlskills.com\/blogs\/jonathan\/sql-server-2016-distributed-replay-errors\/"},"author":{"name":"Jonathan Kehayias","@id":"https:\/\/www.sqlskills.com\/blogs\/jonathan\/#\/schema\/person\/01c10d94f3648654ef706d5e6305f69c"},"headline":"SQL Server 2016 Distributed Replay Errors","datePublished":"2017-05-31T16:23:12+00:00","mainEntityOfPage":{"@id":"https:\/\/www.sqlskills.com\/blogs\/jonathan\/sql-server-2016-distributed-replay-errors\/"},"wordCount":2285,"commentCount":4,"image":{"@id":"https:\/\/www.sqlskills.com\/blogs\/jonathan\/sql-server-2016-distributed-replay-errors\/#primaryimage"},"thumbnailUrl":"https:\/\/www.sqlskills.com\/blogs\/jonathan\/wp-content\/uploads\/2017\/05\/image_thumb-6.png","articleSection":["Benchmarking","Building a Test Environment","Distributed Replay","Security","SQL Server 2016"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.sqlskills.com\/blogs\/jonathan\/sql-server-2016-distributed-replay-errors\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.sqlskills.com\/blogs\/jonathan\/sql-server-2016-distributed-replay-errors\/","url":"https:\/\/www.sqlskills.com\/blogs\/jonathan\/sql-server-2016-distributed-replay-errors\/","name":"SQL Server 2016 Distributed Replay Errors - Jonathan Kehayias","isPartOf":{"@id":"https:\/\/www.sqlskills.com\/blogs\/jonathan\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.sqlskills.com\/blogs\/jonathan\/sql-server-2016-distributed-replay-errors\/#primaryimage"},"image":{"@id":"https:\/\/www.sqlskills.com\/blogs\/jonathan\/sql-server-2016-distributed-replay-errors\/#primaryimage"},"thumbnailUrl":"https:\/\/www.sqlskills.com\/blogs\/jonathan\/wp-content\/uploads\/2017\/05\/image_thumb-6.png","datePublished":"2017-05-31T16:23:12+00:00","author":{"@id":"https:\/\/www.sqlskills.com\/blogs\/jonathan\/#\/schema\/person\/01c10d94f3648654ef706d5e6305f69c"},"breadcrumb":{"@id":"https:\/\/www.sqlskills.com\/blogs\/jonathan\/sql-server-2016-distributed-replay-errors\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.sqlskills.com\/blogs\/jonathan\/sql-server-2016-distributed-replay-errors\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.sqlskills.com\/blogs\/jonathan\/sql-server-2016-distributed-replay-errors\/#primaryimage","url":"https:\/\/www.sqlskills.com\/blogs\/jonathan\/wp-content\/uploads\/2017\/05\/image_thumb-6.png","contentUrl":"https:\/\/www.sqlskills.com\/blogs\/jonathan\/wp-content\/uploads\/2017\/05\/image_thumb-6.png","width":644,"height":339},{"@type":"BreadcrumbList","@id":"https:\/\/www.sqlskills.com\/blogs\/jonathan\/sql-server-2016-distributed-replay-errors\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.sqlskills.com\/blogs\/jonathan\/"},{"@type":"ListItem","position":2,"name":"Benchmarking","item":"https:\/\/www.sqlskills.com\/blogs\/jonathan\/category\/benchmarking\/"},{"@type":"ListItem","position":3,"name":"SQL Server 2016 Distributed Replay Errors"}]},{"@type":"WebSite","@id":"https:\/\/www.sqlskills.com\/blogs\/jonathan\/#website","url":"https:\/\/www.sqlskills.com\/blogs\/jonathan\/","name":"Jonathan Kehayias - The Rambling DBA","description":"The Rambling DBA","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.sqlskills.com\/blogs\/jonathan\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.sqlskills.com\/blogs\/jonathan\/#\/schema\/person\/01c10d94f3648654ef706d5e6305f69c","name":"Jonathan Kehayias","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/86630e27f5deecc5c393ea57fc7c3b6a068949f4fd6b5309f81de5a276f12855?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/86630e27f5deecc5c393ea57fc7c3b6a068949f4fd6b5309f81de5a276f12855?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/86630e27f5deecc5c393ea57fc7c3b6a068949f4fd6b5309f81de5a276f12855?s=96&d=mm&r=g","caption":"Jonathan Kehayias"},"sameAs":["http:\/\/3.209.169.194\/blogs\/jonathan"]}]}},"_links":{"self":[{"href":"https:\/\/www.sqlskills.com\/blogs\/jonathan\/wp-json\/wp\/v2\/posts\/2082","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.sqlskills.com\/blogs\/jonathan\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.sqlskills.com\/blogs\/jonathan\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.sqlskills.com\/blogs\/jonathan\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/www.sqlskills.com\/blogs\/jonathan\/wp-json\/wp\/v2\/comments?post=2082"}],"version-history":[{"count":1,"href":"https:\/\/www.sqlskills.com\/blogs\/jonathan\/wp-json\/wp\/v2\/posts\/2082\/revisions"}],"predecessor-version":[{"id":2083,"href":"https:\/\/www.sqlskills.com\/blogs\/jonathan\/wp-json\/wp\/v2\/posts\/2082\/revisions\/2083"}],"wp:attachment":[{"href":"https:\/\/www.sqlskills.com\/blogs\/jonathan\/wp-json\/wp\/v2\/media?parent=2082"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.sqlskills.com\/blogs\/jonathan\/wp-json\/wp\/v2\/categories?post=2082"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.sqlskills.com\/blogs\/jonathan\/wp-json\/wp\/v2\/tags?post=2082"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}