SQLintersection Spring 2016 – Orlando, FL – April 16-22, 2016

I’ll be presenting multiple sessions at the SQLintersection Spring 2016 conference in Orlando, FL on April 16-22, 2016. This conference has an exclusive slate of top-tier speakers, with great content, and a nice, intimate atmosphere that lets attendees get more personal interaction with the speakers compared to most other SQL Server conferences.

Here are the sessions that I will be presenting:

Workshop Session

Analyzing and Improving I/O Subsystem Performance  (All day workshop on Monday, April 18, 2016)

SQL Server is often I/O bound, but proving it to your storage or SAN administrator can be challenging! You will learn about the different types of storage that are available for SQL Server, and how to decide what type of storage to use for different SQL Server workload and file types. You will also learn useful tips and techniques for configuring your storage for the best performance and reliability for your workload. There will be extensive coverage on how to use disk benchmark tools like CrystalDiskMark 5.0 and Microsoft DiskSpd, so you can confidently understand the performance that your I/O subsystem can deliver. We’ll also cover methods to effectively measure and monitor your storage performance from an OS and SQL Server perspective so that you will have valuable information and evidence available the next time you have to discuss I/O performance with your storage administrator. You will also learn a number of valuable OS and SQL Server configuration settings that will help you get the best I/O performance possible from your storage subsystem.

 

Regular Sessions

Dr. DMV’s Troubleshooting Toolkit – Part 1

Dynamic Management Views and functions allow you to easily see exactly what is happening inside your SQL Server instances and databases with a high level of detail. You can discover your top wait types, most CPU intensive stored procedures, find missing indexes, and identify unused indexes, to name just a few examples. This session presents, demonstrates and explains a complete set of diagnostic DMV queries that you can easily use to detect and diagnose configuration and performance issues in your SQL Server instances and databases.

Dr. DMV’s Troubleshooting Toolkit – Part 2

A continuation of the first session!
Dynamic Management Views and functions allow you to easily see exactly what is happening inside your SQL Server instances and databases with a high level of detail. You can discover your top wait types, most CPU intensive stored procedures, find missing indexes, and identify unused indexes, to name just a few examples. This session presents, demonstrates and explains a complete set of diagnostic DMV queries that you can easily use to detect and diagnose configuration and performance issues in your SQL Server instances and databases.

Scaling SQL Server

SQL Server implementations can quickly evolve and become more complex, forcing DBAs and developers to think about how they can scale their solution quickly and effectively. Scaling up is relatively easy (but can be expensive), while scaling out requires significant engineering time and effort. If you suggest hardware upgrades you may be accused of simply “throwing hardware at the problem”, and if you try to scale out, you may be thwarted by a lack of development resources or 3rd party software restrictions. As your database server nears its load capacity, what can you do? This session gives you concrete, practical advice on how to deal with this situation. Starting with your present workload, configuration and hardware, we will explore how to find and alleviate bottlenecks, whether they are workload related, configuration related, or hardware related. Next, we will cover how you can decide whether you should scale up or scale out your data tier. Once that decision is made, you will learn how to scale up properly, with nearly zero down-time. If you decide to scale out, you will learn about practical, production-ready techniques such as vertical partitioning, horizontal partitioning, and data dependent routing. We will also cover how to use middle-tier caching and other application techniques to increase your overall scalability.

 

You can register for the conference and additional workshops here. If you register by March 1, 2016, you will get (depending on which package you sign up for), your choice of a Microsoft Band 2, a $200.00 hotel gift card, a Microsoft Surface 3, or a Microsoft XBOX One. You can also get a $50.00 discount on your registration if the use the discount code “Berry” when you sign up.

I think this will be a very useful and fun conference, and I hope to see you there!

SQL Server Diagnostic Information Queries Detailed–Recap

Over the course of the month of January 2016, I wrote a series of daily blog posts that went through each of the 70 queries in the January 2016 version of my SQL Server 2016 Diagnostic Information Queries, with documentation of the various views and functions behind each query, along with what I hope is some useful information about how to interpret the results of each query.

Below are links to each of the daily blog posts for this series, along with a list of the queries that were covered that day:

SQL Server Diagnostic Information Queries Detailed, Day 1

Version Info, Core Counts

SQL Server Diagnostic Information Queries Detailed, Day 2

Server Properties, Configuration Values

SQL Server Diagnostic Information Queries Detailed, Day 3

Global Trace Flags, Process Memory

SQL Server Diagnostic Information Queries Detailed, Day 4

SQL Server Services Info, SQL Server Agent Jobs

SQL Server Diagnostic Information Queries Detailed, Day 5

SQL Server Agent Alerts, Windows Info

SQL Server Diagnostic Information Queries Detailed, Day 6

SQL Server NUMA Info, System Memory

SQL Server Diagnostic Information Queries Detailed, Day 7

SQL Server Error Log, Cluster Node Properties, AlwaysOn AG Cluster

SQL Server Diagnostic Information Queries Detailed, Day 8

Hardware Info, System Manufacturer, Processor Description

SQL Server Diagnostic Information Queries Detailed, Day 9

BPW Configuration, BPE Usage

SQL Server Diagnostic Information Queries Detailed, Day 10

Memory Dump Info, Database Filenames and Paths

SQL Server Diagnostic Information Queries Detailed, Day 11

Volume Info, Drive-Level Latency, IO Stalls by File, IO Warnings

SQL Server Diagnostic Information Queries Detailed, Day 12

Database Properties, Missing Indexes All Databases, VLF Counts

SQL Server Diagnostic Information Queries Detailed, Day 13

CPU Usage by Database, IO Usage by Database, Total Buffer Usage by Database

SQL Server Diagnostic Information Queries Detailed, Day 14

Top Waits, Connection Counts by IP Address

SQL Server Diagnostic Information Queries Detailed, Day 15

Avg Task Counts, Detect Blocking

SQL Server Diagnostic Information Queries Detailed, Day 16

CPU Utilization History, Top Worker Time Queries

SQL Server Diagnostic Information Queries Detailed, Day 17

PLE by NUMA Node, Memory Grants Pending,

SQL Server Diagnostic Information Queries Detailed, Day 18

Memory Clerk Usage, Ad hoc Queries, Top Logical Reads Queries

SQL Server Diagnostic Information Queries Detailed, Day 19

File Sizes and Space, IO Stats by File

SQL Server Diagnostic Information Queries Detailed, Day 20

Query Execution Counts, SP Execution Counts

SQL Server Diagnostic Information Queries Detailed, Day 21

SP Avg Elapsed Time, SP Worker Time

SQL Server Diagnostic Information Queries Detailed, Day 22

SP Logical Reads, SP Physical Reads

SQL Server Diagnostic Information Queries Detailed, Day 23

SP Logical Writes, Top IO Statements

SQL Server Diagnostic Information Queries Detailed, Day 24

Bad NC Indexes, Missing Indexes, Missing Index Warnings

SQL Server Diagnostic Information Queries Detailed, Day 25

Buffer Usage, Table Sizes

SQL Server Diagnostic Information Queries Detailed, Day 26

Table Properties, Statistics Update

SQL Server Diagnostic Information Queries Detailed, Day 27

Volatile Indexes, Index Fragmentation

SQL Server Diagnostic Information Queries Detailed, Day 28

Overall Index Usage – Reads, Overall Index Usage – Writes

SQL Server Diagnostic Information Queries Detailed, Day 29

XTP Index Usage, Lock Waits

SQL Server Diagnostic Information Queries Detailed, Day 30

UDF Statistics, QueryStore Options

SQL Server Diagnostic Information Queries Detailed, Day 31

High Aggregate Duration Queries, Recent Full Backups

 

These three Pluralsight Courses go into even more detail about how to run these queries and interpret the results.

SQL Server 2014 DMV Diagnostic Queries – Part 1

SQL Server 2014 DMV Diagnostic Queries – Part 2

SQL Server 2014 DMV Diagnostic Queries – Part 3

SQL Server Diagnostic Information Queries Detailed, Day 31

For Day 31 of this series, we start out with Query #69, which is High Aggregate Duration Queries. This query retrieves information from the sys.query_store_query_text query store catalog view, the sys.query_store_query query store catalog view, the sys.query_store_plan query store catalog view, the sys.query_store_runtime_stats query store catalog view, and the sys.query_store_runtime_stats_interval query store catalog view about the highest aggregate duration queries in the current database over the past hour. Query #69 is shown in Figure 1.

   1: -- Get highest aggregate duration queries over last hour (Query 69) (High Aggregate Duration Queries)

   2: WITH AggregatedDurationLastHour

   3: AS

   4: (SELECT q.query_id, SUM(count_executions * avg_duration) AS total_duration,

   5:    COUNT (distinct p.plan_id) AS number_of_plans

   6:    FROM sys.query_store_query_text AS qt WITH (NOLOCK)

   7:    INNER JOIN sys.query_store_query AS q WITH (NOLOCK)

   8:    ON qt.query_text_id = q.query_text_id

   9:    INNER JOIN sys.query_store_plan AS p WITH (NOLOCK)

  10:    ON q.query_id = p.query_id

  11:    INNER JOIN sys.query_store_runtime_stats AS rs WITH (NOLOCK)

  12:    ON rs.plan_id = p.plan_id

  13:    INNER JOIN sys.query_store_runtime_stats_interval AS rsi WITH (NOLOCK)

  14:    ON rsi.runtime_stats_interval_id = rs.runtime_stats_interval_id

  15:    WHERE rsi.start_time >= DATEADD(hour, -1, GETUTCDATE()) 

  16:    AND rs.execution_type_desc = N'Regular'

  17:    GROUP BY q.query_id),

  18: OrderedDuration AS

  19: (SELECT query_id, total_duration, number_of_plans, 

  20:  ROW_NUMBER () OVER (ORDER BY total_duration DESC, query_id) AS RN

  21:  FROM AggregatedDurationLastHour)

  22: SELECT OBJECT_NAME(q.object_id) AS [Containing Object], qt.query_sql_text, 

  23: od.total_duration AS [Total Duration (microsecs)], 

  24: od.number_of_plans AS [Plan Count],

  25: p.is_forced_plan, p.is_parallel_plan, p.is_trivial_plan,

  26: q.query_parameterization_type_desc, p.[compatibility_level],

  27: p.last_compile_start_time, q.last_execution_time,

  28: CONVERT(xml, p.query_plan) AS query_plan_xml 

  29: FROM OrderedDuration AS od 

  30: INNER JOIN sys.query_store_query AS q WITH (NOLOCK)

  31: ON q.query_id  = od.query_id

  32: INNER JOIN sys.query_store_query_text AS qt WITH (NOLOCK)

  33: ON q.query_text_id = qt.query_text_id

  34: INNER JOIN sys.query_store_plan AS p WITH (NOLOCK)

  35: ON q.query_id = p.query_id

  36: WHERE od.RN <= 50 

  37: ORDER BY total_duration DESC OPTION (RECOMPILE);

  38:  

  39: -- New for SQL Server 2016

  40: -- Requires that QueryStore is enabled for this database

Figure 1: Query #69 High Aggregate Duration Queries

If you are using the QueryStore feature in SQL Server 2016, (meaning that you have enabled it for the current database), then you can either use the built in functionality in SSMS, or use queries like this to examine the data that it collects and exposes. Personally, I like to be able to write custom queries like this to analyze the information.

This query lets you identify which queries in the current database have the highest aggregate duration over the past hour. This lets you find queries that might benefit from your query and index tuning efforts, especially ones that may show a noticeable benefit from any improvements.

 

Query #70 is Recent Full Backups. This query retrieves information from the dbo.backupset table in the msdb system database about the most recent Full database backups for the current database. Query #70 is shown in Figure 2.

   1: -- Look at recent Full backups for the current database (Query 70) (Recent Full Backups)

   2: SELECT TOP (30) bs.machine_name, bs.server_name, bs.database_name AS [Database Name], bs.recovery_model,

   3: CONVERT (BIGINT, bs.backup_size / 1048576 ) AS [Uncompressed Backup Size (MB)],

   4: CONVERT (BIGINT, bs.compressed_backup_size / 1048576 ) AS [Compressed Backup Size (MB)],

   5: CONVERT (NUMERIC (20,2), (CONVERT (FLOAT, bs.backup_size) /

   6: CONVERT (FLOAT, bs.compressed_backup_size))) AS [Compression Ratio], bs.has_backup_checksums, bs.is_copy_only, bs.encryptor_type,

   7: DATEDIFF (SECOND, bs.backup_start_date, bs.backup_finish_date) AS [Backup Elapsed Time (sec)],

   8: bs.backup_finish_date AS [Backup Finish Date]

   9: FROM msdb.dbo.backupset AS bs WITH (NOLOCK)

  10: WHERE bs.database_name = DB_NAME(DB_ID())

  11: AND bs.[type] = 'D' -- Change to L if you want Log backups

  12: ORDER BY bs.backup_finish_date DESC OPTION (RECOMPILE);

  13:  

  14: -- Are your backup sizes and times changing over time?

  15: -- Are you using backup compression?

  16: -- Have you done any backup tuning with striped backups, or changing the parameters of the backup command?

Figure 2: Query #70 Recent Full Backups

This query gives you some useful statistics and properties about your most recent Full database backups for the current database. It shows you the uncompressed size of the backup, along with the compressed size and the compression ratio, if any. It also lets you know if the backup is using checksums, whether it is a copy-only backup, and whether it is using native backup encryption, which was a new feature in SQL Server 2014. Finally, it shows you when each backup finished and how long it took to complete.

Keeping an eye on the size and elapsed time for your database backups is always a good idea. As your database gets larger, you may have to make changes to how and when you do your backups or to the underlying resources at location to where they are going to make sure that they are finishing in a reliable and timely manner.

These three Pluralsight Courses go into even more detail about how to run these queries and interpret the results.

SQL Server 2014 DMV Diagnostic Queries – Part 1

SQL Server 2014 DMV Diagnostic Queries – Part 2

SQL Server 2014 DMV Diagnostic Queries – Part 3

We have finally made it to the end of this series!  I’ll be putting up a recap post for the entire series, with links to each post.

SQL Server Diagnostic Information Queries Detailed, Day 30

For Day 30 of this series, we start out with Query #67, which is UDF Statistics. This query retrieves information from the sys.dm_exec_function_stats dynamic management view about aggregate runtime metrics for user-defined functions in the current database, ordered by object name. Query #67 is shown in Figure 1.

   1: -- Look at UDF execution statistics (Query 67) (UDF Statistics)

   2: SELECT OBJECT_NAME(object_id) AS [Function Name], execution_count,

   3:    total_elapsed_time/1000 AS [time_milliseconds], fs.[type_desc]

   4: FROM sys.dm_exec_function_stats AS fs WITH (NOLOCK) 

   5: WHERE database_id = DB_ID()

   6: ORDER BY OBJECT_NAME(object_id) OPTION (RECOMPILE);

   7:  

   8: -- New for SQL Server 2016

   9: -- Helps you investigate UDF performance issues

Figure 1: Query #67 UDF Statistics

One fairly well-known, long-running issue with SQL Server is how often you see performance problems with scalar, user-defined functions (UDFs). As a DBA, I always tried to avoid having my developers ever find out that scalar UDFs even existed, but they sometimes discovered them on their own, unfortunately. One big problem with scalar UDFs is that their actual cost does not show up when you look at the execution plan or statistics IO output for the query or stored procedure that called them.

In the past, you needed to use tools like SQL Profiler or Extended Events to see what was going on when you had scalar UDF usage. In SQL Server 2016, you will be able to use this new DMV and this query to have some visibility about what your UDFs are doing. As far as mitigation goes, doing things like converting a scalar UDF to a table UDF that just returns just one column and one row, or converting it to a stored procedure are often pretty effective and easy to do.

 

Query #68 is QueryStore Options. This query retrieves information from the sys.database_query_store_options dynamic management view about the current QueryStore options for the current database. Query #68 is shown in Figure 2.

   1: -- Get QueryStore Options for this database (Query 68) (QueryStore Options)

   2: SELECT actual_state, actual_state_desc, readonly_reason, 

   3:        current_storage_size_mb, max_storage_size_mb

   4: FROM sys.database_query_store_options WITH (NOLOCK)

   5: OPTION (RECOMPILE);

   6:  

   7: -- New for SQL Server 2016

   8: -- Requires that QueryStore is enabled for this database

Figure 2: Query #68 QueryStore Options

QueryStore is one of the more exciting new features in SQL Server 2016. It gives you a lot of visibility about what is happening with your query plans in a particular database over time. You also get the ability to force the query optimizer to use a particular “good” query plan. This should make it much easier to troubleshoot and correct plan regression issues. So far, Microsoft has not announced whether this is an Enterprise Edition–only feature or not. Before you can use QueryStore, you have to enable it for the database that you are concerned with.

 

These three Pluralsight Courses go into even more detail about how to run these queries and interpret the results.

SQL Server 2014 DMV Diagnostic Queries – Part 1

SQL Server 2014 DMV Diagnostic Queries – Part 2

SQL Server 2014 DMV Diagnostic Queries – Part 3

SQL Server Diagnostic Information Queries Detailed, Day 29

For Day 29 of this series, we start out with Query #65, which is XTP Index Usage. This query retrieves information from the sys.dm_db_xtp_index_stats dynamic management view and the sys.indexes object catalog view about the overall in-memory OLTP index usage in the current database, ordered by object name. Query #65 is shown in Figure 1.

   1: -- Get in-memory OLTP index usage (Query 65) (XTP Index Usage)

   2: SELECT OBJECT_NAME(i.[object_id]) AS [Object Name], i.index_id, i.name, i.type_desc,

   3:        xis.scans_started, xis.scans_retries, xis.rows_touched, xis.rows_returned 

   4: FROM sys.dm_db_xtp_index_stats AS xis WITH (NOLOCK)

   5: INNER JOIN sys.indexes AS i WITH (NOLOCK)

   6: ON i.[object_id] = xis.[object_id] 

   7: AND i.index_id = xis.index_id 

   8: ORDER BY OBJECT_NAME(i.[object_id]) OPTION (RECOMPILE);

   9:  

  10: -- This gives you some index usage statistics for in-memory OLTP

  11: -- Returns no data if you are not using in-memory OLTP

Figure 1: Query #65 XTP Index Usage

If you are using in-memory OLTP (aka Hekaton), then this query will show how your in-memory OLTP indexes are being used. Perhaps because this is an Enterprise-only feature and perhaps because it has some limitations in SQL Server 2014, I have not seen this feature being used that much out in the field yet. I think the adoption rate will improve with SQL Server 2016.

 

Query #66 is Lock Waits. This query retrieves information from the sys.dm_db_index_operational_stats dynamic management function, the sys.objects object catalog view, and the sys.indexes object catalog view about the cumulative lock waits in the current database. Query #66 is shown in Figure 2.

   1: -- Get lock waits for current database (Query 66) (Lock Waits)

   2: SELECT o.name AS [table_name], i.name AS [index_name], ios.index_id, ios.partition_number,

   3:         SUM(ios.row_lock_wait_count) AS [total_row_lock_waits], 

   4:         SUM(ios.row_lock_wait_in_ms) AS [total_row_lock_wait_in_ms],

   5:         SUM(ios.page_lock_wait_count) AS [total_page_lock_waits],

   6:         SUM(ios.page_lock_wait_in_ms) AS [total_page_lock_wait_in_ms],

   7:         SUM(ios.page_lock_wait_in_ms)+ SUM(row_lock_wait_in_ms) AS [total_lock_wait_in_ms]

   8: FROM sys.dm_db_index_operational_stats(DB_ID(), NULL, NULL, NULL) AS ios

   9: INNER JOIN sys.objects AS o WITH (NOLOCK)

  10: ON ios.[object_id] = o.[object_id]

  11: INNER JOIN sys.indexes AS i WITH (NOLOCK)

  12: ON ios.[object_id] = i.[object_id] 

  13: AND ios.index_id = i.index_id

  14: WHERE o.[object_id] > 100

  15: GROUP BY o.name, i.name, ios.index_id, ios.partition_number

  16: HAVING SUM(ios.page_lock_wait_in_ms)+ SUM(row_lock_wait_in_ms) > 0

  17: ORDER BY total_lock_wait_in_ms DESC OPTION (RECOMPILE);

  18:  

  19: -- This query is helpful for troubleshooting blocking and deadlocking issues

Figure 2: Query #66 Lock Waits

If you are seeing symptoms of locking/blocking/deadlocks (such as high average task counts or actual deadlock errors), then this query can show which tables and indexes are seeing the most lock waits, which can often help you troubleshoot and resolve your blocking issues.

These three Pluralsight Courses go into even more detail about how to run these queries and interpret the results.

SQL Server 2014 DMV Diagnostic Queries – Part 1

SQL Server 2014 DMV Diagnostic Queries – Part 2

SQL Server 2014 DMV Diagnostic Queries – Part 3

SQL Server Diagnostic Information Queries Detailed, Day 28

For Day 28 of this series, we start out with Query #63, which is Overall Index Usage – Reads. This query retrieves information from the sys.indexes object catalog view, and the sys.dm_db_index_usage_stats dynamic management view about the overall index usage in the current database, ordered by reads. Query #63 is shown in Figure 1.

   1: --- Index Read/Write stats (all tables in current DB) ordered by Reads  (Query 63) (Overall Index Usage - Reads)

   2: SELECT OBJECT_NAME(i.[object_id]) AS [ObjectName], i.name AS [IndexName], i.index_id, 

   3:        s.user_seeks, s.user_scans, s.user_lookups,

   4:        s.user_seeks + s.user_scans + s.user_lookups AS [Total Reads], 

   5:        s.user_updates AS [Writes],  

   6:        i.type_desc AS [Index Type], i.fill_factor AS [Fill Factor], i.has_filter, i.filter_definition, 

   7:        s.last_user_scan, s.last_user_lookup, s.last_user_seek

   8: FROM sys.indexes AS i WITH (NOLOCK)

   9: LEFT OUTER JOIN sys.dm_db_index_usage_stats AS s WITH (NOLOCK)

  10: ON i.[object_id] = s.[object_id]

  11: AND i.index_id = s.index_id

  12: AND s.database_id = DB_ID()

  13: WHERE OBJECTPROPERTY(i.[object_id],'IsUserTable') = 1

  14: ORDER BY s.user_seeks + s.user_scans + s.user_lookups DESC OPTION (RECOMPILE); -- Order by reads

  15:  

  16:  

  17: -- Show which indexes in the current database are most active for Reads

Figure 1: Query #63 Overall Index Usage – Reads

This query shows you which indexes in the current database have the most cumulative reads (including seeks, scans and lookups) since the instance was last restarted or the index was created. This helps you understand your workload, and shows you which indexes are the most valuable for your workload.  Another use for this query is to help identify possible data compression candidates. If you find an index on a large table with a high number of reads and a low number of writes, then it might be a good candidate for data compression if the data is highly compressible.

 

Query #64 is Overall Index Usage – Writes. This query retrieves information from the sys.indexes object catalog view, and the sys.dm_db_index_usage_stats dynamic management view about the overall index usage in the current database, ordered by writes. Query #64 is shown in Figure 2.

   1: --- Index Read/Write stats (all tables in current DB) ordered by Writes  (Query 64) (Overall Index Usage - Writes)

   2: SELECT OBJECT_NAME(i.[object_id]) AS [ObjectName], i.name AS [IndexName], i.index_id,

   3:        s.user_updates AS [Writes], s.user_seeks + s.user_scans + s.user_lookups AS [Total Reads], 

   4:        i.type_desc AS [Index Type], i.fill_factor AS [Fill Factor], i.has_filter, i.filter_definition,

   5:        s.last_system_update, s.last_user_update

   6: FROM sys.indexes AS i WITH (NOLOCK)

   7: LEFT OUTER JOIN sys.dm_db_index_usage_stats AS s WITH (NOLOCK)

   8: ON i.[object_id] = s.[object_id]

   9: AND i.index_id = s.index_id

  10: AND s.database_id = DB_ID()

  11: WHERE OBJECTPROPERTY(i.[object_id],'IsUserTable') = 1

  12: ORDER BY s.user_updates DESC OPTION (RECOMPILE);                         -- Order by writes

  13:  

  14: -- Show which indexes in the current database are most active for Writes

Figure 2: Query #64 Overall Index Usage – Writes

This query shows you which indexes in the current database have the most cumulative writes since the instance was last restarted or the index was created. This helps you understand your workload, and shows you which indexes are the most volatile in your workload.  Another use for this query is to help identify possible indexes that you might consider dropping. If you find a non-clustered, non-key index on a table with a high number of writes and a very low number of reads, then you might want to drop that index, after doing some further investigation. You want to make sure that your instance has been running long enough so that you have seen your complete workload so that you don’t drop an index that is actually needed for queries that have not run yet.

SQL Server Diagnostic Information Queries Detailed, Day 27

For Day 27 of this series, we start out with Query #61, which is Volatile Indexes. This query retrieves information from the sys.objects object catalog view, the sys.stats object catalog view, and the sys.dm_db_stats_properties dynamic management function about the most volatile indexes and statistics in the current database. Query #61 is shown in Figure 1.

   1: -- Look at most frequently modified indexes and statistics (Query 61) (Volatile Indexes)

   2: SELECT o.name AS [Object Name], o.[object_id], o.type_desc, s.name AS [Statistics Name], 

   3:        s.stats_id, s.no_recompute, s.auto_created, 

   4:        sp.modification_counter, sp.rows, sp.rows_sampled, sp.last_updated

   5: FROM sys.objects AS o WITH (NOLOCK)

   6: INNER JOIN sys.stats AS s WITH (NOLOCK)

   7: ON s.object_id = o.object_id

   8: CROSS APPLY sys.dm_db_stats_properties(s.object_id, s.stats_id) AS sp

   9: WHERE o.type_desc NOT IN (N'SYSTEM_TABLE', N'INTERNAL_TABLE')

  10: AND sp.modification_counter > 0

  11: ORDER BY sp.modification_counter DESC, o.name OPTION (RECOMPILE);

Figure 1: Query #61 Volatile Indexes

This query shows you which indexes and statistics have had the most updates, which helps you understand your workload in more detail. Understanding which tables, indexes and statistics are most volatile is useful when you are thinking about how to layout your database files, how to design and configure your storage subsystem, and how to handle your index tuning and maintenance.

 

Query #62 is Index Fragmentation. This query retrieves information from the sys.dm_db_index_physical_stats dynamic management function and the sys.indexes object catalog view about the fragmentation status for all of the indexes larger than 2500 pages in the current database. Query #62 is shown in Figure 2.

   1: -- Get fragmentation info for all indexes above a certain size in the current database  (Query 62) (Index Fragmentation)

   2: -- Note: This query could take some time on a very large database

   3: SELECT DB_NAME(ps.database_id) AS [Database Name], OBJECT_NAME(ps.OBJECT_ID) AS [Object Name], 

   4: i.name AS [Index Name], ps.index_id, ps.index_type_desc, ps.avg_fragmentation_in_percent, 

   5: ps.fragment_count, ps.page_count, i.fill_factor, i.has_filter, i.filter_definition

   6: FROM sys.dm_db_index_physical_stats(DB_ID(),NULL, NULL, NULL , N'LIMITED') AS ps

   7: INNER JOIN sys.indexes AS i WITH (NOLOCK)

   8: ON ps.[object_id] = i.[object_id] 

   9: AND ps.index_id = i.index_id

  10: WHERE ps.database_id = DB_ID()

  11: AND ps.page_count > 2500

  12: ORDER BY ps.avg_fragmentation_in_percent DESC OPTION (RECOMPILE);

  13:  

  14: -- Helps determine whether you have framentation in your relational indexes

  15: -- and how effective your index maintenance strategy is

Figure 2: Query #62 Index Fragmentation

This query shows you the current fragmentation status for your heap tables and all indexes above 2500 pages in the current database. This helps you evaluate your index maintenance effectiveness. It is also useful when you are thinking about how you do your index maintenance and whether you might want to consider changing the fill factor on some indexes to reduce how often you need to rebuild or reorganize them.

I would say that most of the customers that I see fall into one of three categories regarding their index maintenance practices:

  1. Organizations with “Accidental DBAs” who don’t even know what index maintenance is, much less the difference between rebuild and reorganize, or about things like fill factors
    1. These organizations often have extremely high index fragmentation across most of their indexes and databases
  2. Organizations that are using the existing SQL Maintenance Plan Wizard (which is evil, BTW), which makes it far too easy for them to do redundant things like reorganize, then rebuild, then update statistics, etc.
    1. The organizations sometimes have less fragmentation, but they also often have a lot of extra transaction log activity, from redundant index maintenance activity
    2. The new SQL Maintenance Plan Wizard in SQL Server 2016 is a huge improvement, at least as far as the UI goes
  3. Organizations that are using something like Ola Hallengren’s free SQL Server Maintenance Solution for their index maintenance
    1. These organizations are usually in the best shape

SQL Server Diagnostic Information Queries Detailed, Day 26

For Day 26 of this series, we start out with Query #59, which is Table Properties. This query retrieves information from the sys.tables object catalog view, and the sys.partitions object catalog view about various table properties in the current database. Query #59 is shown in Figure 1.

   1: -- Get some key table properties (Query 59) (Table Properties)

   2: SELECT OBJECT_NAME(t.[object_id]) AS [ObjectName], p.[rows] AS [Table Rows], p.index_id, 

   3:        p.data_compression_desc AS [Index Data Compression],

   4:        t.create_date, t.lock_on_bulk_load, t.is_replicated, t.has_replication_filter, 

   5:        t.is_tracked_by_cdc, t.lock_escalation_desc, t.is_memory_optimized, t.durability_desc, t.is_filetable,

   6:        t.temporal_type_desc, t.is_remote_data_archive_enabled, t.remote_data_archive_migration_state_desc, t.is_external -- new for SQL Server 2016

   7: FROM sys.tables AS t WITH (NOLOCK)

   8: INNER JOIN sys.partitions AS p WITH (NOLOCK)

   9: ON t.[object_id] = p.[object_id]

  10: WHERE OBJECT_NAME(t.[object_id]) NOT LIKE N'sys%'

  11: ORDER BY OBJECT_NAME(t.[object_id]), p.index_id OPTION (RECOMPILE);

  12:  

  13: -- Gives you some good information about your tables

  14: -- Is Memory optimized and durability description are Hekaton-related properties that were new in SQL Server 2014

  15: -- temporal_type_desc, is_remote_data_archive_enabled, remote_data_archive_migration_state_desc, is_external are new in SQL Server 2016

Figure 1: Query #59 Table Properties

This query helps you understand what is going on with properties for the tables in the current database, showing you things such as whether they are being replicated, being tracked by change data capture, whether they are Hekaton tables, whether they are StretchDB tables, etc. It also shows you the data compression status for every index in each table. This can help you find possible data compression candidates.

 

Query #60 is Statistics Update. This query retrieves information from the sys.objects object catalog view, the sys.indexes object catalog view, the sys.stats object catalog view, and the sys.dm_db_partition_stats dynamic management view about the properties and status of the statistics in the current database. Query #60 is shown in Figure 2.

   1: -- When were Statistics last updated on all indexes?  (Query 60) (Statistics Update)

   2: SELECT SCHEMA_NAME(o.Schema_ID) + N'.' + o.NAME AS [Object Name], o.type_desc AS [Object Type],

   3:       i.name AS [Index Name], STATS_DATE(i.[object_id], i.index_id) AS [Statistics Date], 

   4:       s.auto_created, s.no_recompute, s.user_created, s.is_incremental, s.is_temporary,

   5:       st.row_count, st.used_page_count

   6: FROM sys.objects AS o WITH (NOLOCK)

   7: INNER JOIN sys.indexes AS i WITH (NOLOCK)

   8: ON o.[object_id] = i.[object_id]

   9: INNER JOIN sys.stats AS s WITH (NOLOCK)

  10: ON i.[object_id] = s.[object_id] 

  11: AND i.index_id = s.stats_id

  12: INNER JOIN sys.dm_db_partition_stats AS st WITH (NOLOCK)

  13: ON o.[object_id] = st.[object_id]

  14: AND i.[index_id] = st.[index_id]

  15: WHERE o.[type] IN ('U', 'V')

  16: AND st.row_count > 0

  17: ORDER BY STATS_DATE(i.[object_id], i.index_id) DESC OPTION (RECOMPILE);  

  18:  

  19: -- Helps discover possible problems with out-of-date statistics

  20: -- Also gives you an idea which indexes are the most active

Figure 2: Query #60 Statistics Update

This query shows you a number of relevant properties about the index-associated statistics in your current database, ordered by the last time that statistics were updated. This can help you determine whether you might have a problem with out of date statistics. My general guidance about statistics is that you should use the default database properties of auto create and auto update for statistics, plus I think you should also use the auto update statistics asynchronously database property, along with global trace flag 2371. In some situations, you may also want/need to do additional statistics maintenance on highly volatile tables with SQL Server Agent jobs.

SQL Server Diagnostic Information Queries Detailed, Day 25

For Day 25 of this series, we start out with Query #57, which is Buffer Usage. This query retrieves information from the sys.allocation_units object catalog view, the sys.dm_os_buffer_descriptors dynamic management view and the sys.partitions object catalog view about buffer pool usage in the current database. Query #57 is shown in Figure 1.

   1: -- Breaks down buffers used by current database by object (table, index) in the buffer cache  (Query 57) (Buffer Usage)

   2: -- Note: This query could take some time on a busy instance

   3: SELECT OBJECT_NAME(p.[object_id]) AS [Object Name], p.index_id, 

   4: CAST(COUNT(*)/128.0 AS DECIMAL(10, 2)) AS [Buffer size(MB)],  

   5: COUNT(*) AS [BufferCount], p.Rows AS [Row Count],

   6: p.data_compression_desc AS [Compression Type]

   7: FROM sys.allocation_units AS a WITH (NOLOCK)

   8: INNER JOIN sys.dm_os_buffer_descriptors AS b WITH (NOLOCK)

   9: ON a.allocation_unit_id = b.allocation_unit_id

  10: INNER JOIN sys.partitions AS p WITH (NOLOCK)

  11: ON a.container_id = p.hobt_id

  12: WHERE b.database_id = CONVERT(int,DB_ID())

  13: AND p.[object_id] > 100

  14: GROUP BY p.[object_id], p.index_id, p.data_compression_desc, p.[Rows]

  15: ORDER BY [BufferCount] DESC OPTION (RECOMPILE);

  16:  

  17: -- Tells you what tables and indexes are using the most memory in the buffer cache

  18: -- It can help identify possible candidates for data compression

Figure 1: Query #57 Buffer Usage

This query shows you which tables and indexes are using the most buffer pool space in the current database. This is very important information to understand if you are under internal memory pressure, or you are seeing high read latency for your data file(s). The query also displays the data compression status for the index.

If you see an index that is using a lot of space in the SQL Server buffer pool (because it shows up near the top of this query), then I would investigate whether SQL Server data compression might make sense for that index. What you want to look for are indexes that are highly compressible, on relatively static data, at least as far as UPDATES are concerned.

 

Query #58 is Table Sizes. This query retrieves information from the the sys.partitions object catalog view about the table sizes and clustered index (or heap) data compression status in the current database. Query #58 is shown in Figure 2.

   1: -- Get Table names, row counts, and compression status for clustered index or heap  (Query 58) (Table Sizes)

   2: SELECT OBJECT_NAME(object_id) AS [ObjectName], 

   3: SUM(Rows) AS [RowCount], data_compression_desc AS [CompressionType]

   4: FROM sys.partitions WITH (NOLOCK)

   5: WHERE index_id < 2 --ignore the partitions from the non-clustered index if any

   6: AND OBJECT_NAME(object_id) NOT LIKE N'sys%'

   7: AND OBJECT_NAME(object_id) NOT LIKE N'queue_%' 

   8: AND OBJECT_NAME(object_id) NOT LIKE N'filestream_tombstone%' 

   9: AND OBJECT_NAME(object_id) NOT LIKE N'fulltext%'

  10: AND OBJECT_NAME(object_id) NOT LIKE N'ifts_comp_fragment%'

  11: AND OBJECT_NAME(object_id) NOT LIKE N'filetable_updates%'

  12: AND OBJECT_NAME(object_id) NOT LIKE N'xml_index_nodes%'

  13: AND OBJECT_NAME(object_id) NOT LIKE N'sqlagent_job%'  

  14: AND OBJECT_NAME(object_id) NOT LIKE N'plan_persist%'  

  15: GROUP BY object_id, data_compression_desc

  16: ORDER BY SUM(Rows) DESC OPTION (RECOMPILE);

  17:  

  18: -- Gives you an idea of table sizes, and possible data compression opportunities

Figure 2: Query #58 Table Sizes

This query simply shows you the row counts and data compression status for the clustered index or heap for each table in the current database. I use this query to look for tables that might be good candidates of SQL Server data compression. Again, what you are looking for are large tables, that are relatively static, that compress well.

SQL Server Diagnostic Information Queries Detailed, Day 24

For Day 24 of this series, we start out with Query #54, which is Bad NC Indexes. This query retrieves information from the sys.dm_db_index_usage_stats dynamic management view and the sys.indexes object catalog view about non-clustered indexes that have more writes than reads in the current database. Query #54 is shown in Figure 1.

   1: -- Possible Bad NC Indexes (writes > reads)  (Query 54) (Bad NC Indexes)

   2: SELECT OBJECT_NAME(s.[object_id]) AS [Table Name], i.name AS [Index Name], i.index_id, 

   3: i.is_disabled, i.is_hypothetical, i.has_filter, i.fill_factor,

   4: user_updates AS [Total Writes], user_seeks + user_scans + user_lookups AS [Total Reads],

   5: user_updates - (user_seeks + user_scans + user_lookups) AS [Difference]

   6: FROM sys.dm_db_index_usage_stats AS s WITH (NOLOCK)

   7: INNER JOIN sys.indexes AS i WITH (NOLOCK)

   8: ON s.[object_id] = i.[object_id]

   9: AND i.index_id = s.index_id

  10: WHERE OBJECTPROPERTY(s.[object_id],'IsUserTable') = 1

  11: AND s.database_id = DB_ID()

  12: AND user_updates > (user_seeks + user_scans + user_lookups)

  13: AND i.index_id > 1

  14: ORDER BY [Difference] DESC, [Total Writes] DESC, [Total Reads] ASC OPTION (RECOMPILE);

  15:  

  16: -- Look for indexes with high numbers of writes and zero or very low numbers of reads

  17: -- Consider your complete workload, and how long your instance has been running

  18: -- Investigate further before dropping an index!

Figure 1: Query #54 Bad NC Indexes

What you are looking for with this query are indexes that have high numbers of writes and very few or even zero reads. If you are paying the cost to maintain an index as the data changes in your table, but the index is never used for reads, then you are placing unneeded stress on your storage subsystem that is not providing any benefits to the system. Having unused indexes also makes your database larger, and makes index maintenance more time consuming and resource intensive.

One key point to keep in mind before you start dropping indexes that appear to be unused is how long your SQL Server instance has been running. Before you drop an index, consider whether you have seen your complete normal business cycle. Perhaps there are monthly reports that actually do use an index that normally does not see any read activity with your regular workload.

 

 

Query #55 is Missing Indexes. This query retrieves information from the sys.dm_db_missing_index_group_stats dynamic management view, the sys.dm_db_missing_index_groups dynamic management view, the sys.dm_db_missing_index_details dynamic management view and the sys.partitions catalog view about “missing” indexes that the SQL Server Query Optimizer thinks that it would like to have in the current database. Query #55 is shown in Figure 2.

   1: -- Missing Indexes for current database by Index Advantage  (Query 55) (Missing Indexes)

   2: SELECT DISTINCT CONVERT(decimal(18,2), user_seeks * avg_total_user_cost * (avg_user_impact * 0.01)) AS [index_advantage], 

   3: migs.last_user_seek, mid.[statement] AS [Database.Schema.Table],

   4: mid.equality_columns, mid.inequality_columns, mid.included_columns,

   5: migs.unique_compiles, migs.user_seeks, migs.avg_total_user_cost, migs.avg_user_impact,

   6: OBJECT_NAME(mid.[object_id]) AS [Table Name], p.rows AS [Table Rows]

   7: FROM sys.dm_db_missing_index_group_stats AS migs WITH (NOLOCK)

   8: INNER JOIN sys.dm_db_missing_index_groups AS mig WITH (NOLOCK)

   9: ON migs.group_handle = mig.index_group_handle

  10: INNER JOIN sys.dm_db_missing_index_details AS mid WITH (NOLOCK)

  11: ON mig.index_handle = mid.index_handle

  12: INNER JOIN sys.partitions AS p WITH (NOLOCK)

  13: ON p.[object_id] = mid.[object_id]

  14: WHERE mid.database_id = DB_ID() 

  15: ORDER BY index_advantage DESC OPTION (RECOMPILE);

  16:  

  17: -- Look at index advantage, last user seek time, number of user seeks to help determine source and importance

  18: -- SQL Server is overly eager to add included columns, so beware

  19: -- Do not just blindly add indexes that show up from this query!!!

Figure 2: Query #55 Missing Indexes

This query is very useful, but also very easy to misinterpret and misuse. I have seen many novice DBAs and developers use the results of this query to pretty badly over-index their databases, which affects their database size and hurts insert, update, and delete performance. I like to focus on the last_user_seek column, and see how long ago that was. Was it a few seconds or minutes ago, or was it days or weeks ago?

I then start looking at the user_seeks, avg_total_user_cost, and avg_user_impact columns to get a sense for how often SQL Server thinks it wants this proposed index, how expensive it is not to have the index, and how much the query optimizer thinks the cost of the query would be reduced if it did have this index that it is requesting.

Next, I’ll look at any other proposed indexes on the same table to see if I can come up with a wider, consolidated index that covers multiple requested indexes. Finally, I’ll look at the existing indexes on that table, and look at the index usage metrics for that table to have a better idea of whether a new index would be a good idea, based on the volatility of that table.

This query is very similar to Query #28, but this one is only for the current database. It also pulls back the number of rows in a table, which is useful information when you are considering creating a new index, especially when you are using SQL Server Standard Edition, which does not have online index operations.

 

Query #56 is Missing Index Warnings. This query retrieves information from the sys.dm_exec_cached_plans dynamic management view, the sys.dm_exec_query_plan dynamic management function about missing index warnings in the plan cache for the current database. Query #56 is shown in Figure 3.

   1: -- Find missing index warnings for cached plans in the current database  (Query 56) (Missing Index Warnings)

   2: -- Note: This query could take some time on a busy instance

   3: SELECT TOP(25) OBJECT_NAME(objectid) AS [ObjectName], 

   4:                query_plan, cp.objtype, cp.usecounts, cp.size_in_bytes

   5: FROM sys.dm_exec_cached_plans AS cp WITH (NOLOCK)

   6: CROSS APPLY sys.dm_exec_query_plan(cp.plan_handle) AS qp

   7: WHERE CAST(query_plan AS NVARCHAR(MAX)) LIKE N'%MissingIndex%'

   8: AND dbid = DB_ID()

   9: ORDER BY cp.usecounts DESC OPTION (RECOMPILE);

  10:  

  11: -- Helps you connect missing indexes to specific stored procedures or queries

  12: -- This can help you decide whether to add them or not

Figure 3: Query #56 Missing Index Warnings

This query (which can be time consuming on a busy instance with a large plan cache) shows you where you have cached query plans with missing index warnings. This is very helpful, since it can often help you tie requested “missing” indexes to a particular stored procedure or prepared query plan.