SQL101: Avoiding Mistakes on a Production Database Server

As Kimberly blogged about earlier this year, SQLskills has an ongoing initiative to blog about basic topics, which we’re calling SQL101. We’re all blogging about things that we often see done incorrectly, technologies used the wrong way, or where there are many misunderstandings that lead to serious problems. If you want to find all of our SQLskills SQL101 blog posts, check out SQLskills.com/help/SQL101.

One reason that it is relatively difficult to get your first job as a DBA (compared to other positions, such as a developer) is that it is very easy for a DBA with Production access to cause an enormous amount of havoc with a single momentary mistake.

As a Developer, many of your most common mistakes are only seen by yourself. If you write some code with a syntax error that doesn’t compile, or you write some code that fails your unit tests, usually nobody sees those problems but you, and you have the opportunity to fix your mistakes before you check-in your code, with no one being any the wiser.

A DBA doing something like running an UPDATE or DELETE statement without a WHERE clause, running a query against a Production instance database when you thought you were running it against a Development instance database, or making a schema change in Production that is a size of data operation (that locks up a table for a long period) are just a few examples of common DBA mistakes that can have huge consequences for an organization.

A split-second, careless DBA mistake can cause a long outage that can be difficult or even impossible to recover from. In SQL Server, Cntl-Z (the undo action) does not work, so you need to be detail-oriented and careful as a good DBA. As the old saying goes: “measure twice and cut once”.

Here are a few basic tips that can help you avoid some of these common mistakes:


Using Color-Coded Connections in SSMS

SQL Server Management Studio (SSMS) has long had the ability to set a custom color as a connection property for individual connections to an instance of SQL Server. This option is available in legacy versions of SSMS and in the latest 17.4 version of SSMS. You can get even more robust connection coloring capability with third-party tools such as SSMS Tools Pack.

The idea here is to set specific colors, such as red, yellow, or green for specific types of database instances to help remind you when you are connected to a Production instance rather than a non-Production instance. It is fairly common to use red for a Production instance. This can be helpful if you don’t have red green color blindness, which affects about 7-10% of men, but is much less common among women.

Figure 1 shows how you can check the “Use custom color” checkbox, and then select the color you want to use for that connection. After that, as long as you use the exact same connection credentials for that instance from your copy of SSMS, you should get the color that you set when you open a connection to that instance.

I would not bet my job on the color always being accurate, because depending on exactly how you open a connection to the instance, you may not always get the custom color that you set for the connection. Still, it is an extra piece of added insurance.


image

Figure 1: Setting a custom color for a connection


Figure 2 shows a red bar at the bottom of the query window (which is the default position for the bar) after setting a custom connection color. This would help warn me that I was connected to a Production instance, so I need to be especially careful before doing anything.


image

Figure 2: Query window using red for the connection


Double-Checking Your Connection Information Before Running a Query

Something you should always do before running any query is to take a second to glance down to the bottom right of SSMS Query window to verify your current connection information. It will show the name of the instance you are connected to, your logon information (including the SPID number), and the name of the database you are connected to.

Taking the time to always verify that you are connected to the database and instance that you think you are BEFORE running a query will save you from making many common, costly mistakes.


Wrap Queries in an Explicit Transaction

One common safety measure is to wrap your queries (especially potentially dangerous ones that update or delete data) in an explicit transaction as you see in Figure 3. You open an explicit transaction with a BEGIN TRAN statement, then run just your query, without the COMMIT TRAN statement. If the query does what you expect (which the xx rows affected message can often quickly confirm), then you commit the explicit transaction by executing the COMMIT TRAN statement.

If it turns out that you just made a horrible mistake (like I did in the example in Figure 3) by omitting the WHERE clause, you would execute the ROLLBACK TRAN statement to rollback your explicit transaction (which could take a while to complete).


image

Figure 3: Using an explicit transaction as a safety measure


Test your Update/Delete Queries as Select Queries Before You Run Them

Another common safety measure is to write and run a test version of any query that is designed to change data, where you simply SELECT the rows that you are planning on changing before you actually try to change them with an UPDATE or DELETE statement. You can often just have the query count the number of rows that come back from your test SELECT statement, but you might need or want to to browse the data that comes back to be 100% sure that you don’t have a logic error in your query that would end up deleting or updating the wrong result set.


These are just a few of the most common measures for avoiding common DBA mistakes. The most important step is to always be detail-oriented and very careful when you are making potentially dangerous changes in Production, which is easier said than done. If you do make a big mistake, don’t panic, and don’t try to cover it up. Taking a little time to think about what you did, and the best way to quickly and correctly fix the problem is always the best course of action.





SQL Server 2014 SP2 CU9

On December 18, 2017, Microsoft released SQL Server 2014 SP2 CU9, which is Build 12.05563.0. By my count, this CU has seven public hotfixes, nearly all of which are for the SQL Engine of SQL performance.

Since SQL Server 2014 SP1 and earlier are no longer “supported service packs”, there is no corresponding CU for the SP1 or RTM branches of SQL Server 2014.

As always, I think it is a good idea to make an effort to stay current on cumulative updates, as does Microsoft.

SQL Server Missing Indexes Feature and Trace Flag 2392

Background

Since the SQL Server 2005 release, SQL Server has had the default ability to collect metrics about what the query optimizer thinks are “missing” row store indexes, i.e. indexes that the query optimizer thinks would reduce the cost of particular individual queries that have been executed since SQL Server was last restarted.

The word missing is a little confusing to some people, since I have had many questions over the years about what could have happened to these “missing” indexes (implying that they used to be there, but somebody must have dropped them). Perhaps a better word would have been desired or candidate indexes.

At any rate, while far from perfect, I have found this feature to be a very useful method for helping to get the appropriate row store indexes in place for the overall SQL Server workload on a database. If you blindly add every new index that the query optimizer asks for, you will quickly have a large number of similar, narrow indexes on a table that probably could have been consolidated into fewer, wider indexes.

Using your common sense and experience as a DBA, your knowledge of your overall database workload and by running some queries to better characterize the volatility of a table, it is possible to do a pretty good job of index tuning using this feature. You also want to be on the lookout for indexes that can be dropped as part of your index tuning efforts.

These “missing indexes” are visible as the green missing index warnings in the graphical execution plan for a query, you can also find them by querying the plan cache looking for missing index warnings, and by running DMV queries to find them.


Recent Hotfix Reveals Trace Flag 2392

Microsoft has released a hotfix for the latest round of cumulative updates for SQL Server 2014, SQL Server 2016, and SQL Server 2017 that revealed trace flag 2392. This same fix is in SQL Server 2014 SP2 CU8, SQL Server 2016 SP1 CU6 and SQL 2017 CU2.

FIX: Access violation may occur when you cancel a pending query if the missing indexes feature is enabled in SQL Server 2014

Just to be clear, this hotfix has nothing to do with how the missing index feature functions. It just reveals a previously undocumented trace flag (TF 2392) that can be used to turn the missing index feature off completely (as a workaround to the issue that is corrected by the hotfix). This trace flag has been in the product since SQL Server 2005.

Once I discovered this trace flag, I was curious as to how it actually behaves, which is somewhat different than I expected. First, it is a startup-only trace flag, in terms of what it actually does. You can enable/disable it to your hearts content while SQL Server is running, and it will show as being enabled/disabled (with no error message about only being a startup-only TF).

The problem is, it will not disable/enable missing index stats collection unless it is enabled at startup. If you set it as a startup TF and restart SQL Server, then no missing index stats are collected. If you then subsequently disable TF 2392 while SQL Server is running, it still won’t collect any missing index stats (despite what you may expect).

You have to remove the startup TF 2392 and then restart SQL Server for the change to actually take effect (both ways, enabling and disabling). This is from my testing on SQL Server 2017 CU2.

Personally, I don’t think this is a big deal (unless you really need to enable the workaround described in the KB article). I think most people are going to want to leave the missing index feature enabled. The performance overhead of the feature is probably pretty insignificant.

You just need to be aware of how TF 2392 actually works and know that a restart of SQL Server is required to actually turn the missing index feature off or back on. It would be nice if you did get a message about it being a startup-only trace flag if you tried to enable/disable it while SQL Server was running.