SQLskills SQL101: Query plans based on what’s in memory

As Kimberly blogged about recently, SQLskills is embarking on a new initiative to blog about basic topics, which we’re calling SQL101. We’ll all be blogging about things that we often see done incorrectly, technologies used the wrong way, or where there are many misunderstandings that lead to serious problems. If you want to find all of our SQLskills SQL101 blog posts, check out SQLskills.com/help/SQL101.

One of the topics that I discuss in class today is why the query optimizer doesn’t know (or care) what’s in the buffer pool. (The query optimizer is the part of the query processor that’s responsible for compiling an efficient query plan, and the buffer pool is the cache of database data file pages that are in memory.)

Let’s investigate…

Scenario

Here’s a scenario:

  • Table T has two nonclustered indexes, A and B, that both cover query Q (a simple SELECT query)
  • Query Q will require a complete index scan of either index
  • Index A has 10,000 pages at its leaf level
  • Index B has 50,000 pages at its leaf level

Which index will the query optimizer use when compiling the query plan?

Cost-based…

SQL Server uses a cost-based optimizer, which uses various metrics and statistics to determine the most efficient query plan for the query (given the time limits imposed on its search of the space of all possible query plans). The ‘cost’ in ‘cost-based’ means that it considers the CPU cost and I/O cost of the various operators in the query plan, with the I/O cost essentially being relative to the number of physical reads required. And it assumes that nothing is in memory.

In the scenario above, the optimizer will choose a query plan using index A, as the most efficient plan will be the one involving the fewest phsyical reads and with such a large difference between the page counts of indexes A and B, index A will be chosen for sure.

Hypothetical memory-based…

Now let’s allow a hypothetical optimizer to base its plan choice on what’s in the buffer pool.

If index A is mostly not in the buffer pool and index B is mostly in the buffer pool, it would be more efficient to compile the query plan to use index B, for a query running at that instant. Even though index B is larger, and would need more CPU cycles to scan through, physical reads are waaaay more expensive (in terms of elapsed time) than CPU cycles so a more efficient query plan is the one that minimizes the number of physical reads.

This argument only holds, and a ‘use index B’ query plan is only more efficient than a ‘use index A’ query plan, if index B remains mostly in memory, and index A remains mostly not in memory. As soon as the relative proportions of indexes A and B that are in memory become such that the ‘use index A’ query plan would be more efficient, the ‘use index B’ query plan is the wrong choice.

The situations when the compiled ‘use index B’ plan is less efficient than the cost-based ‘use index A’ plan are (generalizing):

  • Indexes A and B are both memory resident: the compiled plan will use roughly 5 times more CPU than the optimal plan, as there are 5 times more pages to scan.
  • Neither index is memory resident: the compiled plan will do 5 times the number of physical reads AND use roughly 5 times more CPU.
  • Index A is memory resident and index B isn’t: all physical reads performed by the plan are extraneous, AND it will use roughly 5 times more CPU.

This means that the ‘use index B’ plan is really only the optimal plan at the time the query was compiled.

So although a hypothetical optimizer could make use of buffer pool contents knowledge to compile a query that is the most efficient at a single instant, it would be a very dangerous way to drive plan compilation because of the potential volatility of the buffer pool contents, making the future efficiency of the cached compiled plan highly unreliable.

And I also haven’t mentioned the extra cost of maintaining buffer pool contents knowledge in real time, and then potentially having to recompile queries that are now deemed to be inefficient because buffer pool contents have changed.

Summary

Although it doesn’t always get it right, the optimizer strives to produce the most efficient plan, assuming nothing is in the buffer pool. Understanding how the query optimizer comes to plan choice decisions is extremely useful for understanding query plans themselves and relating them to the code driving the plan.

Hope you found this helpful!

How are default column values stored?

An interesting question came up in class yesterday: how is a default column value stored, and what if some rows exist when a column is added and then the default value changes?

An example scenario is this:

  • Step 1: Create a table with two columns
  • Step 2: Add 10 rows
  • Step 3: Add a third column to the table with a non-null default
  • Step 4: Drop the default for the third column
  • Step 5: Add a new default for the third column

And selecting the initial 10 rows can be demonstrated to return the 3rd column using the initial default set in step 3. (It makes no difference if any rows are added between steps 3 and 4.)

This means that there *must* be two default values stored when a new column is added: one for the set of already-existing rows that don’t have the new column and one for any new rows. Initially these two default values will be the same, but the one for new rows can change (e.g. in steps 4 and 5 above) with breaking the old rows. This works because after the new column is added (step 3 above), it’s impossible to add any more rows that *don’t* have the new column.

And this is exactly how it works. Let’s investigate!

Firstly I’ll create a simple database and test table and insert 10 rows. I’ll use the simple recovery model so I can clear the log by doing a checkpoint:

CREATE DATABASE [Company];
ALTER DATABASE [Company] SET RECOVERY SIMPLE;
GO
USE [Company];
GO

-- Create a test table to use
CREATE TABLE [Test] ([c1] INT IDENTITY, [c2] INT AS ([c1]));
GO
INSERT INTO [Test] DEFAULT VALUES;
GO 10
SELECT * FROM [Test];
GO
c1          c2
----------- -----------
1           1
2           2
3           3
4           4
5           5
6           6
7           7
8           8
9           9
10          10

Now I’ll clear the log, add the third column with the default, and look to see which system tables were added to because of addition:

CHECKPOINT;
GO

-- Add column with default value
ALTER TABLE [Test] ADD [c3] CHAR (6) NOT NULL CONSTRAINT [OriginalDefault] DEFAULT 'BEFORE';
GO

SELECT [AllocUnitName] FROM fn_dblog (NULL, NULL)
WHERE [Operation] = 'LOP_INSERT_ROWS';
GO
AllocUnitName
-----------------------
sys.syscolpars.clst
sys.syscolpars.nc
sys.sysrscols.clst
sys.sysseobjvalues.clst
sys.sysschobjs.clst
sys.sysschobjs.nc1
sys.sysschobjs.nc2
sys.sysschobjs.nc3
sys.sysobjvalues.clst

Cool. These system tables have the following functions:

  • sys.syscolpars: table column definitions (relational metadata)
  • sys.sysrscols: rowset column definitions (Storage Engine metadata – info to allow interpretation of record structures on pages)
  • sys.seobjvalues: various Storage Engine values of different uses
  • sys.sysschobjs: relational objects (e.g. tables, constraints)
  • sys.sysobjvalues: various relational values of different uses

I’m going to look in these to find how the inserted rows relate to our table. I’m going to need a few IDs first (using my handy procedure to do that):

EXEC sp_allocationmetadata N'Test';
GO
Object Name Index ID Partition ID      Alloc Unit ID     Alloc Unit Type First Page Root Page First IAM Page
----------- -------- ----------------- ----------------- --------------- ---------- --------- --------------
Test        0        72057594040549376 72057594045792256 IN_ROW_DATA     (1:247)    (0:0)     (1:288)

And now I can query those system tables. Note that they’re ‘hidden’ system tables so you can’t query them unless you connect using the Dedicated Admin Connection. Easiest way to do that is to prefix your SSMS connection string with ‘admin:’ (and if you’re connecting to a remote server, the server needs to have sp_configure option remote admin connections enabled). Make sure you use the correct database after connecting, as the DAC drops you into master.

SELECT * FROM sys.syscolpars WHERE [id] = OBJECT_ID (N'Test');
GO
id        number colid name xtype utype length prec scale collationid status maxinrow xmlns dflt        chk idtval
--------- ------ ----- ---- ----- ----- ------ ---- ----- ----------- ------ -------- ----- ----------- --- ---------
245575913 0      1     c1   56    56    4      10   0     0           5      4        0     0           0   0x0A000000010000000100000000
245575913 0      2     c2   56    56    4      10   0     0           209    4        0     0           0   NULL
245575913 0      3     c3   175   175   6      0    0     872468488   3      6        0     261575970   0   NULL

These are the relational definitions of the columns in the table, and you can see that c3 is listed as having a default constraint, with ID 261575970.

SELECT * FROM sys.sysschobjs WHERE [id] = 261575970;
GO
id          name            nsid nsclass status type pid       pclass intprop created                 modified                status2
----------- --------------- ---- ------- ------ ---- --------- ------ ------- ----------------------- ----------------------- -------
261575970   OriginalDefault 1    0       131072 D    245575913 1      3       2017-04-26 13:37:42.463 2017-04-26 13:37:42.463 0

This is the constraint named OriginalDefault with type D (default) and the default value has ID 245575913.

SELECT * FROM sys.sysobjvalues WHERE [objid] = 261575970;
GO
valclass objid     subobjid valnum value imageval
-------- --------- -------- ------ ----- ----------------------
1        261575970 0        0      2     0x28274245464F52452729

And the imageval column has the default value as hex-encoded ASCII values. Using the ASCII map on Wikipedia, the value is (‘BEFORE’), including the parentheses.

So that’s the default value for new rows. What about the default value for rows that already exist?

SELECT * FROM sys.sysrscols WHERE [rsid] = 72057594040549376;
GO
rsid              rscolid hbcolid rcmodified ti   cid       ordkey maxinrowlen status offset nullbit bitpos colguid
----------------- ------- ------- ---------- ---- --------- ------ ----------- ------ ------ ------- ------ -------
72057594040549376 1       1       10         56   0         0      0           128    4      1       0      NULL
72057594040549376 2       2       10         56   0         0      0           128    8      2       0      NULL
72057594040549376 3       3       0          1711 872468488 0      0           640    12     3       0      NULL

These are the Storage Engine definitions of the columns in the table. The status value indicates that the value may not be in the row, and where to get the default value from.

SELECT * FROM sys.sysseobjvalues WHERE [id] = 72057594040549376;
GO
valclass id                subid valnum value  imageval
-------- ----------------- ----- ------ ------ --------
1        72057594040549376 3     0      BEFORE NULL

And there is the Storage Engine storage for the default value for the c3 column for those rows that existed before c3 was added.

Now I’ll checkpoint, drop the default constraint, and look to see what happened in the log:

CHECKPOINT;
GO

ALTER TABLE [Test] DROP CONSTRAINT [OriginalDefault];
GO

SELECT [AllocUnitName] FROM fn_dblog (NULL, NULL)
WHERE [Operation] = 'LOP_DELETE_ROWS';
GO
AllocUnitName
---------------------
sys.sysobjvalues.clst
sys.sysschobjs.nc1
sys.sysschobjs.nc2
sys.sysschobjs.nc3
sys.sysschobjs.clst

So that’s the relational default value being deleted, in the reverse order from how it was added. Note that the Storage Engine default value wasn’t deleted.

Now I’ll create a new default constraint for the c3 column:

CHECKPOINT;
GO

ALTER TABLE [Test] ADD CONSTRAINT [NewDefault] DEFAULT 'AFTER' FOR [c3];
GO

SELECT [AllocUnitName] FROM fn_dblog (NULL, NULL)
WHERE [Operation] = 'LOP_INSERT_ROWS';
GO
AllocUnitName
---------------------
sys.sysschobjs.clst
sys.sysschobjs.nc1
sys.sysschobjs.nc2
sys.sysschobjs.nc3
sys.sysobjvalues.clst

And doing the various queries again gets us to the new relational column stored default value of (‘AFTER’), including the parentheses.

So just to prove what I said before investigating, I’ll add ten new rows, which will have the c3 value AFTER, and then query the table and I’ll see that the initial ten rows that don’t have c3 in will be given the original default value of BEFORE:

INSERT INTO [Test] DEFAULT VALUES;
GO 10

SELECT * FROM [Test];
GO
c1          c2          c3
----------- ----------- ------
1           1           BEFORE
2           2           BEFORE
3           3           BEFORE
4           4           BEFORE
5           5           BEFORE
6           6           BEFORE
7           7           BEFORE
8           8           BEFORE
9           9           BEFORE
10          10          BEFORE
11          11          AFTER 
12          12          AFTER 
13          13          AFTER 
14          14          AFTER 
15          15          AFTER 
16          16          AFTER 
17          17          AFTER 
18          18          AFTER 
19          19          AFTER 
20          20          AFTER 

Hope you found this interesting! (And don’t forget to drop your DAC connection.)

October 2017 classes in Chicago open for registration

I’ve just released our final set of 2017 classes for registration, including the new two-day class on Azure, the new three-day class on upgrading and migrating, and the new two-day class on clustering and availability groups.

Our classes in October will be in Chicago, IL:

  • IEPTO1: Immersion Event on Performance Tuning and Optimization – Part 1
    • October 2-6
  • IESSIS1: Immersion Event on Learning SQL Server Integration Services
    • October 2-6
  • IE0: Immersion Event for Junior/Accidental DBAs
    • October 2-4
  • ** NEW ** IECAG: Immersion Event on Clustering and Availability Groups
    • October 5-6

    IEPTO2: Immersion Event on Performance Tuning and Optimization – Part 2

    • October 9-13
  • IEPS: Immersion Event on PowerShell for SQL Server DBAs
    • October 9-11
  • ** NEW ** IEAzure: Immersion Event on Azure SQL Database and Azure VMs
    • October 9-10
  • ** NEW ** IEUpgrade: Immersion Event on Upgrading SQL Server
    • October 11-13

You can get all the logistical, registration, and curriculum details by drilling down from our main schedule page.

We hope to see you there!

SQLskills SQL101: Restoring to an earlier version

As Kimberly blogged about recently, SQLskills is embarking on a new initiative to blog about basic topics, which we’re calling SQL101. We’ll all be blogging about things that we often see done incorrectly, technologies used the wrong way, or where there are many misunderstandings that lead to serious problems. If you want to find all of our SQLskills SQL101 blog posts, check out SQLskills.com/help/SQL101.

One of the questions I get asked every so often is whether it’s possible to attach or restore a database to an earlier version of SQL Server. Usually the explanation behind the question is that the person accidentally attached the only copy of their database to a newer version than they wanted, or they were just trying out a pre-release version and now want to put their database back into their production system.

So is this possible? The very simple answer is: No.

SQL Server is down-level compatible but is not up-level compatible. This means you can take a database from an earlier version and attach/restore it to a newer version (I explained about this in a post here), but you can’t go backwards to an earlier version.

Why is this the case?

Upgrade steps

An upgrade, whether intentional or accidental, is a one-way operation and it is extremely difficult to reverse its effects. When you upgrade between versions of SQL Server, a series of upgrade steps are performed on the database. Each step usually involves some physical changes to the database, and each step increases the physical version number of the database.

For example, one of the major changes performed when a database was upgraded from SQL Server 2000 to SQL Server 2005 (yes, old and unsupported, but an easy-to-explain example) was to change the structure of the database’s system catalogs (often called the system tables or database metadata) that hold various metadata about tables, indexes, columns, allocations, and other details regarding the relational and physical structure of the database.

As each of these upgrade steps is performed, the database version number is increased. Here are some examples:

  • SQL Server 2016 databases have version number 852
  • SQL Server 2014 databases have version number 782
  • SQL Server 2012 databases have version number 706
  • SQL Server 2008 R2 databases have version number 661

This version number allows SQL Server to know the last upgrade step performed on the database, and whether the in-use SQL Server version can understand the database being attached/restored.

Here’s an example of restoring a SQL Server 2012 database to a SQL Server 2014 server:

RESTORE DATABASE [Company2012]
FROM DISK = N'D:\SQLskills\Company2012_Full.bak'
WITH REPLACE;
GO
Processed 280 pages for database 'Company', file 'Company' on file 1.
Processed 3 pages for database 'Company', file 'Company_log' on file 1.
Converting database 'Company' from version 706 to the current version 782.
Database 'Company' running the upgrade step from version 706 to version 770.
Database 'Company' running the upgrade step from version 770 to version 771.
Database 'Company' running the upgrade step from version 771 to version 772.
Database 'Company' running the upgrade step from version 772 to version 773.
Database 'Company' running the upgrade step from version 773 to version 774.
Database 'Company' running the upgrade step from version 774 to version 775.
Database 'Company' running the upgrade step from version 775 to version 776.
Database 'Company' running the upgrade step from version 776 to version 777.
Database 'Company' running the upgrade step from version 777 to version 778.
Database 'Company' running the upgrade step from version 778 to version 779.
Database 'Company' running the upgrade step from version 779 to version 780.
Database 'Company' running the upgrade step from version 780 to version 781.
Database 'Company' running the upgrade step from version 781 to version 782.
RESTORE DATABASE successfully processed 283 pages in 0.022 seconds (100.430 MB/sec).

Up-level compatibility (or lack thereof…)

Versions of SQL Server cannot read databases upgraded to more recent versions of SQL Server – for instance, SQL Server 2012 cannot read a database that’s been upgraded to SQL Server 2014. This is because older versions do not have the code needed to interpret the upgraded structures and database layout.

Here’s an example of trying to restore a SQL Server 2014 database to a SQL Server 2012 server:

RESTORE DATABASE [Company2014]
FROM DISK = N'D:\SQLskills\Company2014_Full.bak'
WITH REPLACE;
GO
Msg 3169, Level 16, State 1, Line 51
The database was backed up on a server running version 12.00.4422. That version is incompatible with this server, which is running version 11.00.5343. Either restore the database on a server that supports the backup, or use a backup that is compatible with this server.
Msg 3013, Level 16, State 1, Line 51
RESTORE DATABASE is terminating abnormally.

In earlier versions, the messages weren’t always quite as nice and easy to understand.

And some people confuse database compatibility level with the database version. Compatibility level has nothing to do with up-level compatibility – it just changes how some query processor features behave.

Summary

The simple thing to bear in mind is not to attach the only copy of your database to a newer version. It’s always better to restore a copy of a database, then you’ve still got the original database to fall back on, for whatever reason. This applies even if you’re deliberately performing an upgrade – I’d still want to keep the older copy of the database around in case some problem occurs with the upgrade.

If you *have* attached your only copy of the database to a newer version and want to go back to an earlier version, your only option is to script out the database structure, create the database again on the older version, and then transfer all the data from the newer version to the older version. Very tedious.

Hope you found this helpful!

PS There’s a comment below asking whether you can move back to an earlier SP or CU. Yes, for user databases, as long as the newer SP/CU didn’t change the physical version number (and none of them since 2005 SP2 and 2008 SP2 have done that).

New course: Understanding and Using Azure SQL Database

And to coincide with our new training class on Azure, we’ve just published a new Pluralsight course on Azure too!

Tim’s latest Pluralsight course – SQL Server: Understanding and Using Azure SQL Database – and is just under two hours long and is based on his very popular user group and conference session.

The modules are:

  • Introduction
  • Understanding Azure SQL Database Features
  • Exploring Unsupported Features
  • Understanding Azure SQL Database Pricing
  • Migrating Data to Azure SQL Database
  • Monitoring and Tuning Azure SQL Database

Check it out here.

We now have 150 hours of SQLskills online training available (see all our 51 courses here), all for as little as $29/month through Pluralsight (including more than 5,000 other developer and IT training courses). That’s unbeatable value that you can’t afford to ignore.

Enjoy!

New class: Immersion Event on Upgrading SQL Server

We have a second exciting new class debuting this October in Chicago: Immersion Event on Upgrading SQL Server.

It’s a 3-day class, taught by Glenn Berry and Tim Radney. We’ve seen a huge surge in clients upgrading from out-of-support versions of SQL Server, so this class will be really useful to many organizations.

The modules are as follows:

  • Upgrade Planning
  • Hardware and Storage Selection and Configuration
  • SQL Server 2016 Installation and Configuration
  • Upgrade Testing
  • Migration Planning
  • Migration Testing
  • Production Migration Methods

You can read a more detailed curriculum here and all the class registration and logistical details are here.

We hope to see you there!

New class: Immersion Event on Azure SQL Database and Azure VMs

We have a really cool new class debuting this October in Chicago: Immersion Event on Azure SQL Database and Azure VMs.

It’s a 2-day class, taught by Tim Radney. Azure is getting more and more popular, and we’re seeing many clients using it.

The modules are as follows:

  • Azure Virtual Machines
  • Migrating to Azure Virtual Machines
  • Azure SQL Database
  • Migrating to Azure SQL Database
  • Additional Azure Features

You can read a more detailed curriculum here and all the class registration and logistical details are here.

We hope to see you there!

SQLskills SQL101: Switching recovery models

As Kimberly blogged about recently, SQLskills is embarking on a new initiative to blog about basic topics, which we’re calling SQL101. We’ll all be blogging about things that we often see done incorrectly, technologies used the wrong way, or where there are many misunderstandings that lead to serious problems. If you want to find all of our SQLskills SQL101 blog posts, check out SQLskills.com/help/SQL101.

One of the things that can catch people out is the effect of switching out of the full recovery model temporarily. In this post I’ll briefly describe the three recovery models and then the problems you can have switching from full to simple, and from full to bulk-logged.

Recovery models

There are three recovery models:

  • Full recovery model (the default and the most commonly used)
    • All modifications in the database a fully logged. This doesn’t mean that every modification has a separate log record, as some operations are logged with fewer log records while still logging the entire effect of the operation (for instance, TRUNCATE TABLE operations – see here for a deep explanation).
    • The transaction log will not clear (i.e. portions are available for reuse) until a transaction log backup is performed (see here for a deep explanation).
    • All recovery options are available when a database is in the full recovery model (and has been since the last backup).
  • Bulk-logged recovery model
    • Some modifications (like an index rebuild or a bulk load, but NOT regular insert/update/deletes) can be minimally logged, which reduces the amount of log records generated so the transaction log does not have to grow really large during the operation. Note that this doesn’t change the size of subsequent log backups. For full instructions on how to get minimal logging for your operation, see the Data Loading Performance Guide whitepaper, which lists all the various conditions that have to be met.
    • The transaction log will not clear until a transaction log backup is performed (exactly the same as the full recovery model).
    • Using bulk-logged, you trade off some recovery options (point-in-time restore and tail-of-the-log backups) for the performance gains associated with minimally logged operations.
  • Simple recovery model
    • Some modifications can be minimally logged (exactly the same as the bulk-logged recovery model).
    • The log will not clear until a checkpoint occurs (usually automatically).
    • Transaction log backups are not possible, so this is the most limited in terms of recovery options.

Most people use the full recovery model, to allow log backups and permit all possible restore operations. The main thing to remember is that if your database uses the full or bulk-logged recovery model, you must perform periodic transaction log backups or the transaction log will grow forever.

Some circumstances call for simple; if you don’t need the ability to do point-in-time restore or zero-to-minimal data loss restores using log backups. An example would be a scratch database that’s repopulated once per day and any changes can be lost or easily regenerated.

Switching to Simple

Often I hear of people who switch to the simple recovery model  to try to avoid transaction log growth during a bulk load or index rebuild, when what they really mean to do is to use the bulk-logged recovery model. There are also persistent myths out there that some regular operations *require* being in the simple recovery model – this is simply (ha ha) not true.

Switching to the simple recovery model breaks the log backup chain, requiring a full or differential backup before any further log backups can be performed.

Furthermore, it limits your ability to recover during a disaster because you’ve now only got one full backup from which you can restore: the one you performed most recently. Think about it: your restore options become:

  • Full backup after switch to simple, plus the latest differential backup after that full (if you’re using differential backups) and any log backups since the switch back; or
  • Most recent full backup before switch to simple, plus the latest differential after the switch back from simple, plus any log backups

If that most-recent full backup (before or after the switch to simple) is damaged, you cannot restore – period. You can’t fall back on using the next older full backup, as that only allows the restore sequence up to, but not past, the switch to simple. Well, I guess you could do that, but then you lose all work since the switch to simple.

Switching to the simple recovery model is not something you automate or do repeatedly. About the only time when you would temporarily switch to simple is if your transaction log had run out of space and there is no way to allow it to clear (i.e. you cannot perform a log backup or add another log file) except by switching to simple and forcing a checkpoint operation. In that case you’re taking a drastic step to allow operations to continue, and being fully cognizant of the limited restore options available to you right then.

Unless you have this emergency situation, or you decide to use the simple recovery model permanently, you should not switch to simple ever.

Switching to Bulk-logged

Switching to bulk-logged during a load or index maintenance process is acceptable to avoid transaction log growth. In fact, switching back-and-forth between full and bulk-logged repeatedly doesn’t affect the log backup chain in any way. And doing so also doesn’t have any effect on log shipping or replication, but you can’t switch out of full when using database mirroring or an availability group as they mandate the full recovery model.

However, using bulk-logged can cause problems for disaster recovery, so even though its behavior may be desirable, you may need to avoid using it so you don’t risk compromising your disaster recovery options.

Problem 1: a log backup that contains a minimally-logged operation cannot be used during a point-in-time restore. This means the time you specify in the WITH STOPAT clause of the restore statement cannot be a time covered by such a log backup. You can use that log backup as part of a restore sequence, and stop at any point in time after it (as long as that point in time is not covered by another log backup containing a minimally-logged operation, of course), but just not during it.

Problem 2: if you need to perform a tail-of-the-log backup to capture all the log generated since the most recent scheduled log backup, the data files are inaccessible or damaged, and the log to be backed up contains a minimally-logged operation, that backup will fail prior to SQL Server 2008 R2, and from SQL Server 2008 R2 onward it will succeed, but be will corrupt the database when restored.

So if you’re going to use bulk-logged to save on log space during large operations, you need to make sure that a) there’s no possibility you’re going to want to restore between the last log backup and the next one, and b) there are no changes made to the database that you cannot recreate in case a disaster occurs and you can’t take a valid tail-of-the-log backup.

Switching recovery models between full and bulk-logged may not be as safe as you might think.

Summary

For every database that you’re responsible for, make sure that you understand the ramifications of changing the recovery model, as doing so could cause you problems with disaster recovery.

Do you get our bi-weekly newsletter?

You may not know this, but for the last six years I’ve been producing a bi-weekly newsletter for everyone signed up for our mailing list (which we only use for the newsletter). It’s a bit of a labor of love, and a way for me to post opinions and other information that doesn’t really fit on my blog.

I’ve just expanded the sections in the newsletter so I want to make sure you all know about it. It’s 100% free of any 3rd-party adverts and marketing and your email address will only ever be used for sending the newsletters – nothing else. And it’s 100% free to join, no special requirements.

Every newsletter has the following sections:

  • SQLskills News A quick round-up of new Pluralsight courses, classes, conferences, user groups, and SQLSaturdays where we’ll be presenting.
  • Book Review I read a huge amount and like to post a 1-2 paragraph review of something interesting I’ve read since the last newsletter. I’ve been told by many people that this is one of the sections they look forward to most!
  • The Curious Case of… Describing and explaining a problem we’ve recently found on a client system (or come across in an email question or while teaching a class) that is something others are likely to come across in their environments.
  • Paul’s Ponderings Where I write about a topic of interest to the SQL Server community.
  • Video Demo A never-before-seen demo video of something to do with SQL Server from one of the team. Occassionally I’ll pick a demo video from a Pluralsight course too.
  • #TBT (Turn Back Time) Highlighting some older resources around a particular topic, plus a selection of recent SQLskills blog posts.

Here’s an example newsletter from March 2017 so you can see what it’s like.

If you’re interested, just enter your email address below and hit Subscribe and you’ll be added to our newsletter list. No in-depth registration, no fuss, no time! Once you sign up you’ll be able to read all the previous newsletters too.

Enjoy!

SQLskills SQL101: Temporary table misuse

As Kimberly blogged about recently, SQLskills is embarking on a new initiative to blog about basic topics, which we’re calling SQL101. We’ll all be blogging about things that we often see done incorrectly, technologies used the wrong way, or where there are many misunderstandings that lead to serious problems. If you want to find all of our SQLskills SQL101 blog posts, check out SQLskills.com/help/SQL101.

After I left Microsoft in 2007, one of the first clients I worked with (who we’re still working with today) threw an interesting problem at me: “We can’t run stored proc X any more because it causes tempdb to fill the drive and then it fails.” Game on. I built some monitoring infrastructure into the proc using the DMV sys.dm_db_task_space_usage to figure out how much tempdb space was being used at various points and find the problem area.

It turned out to be problem *areas*, and in fact the proc was loaded with temporary table (I’ll just use the common contraction ‘temp table’ from now on) misuse, illustrating all three of the common temp table problems I’m going to describe below. Once I fixed that proc, (reducing the tempdb usage from more than 60GB down to under 1GB, and the run time from many minutes to a few tens of seconds) I implemented some automated monitoring built around the sys.dm_db_task_space_usage DMV to identify procedures and ad hoc queries that were misusing temp tables. We’ve since used this monitoring at many other clients to identify temp table misuse.

In this post I’d like to describe the three main ways that temp table are misused:

  • Over-population of temp tables
  • Incorrect indexing on temp tables
  • Using a temp table where none are required

Don’t get me wrong though – temp tables are great – when they’re used efficiently.

Over-Population of a Temp Table

This problem involves creating a temp table using something like a SELECT … INTO #temptable construct and pulling far more data into the temp table than is necessary.

The most common thing we see is pulling lots of user table columns into the temp table, where some of the columns are not used ever again in subsequent code. This is a HUGE waste of I/O and CPU resources (extracting the columns from the user table in the first place – and imagine the extra CPU involved if the source data is compressed!) and a big waste of tempdb space (storing the columns in the temp table). I’ve seen code pulling large varchar columns into a temp table that aren’t used, and with multi-million row datasets…

The other facet of over-population of temp tables is pulling in too many rows. For instance, if your code is interested in what happened over the last 12 months, you don’t need to pull in all the data from the last ten years. Not only will it be bloating the temp table, it will also drastically slow down the query operations. This was one of the biggest problems in the client scenario I described above.

The key to better performance is making sure your selection/projection is as focused as possible. To limit your selection, use an effective WHERE clause. To limit your projection, list only the necessary columns in your select list.

Incorrect Indexing on a Temp Table

This problem involves either creating indexes before populating the table (so that no statistics are generated) or creating a bunch of inappropriate indexes that are not used.

The most common example we see is creating a single-column nonclustered index for each of the temp table columns. Those are usually just taking up space for no use whatsoever. Temp tables *DO* need indexes (preferably after load) but as with any form of query tuning – only the RIGHT indexes. Consider creating permanent tables that mimic what’s going on in your temporary objects and then using the Database Tuning Advisor (DTA) to see if it has recommendations. While DTA’s not perfect, it’s often WAY better than guessing. Kimberly has a great post in our Accidental DBA series that discusses indexing strategies – start there.

Also, don’t create any nonclustered indexes until the temp table has been populated, otherwise they won’t have any statistics, which will slow down query performance, possibly drastically.

Oh yes, and, don’t create a clustered index for the temp table before populating it unless you know that the data being entered is already sorted to exactly match the cluster key you’ve chosen. If not, inserts into the temp table are going to cause index fragmentation which will really slow down the time it takes to populate the temp table. If you know the data is sorted and you create the clustered index first, there’s still no guarantee that the Storage Engine will feed the data into the temp table in the right order, so be careful. And if you go that route, remember that you’ll need to update the statistics of the clustered index after the temp table creation.

You need to be careful here because in some versions of SQL Server, changing the schema of a temp table in a stored proc can cause recompilation issues. Do some testing and pick the sequence of events that makes the most sense for performance in your situation.

Using a Temp Table Where None is Required

The SQL Server Query Optimizer is a fabulous beast and is very good at figuring out the most efficient way to execute most queries. If you choose to take some of the query operation and pre-calculate it into a temp table, sometimes you’re causing more harm than good. Any time you populate a temp table you’re forcing SQL Server to materialize the complete set of results of whatever query you ran to populate the temp table. This can really limit SQL Server’s ability to produce a pipeline of data flowing efficiently through a query plan and making use of parallelism and collapsing data flows when possible.

While it’s true that you might be able to do better than the optimizer sometimes, don’t expect that it’s the case all the time. Don’t just go straight to using temp tables, give the optimizer a chance – and, make sure to retest your code/expectations around Service Packs and hot fixes as these may have eliminated the need for temp tables as well.

A good way to test whether a temp table is actually a hindrance to performance is to take the tempdb-creation code, embed it as a derived table in the main query, and see if query performance improves.

It’s quite often the case that temp tables because an architectural standard in an environment when they proved useful long ago and now everyone used them, without ever checking if they’re *really* good for all cases.

One other thing you can consider is replacing temp tables with In-Memory OLTP  memory-optimized tables, in all Editions of SQL Server 2016 SP1 and later, and in Enterprise Edition of SQL Server 2014. That’s beyond the scope of this post, but you can read about it in this Books Online page on MSDN.

Summary

Always try to follow these guidelines when using a temp table:

  • Determine if a temp table is the most efficient way to achieve the goal of the code you’re writing
  • Limit the number of columns being pulled into the temp table
  • Limit the number of rows being pulled into the temp table
  • Create appropriate indexes for the temp table

Take a look at your current temp table usage. You may be surprised to find a lot of tempdb space and CPU resources being consumed by inappropriate temp table usage, population, and indexing.

Hope you found this helpful!