This is a tough topic. It's a big topic and more than any other – I think there are a lot of misunderstandings about what the log is for, why it's so critical and ESPECIALLY when/why it gets extrememly large. Simply put, it gets large when it's not managed correctly. OK, there are times when it can become large – even if it is well managed. But, more often than not, when a transaction log is wildly out of control (orders of magnitude larger than the data itself) it indicates a management/maintenance problem.
There are a lot of places where you can go to find out the technical details behind the transaction log but I'm going to target this blog post to the relatively straightforward easy (no, really easy!!) facts about transaction log maintenance.
What kind of transaction log management is right for YOUR database?
First and foremost, you MUST decide whether or not you need to do log backups. SQL Server *requires* you to make some form of decision. Well, I take that back. They don't tell you anywhere that you need to make this decision but the transaction log can get wildly out of control if you don't (see the next section for more details on this one :)).
Why? Transaction log backups will allow you better recovery options in the event of a disaster. If you create a good backup strategy, you should be able to recover from a disaster very close (possibly even up-to-the-minute) to the time of the disaster. Howevre, you are not required to do log backups. Instead you can do only database-level backups and recover with those. That's fine. There's really nothing wrong with that strategy. However, it does mean that you have a greater potential for data loss. Basically, if you decide that you're doing to do weekly full backups – then you need to be OK with losing everything that's happened since your last full backup. If that's OK, then performing full database backups (and never worrying about the log) is absolutely fine.
However, if you want more granular control and more recovery options (again, possibly even up-to-the-minute recovery – which is transactional recovery up to the time of the disaster), then you MUST add transaction log backups into your disaster recovery strategy.
So, make this decision FIRST:
Am I OK with some data loss? (then you're probably OK with just database-level backups… but, you will need to do something else! be sure to keep reading!!!)
Do I want to minimize data loss to the smallest amount possible? (then you're going to want to AUTOMATE transaction log backups)
But I didn't do anything – why is the log WAY out of control (in terms of size)?
OK, even if you consciously make the decision to ONLY do database-level backups, you are NOT DONE!!! In fact, this is actually what led me to do this post. I found these two (relatively dated but interesting nonetheless) MSDN forum discussions for TFS (Team Foundation Server) databases:
MSDN Forum discussion "Recommended SQL Maintenance Plan": http://social.msdn.microsoft.com/forums/en-US/tfsadmin/thread/b23f7018-3eaa-4596-96e4-728b02cf6211/
MSDN Forum discussion "Huge log files": http://social.msdn.microsoft.com/forums/en-US/tfsadmin/thread/605d51f7-23fd-470c-945e-53fa7ed5aa87/
And, I know EXACTLY what happened in ALL of these cases (and MANY more… Paul and I see this ALL the time, in fact). In the "Huge log files" thread, there's a database mentioned (TfsWareHouse) with a 124MB mdf and a transaction log of 61.8GB. It didn't mention whether or not there were other data files but my guess is that there weren't. My guess is that they were completely shocked by why the data portion had grown to a size that's 510 TIMES the size of the database… The reason is actually somewhat simple (no pun intended). If you're not going to do transaction log maintenance (meaning transaction log backups), then you need to tell SQL Server that. (This is the part that's completely unexpected.)
When a database is created, SQL Server runs that database in a "pseudo simple recovery model". (Yes, I know – that didn't help.) What that means is that SQL Server automatically clears inactive records from the transaction log once it knows that it no longer needs them. It no longer needs them to be stored in the log because no one is using the log (i.e. you're not doing ANY backups). However, once you do start to do backups (and, people generally start by doing a full database backup), then SQL Server looks to your recovery model to determine what to do with log records. If the recovery model is set to full (and, yes, this is the default), then SQL Server gives you the "full feature set" with regard to backup/restore. SQL Server is expecting YOU to manage the transaction log by backing it up. Once it's backed up, SQL Server can remove the inactive records from the transaction log (and when you do a transaction log backup, it automatically clears the inactive records by default).
So, there are really two choices – and ONLY two choices here:
- Perform transaction log backups as part of your maintenance plan
- Change the recovery model to the SIMPLE recovery model so that SQL Server clears inactive transactions from the log automatically
Is there anything else to do for the transaction log?
Yes! If you decide that you want to do transaction log backups then I would recommend a few things. I'd first recommend reading 8 Steps to Better Transaction Log Throughput and when you decide how large your transaction log needs to be, then also read Transaction Log VLFs – too many or too few?. These two posts will help you to create a more appropriately sized log as well as one that won't be prone to performance problems (such as internal VLF fragmentation).
If you want to learn more about the transaction log, I'd suggest a few of Paul's resources (it's probably because he has such a fantastic tech editor… oh, I'm asking for trouble with this comment!! ;-):
- Read Paul’s blog post to his TechNet article on Logging & Recovery. It’s a great article that covers a lot of different aspects of logging. He also did a great short video on why the transaction log grows wildly out of control. Here’s a link to the blog post that pulls all of the TechNet resources together: https://www.sqlskills.com/blogs/PAUL/post/TechNet-Magazine-feature-article-on-understanding-logging-and-recovery.aspx.
- Read Paul’s blog post to his TechNet article on Database Maintenance. It’s a great overview of all of these maintenance tasks and will give you a good overview of what each one does. Here’s a link to the blog post that pulls all of the TechNet resources together: https://www.sqlskills.com/blogs/PAUL/post/TechNet-Magazine-Effective-Database-Maintenance-article-and-August-SQL-QA-column.aspx
OK, so, I think that sums up part III. I think that's the last one in the series for now. I'll go through and explain "The Tipping Point" next. However, I was hoping for more results to my brain teasers (in those two posts)!!
7 thoughts on “Database Maintenance Best Practices Part III – Transaction Log Maintenance”
Transaction Logs are funny – because once you have a good understanding of what they do, and how they work they’re simple. But if you don’t, they seem almost mystical.
For anyone following up on this post, I’ve got a couple of free SQL Server Videos showing how they work too:
I’ve been changing the recovery model for my ‘Model’ database from Full to Simple, to prevent new databases being created with a Full recovery model by default. Seems like there are fewer pitfalls to this approach. At least runaway transaction logs will tend to be prevented. Had a couple problems with that in our ‘getting feet wet with SQL Server’ days, when contractors working on our Peoplesoft database were ignorant about this issue – so I learned about it FAST. It’s not a perfect/complete solution – someone really needs to make the decision what kind of recovery strategy is NEEDED for a particular database, then set the recovery model for that database specifically – but it tends to prevent some problems in the short term.
FWIW, I also use maintenance plans that run against ‘all non-system’ databases that will pick up new databases by default (and make sure they get at least some backup on a regular schedule), but that’s also a broad solution for a problem that needs more fine-grained attention in the long run.
So many hats to wear, so little time.
Great stuff, Kimberly. I can’t tell you how much I enjoy your insight – on the web and at the conferences.
I have a database set in Full Recovery mode. Due to my newbie stupidity, the log file grew out of control due to lack of t-log backups. So I followed your advice and in order to offer point in time recoveries for the database I setup the following backup scheme:
1) Full backups on Fridays
2) Differentials on Monday through Thursday
3) T-log backup from 7 AM to 6 PM Monday through Friday
Transaction log size stabilized and remained a manageable size throughout the workweek. However, I came back on Monday and low and behold the t-log ballooned again to 43 GBs. Granted, I didn’t run the t-log backups on the weekend, but there should be close to or absolutely zero activity to this database on the weekends. Even when it is active, it shouldn’t see what I imagine to be 43 GB worth of activity as it isn’t yet a production database.
Do you have any guidance?
I am facing one problem is, our client is using VB application software is searching for some information.
he get the error is: Index searching on table ‘table_name’ (specify in the From Claus) does not exist.
Please help for the above error.
No idea – except to check that the table exists. This isn’t relevant to Kimberly’s post.
Just wanted to say thank you. 4 years later and what you shared is still valuable insight.