Better together – the importance of great inter-team relationships

Back at the end of January I kicked off a survey about the relationship between your DBA and dev teams. See here for the survey.

Here are the results:

The 22 Other values are:

• 5 x The devs are great – it's the apps and the process that suck.
• 4 x It depends on the development team.
• 3 x I'm the team lead and the DBA.
• 2 x It's fine with the internal developers, but the consultants… not so much.
• 2 x It's non-existent. Management does not have the DBA and dev team work together until there is an issue.
• 1 x Essentially non-existant, between all groups: Infrastructure, DEV, DBA. Management has no interest in fixing it. Total crap gets moved to prod all the time, DBA's are the bad guys cause I've stopped the practice of moving db side "things" into prod without testing.
• 1 x It is spread out across country, DEV team think they are gods, what's a DBA?
• 1 x The DBA/Dev relationship is good, it's the DBA/Vendor relationship that sucks because vendor apps suck!
• 1 x The dev team have no clue what Data Architecture is all about 1The dev team is learning. Getting better all the time.
• 1 x We are a small Admin team supporting scores of development teams. The teams are very segregated so relationships are very weak.

A very interesting variety of situations and views. Good to see that the majority of people are experiencing good relations between teams (answers 1 + 2 + 7 + 8 = 59%).

Here's a comment from the original post that sums up what I see a lot of the time when first working with clients:

The developers think the DBAs are a bunch of idle boneheads, and the DBAs think the developers are a bunch of cowboys with little regard for the availability of the live servers.

Strong words, which demonstrate a fundamental lack of understanding about what the various teams do. That is very often the root cause of all the animosity between teams – not understanding the goals and motivations of the other team. For instance, a dev team is often under a lot of pressure to get the next release finished and put into production. A DBA team needs to preserve the availability and performance characteristics of the production system. These two goals are often at odds as the dev team wants to throw code over the wall and get it deployed, whereas the DBA team wants the code to be tested to ensure it will not compromise the production system in any way.

There's also the case of those who answered #6 – one guy who's the problem between the teams. This could be someone in an architect or lead developer role, who has a lot of experience, but maybe some misconceptions, who has a domineering personality and is always right – no matter who is trying to show them that they are in fact incorrect. We've never come across this person during our consulting work, oh no. :-)

How do you work towards solving these problems?

Education.

It's no good just complaining and blaming each other when things don't go well. You need to have the other team understand *why* your team does what it does, *why* it's concerned about certain things, and *why* it operates under the pressures it does. I have a lot of experience managing development and program management teams and managing the relationships between the dev teams and the test teams, the dev teams and the build teams, the dev teams and the program management teams – so I'm speaking from experience here.

Last year in London we did a 1-day workshop where the focus was bridging the gap between development and production. We highlighted a bunch of problems we've seen over the years and how some education could have helped. Here are some of the takeaways people had from the day:

• Avoiding known design anti-patterns – don't repeat mistakes that others have made. Once you find a performance problem, give feedback on the design anti-pattern that caused it to everyone involved. A great example of this is thinking that single-column non-clustered indexes on every table column will be useful. There are some great posts and books about SQL anti-patterns, such as this one.
• Prototype and test design strategies – it's imperative that the developers have a test system that reproduces production workloads to the extent that proper scalability and load testing can be performed; otherwise it's not going to work in production. I see this over and over. It's also very important that developers and architects understand the ramifications of data type and schema design choices on the operational aspects of SQL Server (for instance, a LOB column in a table prevents online index operations on the clustered index).
• Workload analysis and performance base-lining – nothing allows you to identify that a new set of code has a problem faster than having a performance baseline, and monitoring so you can easily identify that the latest code change caused a performance change.
• Troubleshooting to identify problems rather than throwing hardware at the problem – it's always better to find the root cause of a problem rather than trying to work around it with bigger iron or faster disks. You'll eventually run into the problem again. A good example is writing a query that is non-SARGable and requires an index or table scan. I blogged about this recently here and many others have too. Implicit conversions are another great example – Jonathan has a great blog post about finding them here.
• Build trust and partnership between teams – success is not one-sided and both teams look bad when an application/project fails. A company with teams at odds will suffer.
• Consistent post-mortems and “lessons-learned” from problems and mistakes rather than laying blame – being able to say mea culpa and proposing changes will elevate you above your peers whether you're a dev or a DBA. Just remember that if it's someone else's fault, constructive criticism needs to be well-phrased to be well-received. Don't make it personal.

And the one we stressed the most was:

• Work to cross-train and educate both teams for best practices

This is the most powerful way to foster relationships between teams in my experience – offering to educate the other team on, say, one issue a week. This will show them that you're willing to put skin in the game (and hopefully they'll reciprocate) and in the long-term it will save you time. And it's a great way to show initiative and that you're a team player  and motivator to your management. Some ideas for you on this:

• After a post-mortem, put together a list of links to blog posts and articles that explain facets of the problem and how to identify and or avoid it
• Organize a weekly self-study group that involves both teams. Use online material like our MCM prep videos as a base. Management will love this, and people will almost feel compelled to be part of it, rather than showing management that they don't care about working better together. Even the nay-sayers will eventually see the benefit and come around. Persevere!
• Get the team leads together to voice concerns and clear the air in a pre-agreed no-offense-will-be-taken closed-door meeting. You may be surprised at what lingering misconceptions, grudges, and misunderstandings can be cleared up by letting people vent and clearing the air.
• Bring in an outside consultant who knows what they're talking about to be an independent voice of reason with no political baggage for either team. We get asked to do this every so often – it can be challenging but very effective, especially for the "there's this one guy" problems where no-one can convince them, or wants to publicly stand up to them. Often helping the person see the light and think they've made the leap themselves is the best way forward – which can be hard to do from within the team, but a lot easier as an outsider, where we don't care about a position in the company pecking-order.
• Send one dev and one DBA to a training class where they can both learn and bring back the learning to the teams with the two perspectives. Of all the classes we teach, the Internals and Performance class is the best for this.

To summarize, a bad working environment is toxic. You can fix it. It'll take some effort on your part but the pay-off is really worthwhile. And there are lots of resources out there to help you out.

What are you waiting for?