What does DevOps mean for BI?

Tom Breur

14 September 2016

DevOps has become a big deal for application developers. It started off as a little ripple, but has grown into a big wave. Since my perspective has always been that application developers “led” the Agile movement, too, I have been closely following these trends, trying to cherry-pick what might work to improve BI practices. Mainstream developers embrace DevOps because they are motivated to improve the quality and speed of applications. Well, that sounds to me like a lofty goal for BI teams, too. Another big driver of DevOps has been to minimize the friction between development and production, a handoff that traditionally has been fraught with trouble. Well, BI teams often struggle with this same issue, so that seems pretty relevant, too.

I see two prime drivers for embracing DevOps if you are currently working in BI:

1. “Improving the quality and speed of applications deployment and operations” sounds like a no-brainer to me. What’s not to like about that?!?
2. When the rest of the organization is embracing DevOps, and you know this has implications for how we work and interact with IT, it seems at least prudent to stay apprised of their views.

DevOps unites the often separate functions of development and operations. The rift between developers and people who have to maintain their solutions once they have been promoted to production status is well known. We’ve all seen it. Breaking down the barriers between these two enables seamless delivery, collaboration that drives down defects and costs, ensures that not only speed to market, but also stability are on developers’ radar. TDD and continuous integration come to mind, but also tight integration between developers and testers, and awareness that a “development job” is not finished once you move to production, even if the product changes hands after that.

In Business Intelligence we often see a similar rift between backroom and frontroom developers. The former propagate data from source systems to some central repository. For convenience, let’s call that a data warehouse. Frontroom developers take ownership of deploying and disseminating these data, and putting it in users’ hands in the most convenient and useful format. The handoff looks and feels similar, and in the best of teams it mirrors continuous integration and TDD. In dysfunctional teams, you see the same problems that DevOps tries to address. Even though these are not elements of DevOps per se, these behaviors tend to be fostered in a DevOps environment.

Once I started to “see” all these parallels, I became aware of what DevOps has to offer to “us” BI teams. Friction-free delivery of data, early discovery of data quality issues, and increasing reliance on virtualization. Sound familiar? Building of automated testing suites. Haven’t we been talking about that, too? By looking at the end-to-end value chain, and detecting problems earlier, team effectiveness grows. Building the right solutions, and building them in the right way. This shouldn’t be too much of hard sell, now should it?