Confusing BI with traditional IT applications development – 25 years later

Apologies for the delay between posts.  The long weekend afforded me the time to get back at it.

When I began my career as BI practitioner over 25 years ago, I came to it trained in Decision Sciences academically and from a professional background as a reformed financial analyst and IT business analyst / project manager.  This made me well qualified to take on early BI projects as, in the early days, most BI work was financial in nature. What I discovered pretty quickly is that traditional IT methods and skill sets did not translate very well to BI projects.  In fact, I had to unlearn much of what I had picked up managing transaction process application development in order to succeed with BI.

Part of this was the direct result of the technology.  ROLAP might have used the same DBMS technology as I was accustomed to, but data modeling took on a whole new meaning with de-normalized read-only star schema.  MOLAP?  That was another animal altogether.  Beyond that, the goal of an enterprise data warehouse required an eye toward not only the current project, but laying the groundwork for inclusion of additional subject areas regardless of how one went about that <insert Inman vs Kimball religious debate here>.

Far more important, however, was the need to ditch standard waterfall methodology in favor of iterative development cycles that would often start with a proof of concept, perhaps part of a vendor bake-off, followed by a prototype, followed by iterative builds of the application.  The good news was that the user interfaces came out of the box; some even had Excel sitting in front of them, so little development or customization was needed there.  This alone was a departure from what we were accustomed to.  All of the effort needed to be focused on the design and optimization of the underlying data structures, along with the extraction jobs from source applications that fed them.

This in turn created a need for an entire new kind of business analysis discipline that defined flexible environments to describe data navigation and investigation rather than pre-defined use cases of deterministic user interactions and transactions. Environments are defined by events, attributes and dimensions instead of business rules that characterize traditional requirements for transaction processing and simple reporting systems.  I elaborate on this in my previous post on Agile BI.

Project managers, for their part had to practice agility long before Agile became fashionable.  They learned quickly that their customers would not tolerate lengthy expensive data warehouse development cycles.  They demanded the rapid value the technology vendors promised.  This mandated breaking big projects up into smaller deliverables and iterative development that allowed users to experience capability they had never seen before and discover opportunities to create additional value even as they refined their original requirements.  Scope creep had to be assumed and managed, along with expectations.  Entirely new processes around data governance and end-user computing management needed to be developed.

BI developers came to understand that they needed to develop a certain tolerance for ambiguity in requirements and need for flexibility in their products, at times even predicting what the next set of requirements might be given their knowledge and experience with the underlying data.  This was a huge advantage on any BI project.

QA folks, for their part also needed to rethink their craft.  For one thing, it became necessary to work very closely with the BAs (if the BAs did not assume this responsibility altogether.) Assuring data quality is a very different task from testing transactions against a set of business rules. It puts an emphasis on automated comparisons with source data and working with users who have tribal experiential knowledge of the data as part of the user acceptance testing process.

So why bring all of this up now if it was all learned a generation ago? For one thing I have come to understand that there are still application development shops out there run by folks without direct BI experience that are just starting to take on BI responsibilities from their users. For another, recent technologies such as Hadoop have created a need to rethink aspects of the BI development discipline and the Agile movement has given the false impression that it translates literally to BI projects without modification.

I will comment in my next post on what characteristics to look for when staffing BI project teams in my next post. Until then, all comments welcome.