Agile is dead: Analytic Applications Edition - continued

In part 1 of this post, I wrote about how the software development community has speculated that Agile methodologies may have become over-hyped and the implications of this for BI/Analytics practitioners.

It’s now official.  Agile methodology for data and analytics has indeed reached the peak of the hype cycle. How do I know this? Because the venerable strategy firm McKinsey has blessed Agile as a data ‘transformation’ enabler.  See this white paper. I encourage you to read it in its entirety, but I will highlight some of the major points here.

The main thesis of the piece is that data has become a key strategic asset for their large enterprise clients (!) and that the diversity, volume, and velocity of that data require a high level of agility to leverage it quickly enough to effectively support decision making and opportunity discovery.

The key challenges that these clients face are integrating the data silos that their IT architecture creates and drawing a direct connection between data management success and quantifiable business benefits. This, in turn makes it difficult to justify the significant investments required to manage diverse data at scale.

They point out that Agile methodology that has been adopted by IT management to make applications software development more responsive to business needs. They then posit that Agile can have a similar impact on the establishment of enterprise data management capabilities in the age of Big Data.

Wow! They are just coming to the realization that enterprise data management is a good thing and traditional IT practices in this sphere can be ponderous, excruciatingly slow, exceedingly expensive, and out of touch with their missions?

The analytics community has known this for decades. Typically, the fix has been for business areas to go ‘rogue’ and build out their own data capabilities to drive their analytics. This is an old issue but it is now getting new currency with McKinsey’s IT clients as technology in the form of cloud-based BI platforms, powerful self-service BI tools, and APIs that make ‘roll your own’ analytics practical and relatively cheap at scale.

Of course, this is always the case when technology creates new-found BI ‘democracy’. Governance suffers and the data silos reach their limits quickly when business operations require true cross-functional analysis with integrated data.

McKinsey's solution approach has 4 major aspects:

• Create and empower Agile cross functional teams (scrums)
• Update the technology infrastructure to support and integrate “Big” and legacy data assets
• Emphasize new forms of communication to demonstrate value and discover new opportunities
• Develop KPIs to measure success

What’s interesting to me about the suggested solution approach is that there is little news here for BI/Analytics professionals and only passing mention of Agile tenets like scrums, user stories, Kanban, etc. What they are really advocating is good old cross-functional engagement, starting with delivery of modest high priority value, constantly iterating, and doing a better job of demonstrating ongoing success.

Here is where I take issue with what they are saying:  It makes perfect sense from a strategic perspective, but can be very difficult to implement tactically. Agile works best when applied to a discreet software product with its own life cycle. Forming scrum teams to work in parallel sprints churning out stories and epics, and then ultimately disband, can be practical in this scenario.

Data management, however, is not a project, product, or even a platform. It must be an ongoing capability if it is to work. To McKinsey, this requires drafting business experts to join their highly talented and experienced IT counterparts and wall them all off in a data lab. This, however, cannot be a short term assignment. In fact, if they succeed in discovering new opportunities, these labs will create an ongoing need to remain in place. Even the largest organizations I have worked in cannot afford to take that talent from their native organizations and send them to a lab for long.

For blue-chip consulting firms, promoting this kind of transformation initiative makes for some very lucrative consulting opportunities (I know. I have worked some of them.) I believe what works better in practice is to take an entire line of business, and building (or rebuilding) it from the ground up to support not only a comprehensive data management capability, but a data driven culture where everyone has some direct responsibility in their job description for acquiring, processing, deploying and using data in their daily work. The success of that effort can be used to propagate the culture across the other businesses in the enterprise.

Perhaps instead of thinking in terms of minimum viable products, we should set a minimum viable business unit as the initial data transformation goal. From that, we can deliver, iterate, improve, and expand by example.

Agile is dead: Analytic Applications Edition

For those of you who are not involved in developing software applications, The Agile movement was, in essence, a revolt against the then common practice of taking on the entirety of large software projects all at once; adhering to a methodology that stressed a strict sequential progression of stages beginning with requirements gathering, followed by programming, testing and culminating in a ‘big bang’ implementation. This came to be known as ‘waterfall’ methodology. Among the many issues with it is the fact that requirements usually change over the course of these lengthy efforts and what ends up being delivered often no longer adequately addresses the needs of the businesses that commissioned them.

Those of us who develop and manage analytic applications came to this conclusion decades ago out of necessity. One reason is that these applications support the very unique business process that is decision making. Unlike more conventional business processes such as booking airline reservations, decision making as a process can take many paths as data is analyzed and new information is discovered. The idea that one can precisely define requirements in advance of implementation simply does not apply for all but the most structured decisions like automated algorithm-based credit decisions. Even those require frequent updates as outcomes drive new learning and improved algorithms. Successful analytic application developers learned to use prototyping and relatively short incremental development cycles to keep their products relevant and achieve customer satisfaction. In essence, we adopted agility long before the Agile development hype cycle began. I went into more detail on this in a previous post on the impact of Agile on Analytics.

Since that time, there has been both a technology and culture-driven boom in analytic applications development as businesses of all sizes and maturities adopt a more data-driven culture. At the same time, the tenets of Agile methodology have been zealously embraced by IT executives who bought into the hype as they sought to deliver better applications sooner and cheaper. In fact, Agile certification became a requirement to work in some shops on all projects. Collisions ensued as the realization set in that orthodox Agile methods were not developed with analytics in mind and often could not be applied to Business Intelligence and Analytic application development projects successfully.

BI/Analytic application project teams were put in a familiar and awkward position. They could either try to explain why a methodology designed for a different purpose does not apply; or create the illusion of compliance with Agile dogma, technology and terminology that added little or no value to their efforts. Meanwhile, vendors and consultants in the Analytics space were all too happy ride the wave, coining the term “Agile Analytics” in an attempt to reconcile Agile mandates with proven methods in the BI/Analytics discipline.\

It now appears that the software developer community at large is having qualms about Agile software ‘revolution’ and what it became. Even the original thought leaders of the Agile movement have reservations about what has come of it. There has even bit something of a developer revolt against it. Then there are the chronicles of the Agile hype cycle and some very thoughtful pieces around how to move forward from Agile.
These critiques of the Agile movement as it is currently practiced have several points in common:

• Agile has passed the peak of its hype cycle and benefits resumes and consultants more than software projects
• Agile, as it is currently practiced, has become more process than objective-driven. This is exactly one of the faults it was designed to cure
• Requirements definition (often in the form of vague ‘user stories’) has suffered to the point where it has degraded testing, compromised necessary documentation and caused a fall in overall quality of delivered products
• Adoption of Agile in Name Only (AINO) practices where a waterfall mentality persists, development performance metrics remain a goal unto themselves, scrums become meetings, sprints become epics before defined value is actually delivered, and development teams remain as disconnected from the business as ever
• Applications architecture suffers as semi-independent project teams ignore standards and governance to meet their time and cost constraints. This one in particularly is deadly in the long run

Aside from sounding very familiar, what does this mean to us in the Analytics community? It means we need to stress  that what works for more traditional business process applications often does not apply to the unique nature of decision support and analytics applications. We define capabilities, not user stories. We lay down a sustainable data platform before we attempt to build applications on it. We prove concepts and prototype before we make major investments, we govern our data relentlessly to preserve credibility, and we develop our environments with an eye towards how they will be maintained and enhanced. Most importantly, we must remain focused on the results we can achieve while they are still needed and not worry so much on how we achieve them.

BI Industry Research: Is There Magic in those Quadrants?

As a Business Intelligence and Analytics (BIA) consultant, I need to keep up with the latest developments in both analytics process and technology. This has become quite a bit easier in recent years as quite a bit of useful information and wisdom is made freely available over the Internet. Premium fee-based research, however, remains an important source as it is generally deeper and more detailed and tends to carry great weight with clients.

I’ve gotten to know several of the analysts who cover the BIA space, and I find them to be among the most hard-working folks around as they juggle their research, conference and consulting responsibilities. Their methods are rigorous and the surveys span a uniquely broad sample of industries and geography.

Given all that, I have become a bit skeptical as to their influence because the research is often misused and/or misinterpreted, resulting in poor purchases or misdirected implementation programs that I have seen firsthand.

Part of the problem is that some readers just look at the pictures. In the case of Gartner’s trademarked Magic Quadrants, they note the vendors in the Northeast corner, and automatically limit their purchase consideration set to the products fortunate enough to be there that year. This is a big mistake. The real value in this content comes from the detailed review of the relative strengths and weaknesses of each vendor’s offerings.

Others get too caught up in the feature function comparisons.  Quality and applicability; not quantity is what matters. Checking off the boxes may be great for sales demos, but many customers get too far over their skis and don’t end up using half the capabilities of the tools they buy.

Another mistake I see is when buyers try to use the research as a substitute for their own reference checks and a well –executed proof of concept to prove out the technology in context.

Keep in mind this research is intended for both buyers and sellers. The vendors themselves are important clients and participants. They provide insight into their product plans and access to their reference clients. They also end up providing much of the research funding. When vendors are reviewed positively, they often purchase the work for redistribution. See here for one example.

Some of the best insights in research pieces are around the overall market as opposed to the individual products and vendors. For example, in this year’s Magic Quadrant, Gartner took the very significant step of redefining their market domain for BIA technology. They completely removed a set of traditional data analysis and reporting tools that have been a major market presence for many years (e.g. Oracle). They limited the coverage to BIA “Platforms” that enable user-driven full-cycle data preparation, integration, visualization and discovery capability. This change makes a strong statement around the direction of the market and the shift of spending away from IT-driven initiatives to user-driven and funded programs where IT is expected to enable and emphasize data provisioning and governance over technology enablement.

The wise consumer of this type of research takes many factors into account when evaluating the products that are covered, including:

• Vertical solutions – does the vendor have a strong record creating solutions specific to your industry?
• Partnerships – does the vendor provide a complete solution or do they rely on partners?
• Costs and pricing – although there is often useful information around pricing models, every sales cycle is unique with regard to the effective costs of purchase and eventual ownership.  Negotiation skills, reference potential, sales incentives etc. all play a significant role.
• The relative importance of sales experience, documentation, training, and support to you as a customer
• Pay close attention to the methodology notes. They are usually very comprehensive important, particularly when they detail the breadth of the surveys and discussion of vendors who were included and excluded.

Careful consideration of these details allows the reader to match the technology for a solution to your specific situation, often preventing expensive mistakes.

My 2016 Business Intelligence & Analytics (BI&A) Wish List

It's that time of year where we bid goodbye to what we did not like about last year and look forward to our best hopes and wishes for this coming year. As I look back on the past year in BI&A, here is what I'd like to see in 2016:

1.       Some real standards the software vendors will respect

I'll admit I've wanted this for a long time, but I can still hope. We are now in the big-data driven third generation of BI&A technology. The first was relational databases. Then, the industry settled on SQL as the standard query language and that facilitated a whole industry with interoperable query tools, ETL tools, database platforms, and a generation of expert professionals. The second was multi-dimensional technology. In this case, we never even saw a standard for query languages aside from some weak attempts to extend SQL. Now 'No SQL' is the emerging non-standard. Metadata standards? Dream on. The one credible attempt at it, the Common Warehouse Metamodel in the 90's, never really caught despite support from several major vendors. Those same vendors eventually decided that proprietary solutions could become de-facto standards if they acquired enough market share through buying out smaller competitors. So metadata standardization is achievable, but mostly with single vendor solutions. This brings me to my second wish.

2.       The next consolidation wave

As is always the case during a technology consolidation, you get a wave of startups with new technology and established players trying to reposition older technology in a fight to win over the early adopters in the market and some love from the industry analysts. In time, the weaker players drop out of sight and the stronger ones get swallowed up by the big enterprise players looking to buy technology and market share. It will be a little different this time as SAAS deployments will allow more of the upstarts to thrive on their own. Some may end up pushing out older players in the process. For me though, there are just too many incomplete solutions out there right now and the consolidation wave cannot come fast enough.

3.     Fewer Big Data wannabees

Another consequence of this technology shift is the emergence of a huge number of resumes on the market promising a depth of knowledge and experience in big data technology stacks that is more hype than substance and becomes clear 10 minutes into an interview session. I’d rather work with seasoned BI&A professionals that have mastered the basics of software engineering, project management, requirements development, data governance, and testing who have demonstrated the ability to learn and adapt to new technology quickly.

4.        More agile BI&A teams and less Agile methodology zealotry

BI&A pros have known for decades that agility is mandatory in our work and waterfall methods do not work. Requirements are generally not completely known in advance and only revealed through prototyping and iterative development. Value should be delivered on an ongoing basis. On the other hand, some of the main tenets of the now-revered Agile movement do not work all that well in the BI&A space. We do not develop structured applications as much as we strive to create environments where our users can create their own applications. This limits the development of workable user stories in advance. The evolution of our platforms over time and need for stable development and support teams does not lend itself well to the scrum concept as it is typically advocated. Yes, we need to be agile (small A) but not necessarily Agile (capital A.)

Business Intelligence and Analytics has never been more recognized as vital to success in business, government science and education. Our tools and technology are better than ever. Those wishes have come true. Now, I wish for all who read this a happy and successful 2016.