Data, the next frontier for true agility

Data challenges often inhibit the autonomy and speed of agile teams — key goals of establishing those teams. To unlock true agility, companies need to recalibrate their approach to data and analytics – embedding both at the core of agile teams. They will benefit from increased team satisfaction, better customer outcomes, and improved business responsiveness.

The changing fabric of an organisation

As digitalisation and globalisation brew up a perfect storm in many markets, companies find themselves in a challenging spot: having to increase their operating speed, while at the same time managing a more complex business environment and multi-modal operating models. Battling for growth, they look to equip themselves with capabilities that improve responsiveness to market changes. 

First, decentralisation is sweeping enterprises, as they push decision making to the front-lines with agile operating model and flat hierarchies. Adopting best practices from digital natives, such as Zappos’ holocracy or Spotify’s tribes and squads, these companies embrace new ways of working. 

Second, data and analytics have become a predominant capability in increasingly digitalised businesses. Digital natives are famous for intense focus on data — companies such as Amazon or Booking.com quantify almost everything through A/B testing. Just recently data scientist was called the sexiest job of the century, acknowledging the impact of data in the modern business.

Unfortunately, our experience proves that organisational inertia and legacy technologies put the two trends at odds, preventing companies from maximising value from their change efforts. To unlock true agility, companies also need to recalibrate their approach to data and analytics.

Data is inhibiting the potential of agile teams

Most companies’ agility journey is held back by their dated data management practices and limited analytics capabilities. This impedes the ability of autonomous and self-managing teams to fully exploit their potential.

Data challenges and inadequate analytics capabilities are prominent in companies embarking on the agile journey. For example, in one of the companies we observed, an agile development team that works with weekly releases had to wait more than 3 months to receive data feeds from another system, which was managed by another project. Other challenges that agile teams encounter include elaborately written, and mismatching requirement specifications from various parties like IT architecture, and operations teams. Teams that should closely collaborate on making “large systems work” often struggle to agree on data integration strategies, mismatching data refresh frequencies and unrealistic quality expectations. These often result in weeks-long debates on architectural decisions.

Similar challenges usually extend far beyond development teams. A product manager in a multinational industrial company shared her frustrations that while she received multiple weekly reports, none of them provided a complete and timely view of the customers’ experience with a company’s solution. And every time she wanted to run her own analysis, she had to submit data requests to IT that usually took days to fulfil. With insufficient data and analytics, product managers like her not only lose critical time but also struggle to maximise the value delivered. Data challenges also frequently lead to negative customer experiences, such as receiving multiple invoices or having to provide information multiple times when in contact with a customer service.

These challenges often inhibit the autonomy and speed of agile teams — key goals of establishing those teams. They also result in overwhelmed IT departments, complex IT architecture, emergence of shadow data repositories, and strained relationships between business and IT. If you cannot bring your data and analytics practices in line with an agile operating model, your agile teams lose momentum, while IT budgets get overstretched. 

Where data and agility clash

In no other domain the conflict between decentralisation and centralisation is so inherent as in the information management. 

One challenge is different views into data: while IT typically structures data according to domains (e.g customer, vendor, etc) or technologies (e.g Salesforce, Clarity, etc), agile teams are oriented towards a particular objective, usually based on customer outcomes. Furthermore, while agile teams have a single focus point, data needs to flow across value chains and between many teams.

This gap is further reinforced by a Single Source of Truth dogma, a long-held belief that companies need a single, uniform view into their whole business. Many companies spend years implementing data models for their products or customers, which should service every conceivable business need. However, as agile emphasises flexibility and responsiveness over rigorous management practices, agile teams find themselves constrained to dated and rigid data models — leading to emergence of shadow data repositories. 

Another challenge is data quality. Data quality has long been an IT pain point, often mitigated through data quality initiatives. In some companies, we have seen as many as 12 data quality dimensions, each measured on a 100% scale. This places strict development requirements that often rely on heavily manual processes— which contrast to agile teams’ needs to embrace Minimum Viable Product, rapid prototyping, and fast decision making.

And numerous other practices put data management and agile delivery practices at odds ranging from documentation requirements, to enterprise architectures, to different vocabularies, to physical separation, and to over specialised roles. As organisations try to navigate around these discrepancies without tackling them head-on, they often compromise both their agility and their analytics capabilities. They lose time, seed frustration, and encourage tribal us-versus-them mentality. 

Data, where the action is

This clash between data and agility is avertable — but it requires to sidestep traditional data management dogmas and embrace a new approach to leveraging analytics across the business. Companies that can bridge this schism are poised to accelerate the value of their agile teams.

To succeed, companies must make data an integral part of their agile operating model. One such approach is by plugging their agile teams into what we call the Smart Information Grid — a decentralised, networked, and empowered ecosystem that emphasises the role of frontline employees in data-driven decision making. Approaches such as the Smart Information Grid force companies to prioritise the information flow — how data moves across teams, over information stock — where data is stored in a company. 

This change has a twofold impact on agile teams. Analytics become interwoven into daily practice, including decision making based on proven hypothesis, performance measurement based on quantified customer outcomes, and process orchestration based on real-time insights. On the other hand, it requires agile teams to be attuned to data management practice — including an ability to analyse data, manage data provisioning, and establish mutually valuable data partnerships. This approach fosters decentralized decision making, pragmatic experimentation, and a quality first mentality.

Here, companies can learn from e-pioneers. At Amazon.com, data is the core fabric of agile end-to-end teams. Every decision needs to be proved with data — as Amazon found out, 2/3 of good ideas are not valuable once quantified with A/B testing. Additional funding is unlocked only once customer outcomes have been proven to be valuable. However, teams at Amazon not only leverage analytics but are also responsible for data. Agile teams own their API design and implementation. They are required to develop solutions with outside-use in mind since the very beginning. Instead of the common data protectionism, it ensures that data is available for reuse across the company and its network.

Bridging the gap between agility and data management requires a significant change in company’s DNA, both culturally and operationally. This change starts with data-driven agile teams and extends into an organisational design, operating models, technology platforms, and new business propositions. Companies that succeed will equip their agile teams with deeper customer understanding and faster decision making – leading to visibly increased business responsiveness. In today’s business environment where speed and customer focus are critical success factors, companies can no longer afford to ignore their data and analytics hurdles.