From Agile Testing to Smart Testing in Business Support System IT Transformations
The imperative of testing in agile transformation
In the rapidly evolving telecommunications landscape, agility in Business Support Systems(BSS) transformation is not just an advantage; it's a necessity. The shift towards agile methodologies signifies a departure from traditional, waterfall models, promising faster delivery times, enhanced flexibility, and better alignment with customer needs. However, this shift brings its own set of challenges, especially when transformations are executed partially or without a full commitment to agile principles, leading to significant roadblocks for a timely software delivery and accurate requirement implementation. Challenges from such half-hearted approaches have been elaborately discussed in the recently published article ‘Five Recipes for Failure in a Telco Transformation’.
One of the most critical yet often underestimated aspects of agile BSS transformation is testing.
Frequently, unsuccessful BSS transformations manifest initial signs of trouble during the testing phase. Delays in release schedules frequently occur because the test organization struggles to complete its integration and user acceptance tests – often due to immature software coded under high time pressure. Often the final software product does not meet the intended product owner requirements, which is why the software cannot be released as planned. Even worse, buggy software is commercially launched, causing increasing service cost and eventually churn.
However, these issues typically represent mere symptoms. The testing approach frequently acts as the final link in a complex software delivery chain, where underlying problems ultimately come to a head.
From our experience there are five common pitfalls that undermine testing effectiveness:
1. Late integration of test resources: Testing is frequently an isolated function that occurs after the software development phase, creating a disconnect where supposedly final software is inadequately tested.
2. Incomplete testing scope: Testing efforts often focus on new features without considering the integration with existing systems. Given that BSS transformations usually involve parallel architectures, this oversight can lead to critical failures.
3. Not living to the full potential of test automation: Many organizations fail to leverage test automation fully, relying heavily on manual testing even for system integration tests. This approach severely limits the number of test cases that can be executed within a given timeframe.
4. Insufficient development resources for bug fixing: During the test phase, especially in a stage-gate development approach, there's often a lack of reserved capacity for bug fixing as developers may already be working on new software branches.
5. Over-reliance on agile testing methods: Many product teams have internalized agile testing methods, including automated testing, within the CI/CD pipeline and sometimes blindly trust these results. However, not all test cases can be automated and therefore more complex end-to-end (E2E) test scenarios are insufficiently covered, which is often only noticed in production.
These challenges emphasize the need for a comprehensive, agile-aligned testing strategy and the importance of early, continuous testing within the software development lifecycle. Although agile testing offers a strong framework, relying solely on it during transformations is insufficient. To address these challenges effectively, agile testing must be supplemented with a smart approach that considers the unique complexities and demands of BSS transformations.
Defining the three elements to right-size testing for BSS transformation
A crucial part of a successful BSS transformation is adopting a holistic approach to testing. Traditional agile methodologies, while effective, often focus on team-level implementation and can miss the broader, overarching perspective. This is where the smart approach to testing comes into play. Smart testing goes beyond merely applying agile principles; it encompasses a strategic, overarching view that aligns testing with larger transformation goals. The complexities and the long-term nature of BSS projects demand a well-organized, agile-aligned test strategy that addresses not only the technical aspects, but also the organizational and procedural nuances. This comprehensive view ensures that testing is a facilitator of progress, enabling faster iterations, better quality outcomes, and more robust systems capable of meeting modern telecommunications demands. To achieve this, three pivotal elements are needed to form the backbone of any agile transformation: Test Organization, Test Process, and Test-IT Infrastructure. Each of these elements plays a crucial role in crafting a dynamic and responsive testing environment that is aligned with agile principles of flexibility and continuous improvement.
Test Organization: The foundation of an effective testing strategy in an agile environment is well-organized testing teams. Test designers should work within cross-functional agile teams from the requirement management phase to ensure test cases are aligned with product goals (shift left of test resources). However, this integration should avoid the pitfall of designing overly complex test cases early on, maintaining a balance between thoroughness and agility.
Test Process: Agile methodologies advocate for early and continuous testing within the development cycle. Agile testing remains core to agile BSS transformation. Due to the complexity of large BSS transformations, agile testing must be enriched with a phase to test complex system integrations from an overall system and E2E customer journey perspective. Incorporating a Go or No-Go decision point before commercial launch helps balance compliance with testing criteria and practical ‘fix later’ decisions, ensuring both quality and timely delivery.
Test IT Infrastructure: The complexity of Business Support System (BSS) transformations necessitates a robust IT infrastructure to effectively manage and execute tests. Integrating automated testing within the CI (continuous integration)/CD (continuous delivery) pipeline and utilizing Infrastructure-as-Code (IaC) and GitOps can significantly enhance the scalability of testing environments. Equally critical is the design of the test environment architecture. Relying on a single test environment in a parallelized SDLC often leads to obstacles. To alleviate this, it is vital to implement multiple, scalable test environments capable of operating in parallel, ensuring that testing does not impede the development process. Adopting these practices facilitates rapid feedback and continuous improvement in software quality, which is key for maintaining the momentum.
Focusing on three fundamental elements enables telcos to skillfully tackle the challenges of BSS transformations, turning potential obstacles into opportunities for improvement and success.
Establishing an agile test organization for large IT transformations
By addressing the Test Organization, Test Process, and Test IT Infrastructure, telecommunications companies are better equipped to navigate BSS transformations. This strategic approach turns potential obstacles into opportunities for refinement and success. However, recognizing the critical elements is only the first step. Implementing a practical, effective agile test organization requires a structured, phased approach that not only addresses these elements but also aligns them with the company's strategic goals and operational realities. The following methodology offers a roadmap for telecommunications companies looking to elevate their testing capabilities, ensuring that testing contributes positively to the agility, quality, and success of BSS transformations.
Step 1: Holistic assessment: Initiating with a comprehensive assessment, our approach leverages an established maturity framework that incorporates data points from more than 20 test organizations. This tool is designed to evaluate the current state of the test organization across multiple dimensions, providing a detailed analysis of where an organization stands in comparison to industry best practices. The assessment typically spans one to two weeks and includes belief audits with senior executives, as well as two to three deep-dive workshops with operational teams. The complexity of the test environment, such as the number of parallel test environments and delivery units, guides the depth of the assessment. This step culminates in defining ambitious yet achievable goals for the testing organization and setting specific KPIs, such as the throughput of test cases for end-to-end (E2E) testing. A heatmap is generated from the maturity assessment, reflecting both the current state and the ambition level, which then guides the team in identifying and addressing gaps effectively.
Step 2: Derive measures: Utilizing the insights gained from the holistic assessment, this step concludes the root causes of inefficiencies and develops specific actions to tackle these challenges. This step involves detailing the measures in terms of actions, expected outcomes, and the resources required. It's about translating the strategic vision into actionable items that will bridge the gap between the current state and the desired future state of the testing organization.
Step 3: Identify lighthouse project to pilot: Remodelling the test organization from head-to-toe during a running BSS transformation is not best practice. Therefore, to embed agile testing practices a pilot program is needed, ideally within a single team, to test the waters. This focused approach allows for the evaluation and refinement of strategies in a controlled setting, ensuring any adjustments needed are identified early on. The pilot serves as a testing ground, allowing for real-world application and feedback from all stakeholders involved.
Step 4: Define implementation roadmap: The ultimate step involves setting up an implementation roadmap by clustering the identified measures into feasible plateaus. Typically, three to four implementation plateaus should be defined, starting with a pilot. The total implementation horizon should be set to one to two years, aligning closely with the release plan of the BSS program. This roadmap serves as a strategic guide, ensuring that each phase of implementation is manageable, measurable, and aligned with overall business objectives.
Conclusion
Testing is often overlooked as a critical lever for stabilizing BSS transformations. The nature of these transformations, coupled with the integration challenges with business-as-usual (BAU) operations, makes agile testing complex. While agile testing is widely recognized and adopted in the telecommunications sector, merely implementing agile testing principles is not sufficient to ensure the success of BSS transformations. The challenge lies in what specific actions are taken and how these principles are applied and integrated into the larger transformation strategy.
A systematic approach that spans the test organization, test processes, and test IT infrastructure is critical. For senior executives in the telecommunications sector, the emphasis needs to be on defining and calibrating the smart testing approach elements (‘what’) and safeguarding the meticulous execution (‘how’). This includes ensuring early and continuous integration of testing resources, implementing a dedicated E2E integration testing phase with rigid gate processes, leveraging automation to its fullest potential, and maintaining robust test environments that support the dynamic nature of agile methodologies.
By focusing on these strategic elements and their effective execution, companies can move beyond the adoption of agile testing methods within product teams and achieve deeper, more effective integration that addresses both technical and organizational challenges. This approach not only enhances the quality and reliability of BSS transformations, but also ensures resilience and agility in responding to the shifting demands of the telecommunications landscape.
