CHOW #9– Long hardening cycles for Product releases

This is an example of an engineering  organization that is structured based on areas of specialization (such as the core product engine, user experience (UX), developer experience, system integration, common engineering services and so on). Any typical product release involves about 15 to 20 different teams belonging to different areas mentioned above and spread across four different global locations (US, Europe and India) coming together for the delivery. The releases typically involve 3 to 4 month cycles with 2-week iterations most of the time. While a significant majority of the teams are cross functional within the engineering areas mentioned above and co-located with their own Product Owner stationed with the team, there are some exceptions especially with the POs located in the US for some teams. In addition, there are a couple of teams focused on end to end integration and inter-operability testing working independent of the scrum teams mentioned above.

The organization has an excellent tool to capture and track status of their release goals, epics and user stories as well as the commitments in terms of sprints that the teams are working on. There is also some collaboration happening at an Engineering heads level as well as amongst Product Management heads on an on-going basis (this typically happens once a week) but there is no centralized release planning or tracking at an overall release level. There is an expectation from the Engineering Management and Product Management for teams to deliver committed features on the release date planned and to work with each other on dependencies to ensure this happens.

One of the things that this organization is facing involves long hardening cycles – a typical 4 month release features 6-8 weeks of hardening effort on the average with two major contributing factors: one is bug leakage from a current sprint into the upcoming sprints and secondly bugs reported by final integration testing which takes significant elapse time and effort to fix. There is also an additional issue of new features coming in towards the end of the release forcing teams to work over time to accommodate them.

The challenge for the organization is to bring down the hardening cycle dramatically – from 6 to 8 weeks to perhaps two or three weeks at the most. How would you deal with this challenge?

Suggested solution:

A recommended approach to deal with the Challenge

The key measure here is the duration of the hardening cycle and you need to know what the contributing factors / reasons are for such a long hardening cycle. Some possible reasons include:

1. What the Definition of Done (DoD) includes and how strictly it is implemented. Is completion of testing and fixing of all defects (at least high priority ones) mandatory for a story to be closed in a sprint? Who is authorized to close the story? Is there pressure from PO or your own management to close a story even if there are bugs unfixed, just to show a higher velocity to your higher management?

One information given is that there is leakage of defects from a sprint into future sprints. What kind of defects are these? Why were they not found in the sprint in which the story was supposed to be completed?

2. Integration testing discovering a lot of defects is an expensive process because bugs found in final integration after your sprints are complete take more time to debug, fix and retest causing an extended hardening cycle. Analysis of bugs from final integration will tell you whether such issues could be avoided with better planning and dependency management across teams.

What do you think?

Leave a Reply

What to read next