Wednesday, 5 March 2014

Launching an Agile Release Train While Standing in a Waterfall

Some days I wonder if someone has put the EDW Release Train on a list of "must see" tourist attractions in Melbourne. We have a least two tour groups a month come and visit us to see how we have gone about scaling agile using the Scaled Agile Framework (SAFe). Some visitors are from within our company, others are from other IT shops in Melbourne, interstate or even occasionally from the US. Many of our local visitors are at the beginning of their agile journey and they always ask "Where should we start?". The answer, of course, is start where you are; which is exactly what we did.

Our implementation of SAFe occurred shortly following an organisational restructure in which I had been appointed to lead a newly formed organisation that was the amalgamation of three interwoven groups: two Program Management functions and a Solution Design & Build team that supplied people to the programs. These groups were also augmented with an outsourced offshore build and test capability. This newly consolidated EDW delivery organisation operated under multiple SDLCs - ranging from agile to waterfall, depending on which group was executing the delivery effort. For the sake of everyone's sanity, my first order of business was to settle on a single SDLC.


Having had some exposure to improvements in business outcomes enabled by more agile delivery approaches, I was keen to move all EDW delivery in this direction. As I have talked about in previous posts, I was concerned about our ability to coordinate across multiple teams and after reading +Dean Leffingwell's Scaling Software Agility, I decided to transform my multi-SDLC organisation into a single Agile Release Train. To do this, I needed the leadership of my new team to buy in to the concept.

The first problem we had to solve was how to organise. Our guidebook,  +Dean Leffingwell's Agile Software Requirements contained material on key roles and groups but didn't go so far as to suggest an organisational structure.  With support from our Agile Coach +Mark Richards, I ran several workshops with my extended leadership team talking about the Scaled Agile Framework and how we might organise ourselves into an Agile Release Train. (These sessions were later followed up through the formation of our first book club.)

We decided that the five most mature agile teams would become the EDW Agile Release Train and we would work to transition the other teams into the train over time.  We landed on a target organisational structure consisting of five groups:
  • Pipeline Services - Get the right stuff on the train
  • Development Services - Deliver the right stuff to the station on time
  • Deployment Services - Get the stuff off the train
  • Transition Services - Get teams ready to join the train
  • Enterprise Services - Help the train operate within a mostly waterfall enterprise.

Reaching consensus on an organisational structure proved more difficult than I had anticipated. The two major sticking points were the organisational alignment of the system team and the distinction between Pipeline and Enterprise Services. The debate regarding the system team oscillated between them being part of Development Services or part of Deployment Services, with a majority vote rather than consensus resulting in them starting life as part of Deployment Services. While this has occasionally caused us pain from an uptake perspective, overall I would say it has been a successful model. One of the main benefits has been the focus the system team gets from from not having to share a line manager with the six feature teams. With respect to Enterprise Services, it ended up a moot point with the Department PMO group offering to provide those functions as a service to us, agreeing that Pipeline Services belonged with the Release Train.

We deliberately chose service orientated names for each of the groups to reinforce the needs for teams to consider who their customers are and provide a service to them. Pipeline Services and Deployment Services would between them co-ordinate the activities associated with the Program layer of the Scaled Agile Framework Big Picture and Development Services would consist of the Design/Build/Test teams. Being brutally transparent, getting this concept of “service” through was a challenge.  The traditionalists among the team looked at the Program Level in SAFe, deduced “they run the show”, and proceeded to debate what fell into Deployment versus Pipeline Services to arrive at an obvious power center since both sat at the Program Level.  It’s “safe” to say we eventually reached a landing point that made the intent clear, although it took some time to make the intent live.  Given we were only implementing SAFe at the program level we also rolled some of the Portfolio level activities into the Pipeline Services function. The functions that did not currently fit into these groups made up Transition Services. The goal of this group was to transition from wagile to agile whilst managing the inflight waterfall projects to their conclusion. It took about six months for to transition the ‘wagilists’ and a year for the waterfall projects to play out.


With an organisational structure in place our next challenge lay in establishing our new ways of operating to become an agile program. We gave Pipeline Services the task of visualising the entire program - including the wagile and outsourced projects. We then reviewed the in flight program of work and the pipeline. If a project had not already been outsourced we considered it as candidates for  delivery by the Release Train. At the conclusion of this triaging process we determined the last few projects we would outsource and made a commitment to cease the practise of outsourcing deliverables going forward. 

In the beginning we had a couple of dicey moments with stakeholders requesting outsourced offshore  delivery because they thought it would be cheaper. In these scenarios I offered the customer two options: (1) We send the project offshore. The customer pays the “cheaper” rate. But we take no responsibility for the quality of the outcome or the timeliness of the delivery, or; (2) The project is delivered by the EDW Agile Release Train and I will guarantee a quality product within the timeframe and dollar range we quoted. Interestingly enough no one ever chose option 1.

The projects that were deemed as destined to be delivered by the Release Train were allocated to the pipeline team for inception work, inline with our interpretation of the analysis function in the Business Epic Kanban. The Pipeline team included a number of business analysts precisely for this purpose. This turned out to be one of our early mistakes. Not only did this approach impede our ability to let go of our old working patterns, we also found that when these projects were handed over to the feature teams they came complete with a design and a bunch of technical rather than business features. This resulted in us moving to a model where the feature teams reserve 10% of their capacity each iteration for inception work on new epics and find the features themselves. After all, as the manifesto says: “The best architectures, requirements, and designs emerge from self-organizing teams.”

Today the Pipeline Services team is headed up by a Program Portfolio Manager and consists of four Project Portfolio Managers and a couple of System Specialists.  For a more detailed understanding of how the Project Manager role evolved into that of a Project Portfolio Manager check out my earlier post: What Happens to Project Managers When You Implement SAFe?. I realise that on paper Pipeline Services could easily be confused with a traditional PMO. This is not our intent. Unlike a traditional PMO, this team’s mission is to provide services to the development teams and not to manage or govern them. Another of our early mistakes was allowing the Pipeline team to push epics to specific delivery teams of their choosing. Over time we have transitioned to a model where by the Pipeline team showcase their work (ready to discover epics) to their product owner (development services). The Feature teams can then pull the priority features into their rolling four iteration plan as required. We still don't have this practice as smooth as I would like it to be, with organisational politics often creating situations where work allocation begins to look a lot more like push than pull, however we are committed to continually refining the process.

Development Services is led by a blended Development Manager / Release Train Engineer and is the home of our six feature teams. For us the Release Train Engineer (RTE) role as articulated in SAFe has never really emerged. Instead the responsibilities have been split across the three services leads, which in +Mark's view makes me the RTE! In the case of the Development Services Manager, he is the facilitator of our cut down version of PSI planning and our adaptation of scrum of scrums (from the Cocktail Party in +Henrik Kniberg's Lean from the Trenches). He leads the train’s continuous improvement effort, drives the adoption of engineering practices and coordinates the train’s approach to communities of practice (based on the Chapters model from +Henrik Kniberg and +Anders Ivarsson's paper Scaling Agile @ Spotify). The Pipeline Services lead and her team look after the bulk of the other RTE responsibilities including facilitating prioritisation, escalation of impediments outside the control of the development teams, managing risk and formal epic status reporting (where required).

One of the more unusual and potentially controversial aspects of the Development Services structure is alignment of the Scrum Master role with line management accountability for their teams. While this  may not sit comfortably with many agilists, for us it has been a significant improvement to our prior state and provided better support to our people. While we have experimented with the size and shape of our feature teams over time, for now we have landed on eight being the right size - a Scrum Master, a Technical Lead, a Test Specialist and five T-shaped developers. It is worth noting that reducing team size was a key enabler for growing T-shaped developers, as the non-homogeneous nature of the work required by the feature teams meant that teams needed developers to work outside their specialisations in order to meet their delivery commitments.

The third component of the EDW Agile Release Train, Deployment Services is lead by our Release Manager. This team has three functions - release services, environments services and the system team. Release services co-ordinate all our production deployments, including hotfixes. On behalf of the EDW Release train they have primary relationship with the Enterprise Release & Test Management function and EDW production support operations. The Environment team provide DBA services to the feature teams, including the provision of development and test environments. The system team (known as the Three Amigos) are responsible for the build of our automated deployment and continuous integration capability. (For more on how this team has helped with improving the software engineering practices see my earlier post on applying business change management to software engineering.)


It has been almost two years since we launched the EDW Agile Release Train and I think it would be fair to say that we have been through an extraordinary amount of change. Today we still have the three service orientated groups and the transition of the less mature agile teams into this structure was completed well over a year ago. Whilst the names and purpose of the groups has stayed the same, in true agile fashion the specifics of their makeup and responsibilities have evolved over time. I think it would be fair to say that giving the groups service orientated names has not always resulted in service orientated behaviours. Breaking the traditional mindsets of all three groups has been a challenge and continues to be an area of focus for me in fostering our culture of servant leadership.

Whatever approach you choose to take in structuring your release train, keep Dean's words top of mind: "We must constantly be aware that it is our people who actually do all the value-added work." and consider how your organisation will support your people.