I work for a mid-sized software company with around 27 development teams, mostly Scrum but also a few doing Lean and Kanban. We have a large and diverse product of HR-related software, I think there are around 6 million LOC, on three different platforms, including an ASP product and a .NET product. I’m fairly new, I started at the beginning of June, so I’m still learning about how everything gets done. We release four times a year and the software has won many prestigious awards. I’ve been impressed with this large-scale implementation of agile development. Just the speed of the daily Scrum of Scrums is amazing.
Many of the teams do a great job of automating their tests, and a few teams have their own continuous integration (CI) that runs multiple times a day. Other teams have automated tests, but must wait for the company-wide build to run, which is usually only once per day and sometimes not that often. They don’t get feedback fast enough, and that slows them down – a lot.
I’ve been asking why some teams struggle to automate tests, and why some teams don’t have their own CI. I’ve also asked my own team why most of our functional tests are automated through the GUI, rather than behind the GUI. We use FitNesse, but mainly to drive GUI tests with SWAT.
The answer I get a lot is “It’s the framework”. We have three architecture teams, and there is a framework (maybe there is more than one, I don’t fully understand it yet) that apparently makes it hard to test at the page or object layer. In addition, some teams’ code has tentacles into legacy code written in Delphi whose tests are written in WinRunner. Also, the current framework appears to not be fully supported with automated unit tests, it was developed a few years back. Programmers new to the framework seem to have a hard time understanding how to work with it. The framework clearly causes a lot of frustration. A new framework is under development, but it won’t be available for some time.
In order to understand the existing framework and find out whether the new one will facilitate testing behind the GUI, I was fortunate to get to meet with the leader of the architecture side of the development group. He gave me an overview of the new framework’s current design (which may change), aimed at making our product easily configurable by customers, using web services. It looks like this design allows testing under the GUI level.
Additionally, I learned that it is possible to test under the GUI in the current framework, and I met someone on another team who has automated many FitNesse tests at the object level. I plan to get her to show me how these tests work, and see if what she did would work for our code.
I haven’t worked at a company this large since the early 90s, and I’ve never worked in a company, agile or otherwise, that had “architecture teams” and “frameworks”. I get it that our product needs to look and work consistently across the board, so it does make some sense to centralize the design of the UI and perhaps other parts of the system. However, if the architecture is done by one team, what does that mean for programmers on other teams? Is the fun part of their job taken away? Do they get a chance to think for themselves and improve the product?
It also seems to me that there needs to be a lot of communication amongst the entire development community of our company, that all programmers should have some input to the architecture process. I’ve found that communication between teams in general is a bit lacking. Each team is so focused on delivering its part of the product, they don’t always feel they have time to go talk to other teams. So, for example, perhaps nobody on my team has talked to the person who did a lot of FitNesse functional tests.
Does your company have an architecture team, and/or a framework that everyone works with? Please comment and share your experiences. As a tester, I’d like to know what I can do to help my team deliver the highest possible quality when we have to work with a framework designed by someone else.
10 comments on “Architecture teams, frameworks, and agile development”
I haven’t worked in that environment but I can tell you were to look. You need to find someone in the video game industry. They have developers building these frameworks and tools to be used by the game development team.
I know for fact that in that case that the framework and the game it self are treated as two different products, having there own cycles but that there is still a form of communication where the gaming side asks for new framework features.
Thanks
We have a framework shared by two teams. Opinions are divided – it’s nice to get code written by one team reused by the other, but having one team able to update the framework and break the other is bad for productivity and morale. On balance we would probably be better off with some common small libraries and much less shared code – or even better, a commodity framework like Django or Rails.
We don’t have a lot of testing-specific problems with the shared framework – in fact, this is one place where sharing does help us a lot, as we run hundreds of Selenium tests in a fairly large grid using a mechanism shared by both teams.
Our company is a lot smaller than yours though (we are under 50).
The internal development group I work for does indeed have architectural frameworks that all teams are encouraged (but not usually forced) to use. The framework in most common use is for security. It’s used to secure WCF services which are gaining traction. While robust, it is somewhat difficult to get the security implemented because of some rather arcane configuration options. This causes a lot of complaints, but as more teams are adopting it, the knowledge of how to properly implement it is spreading.
The biggest problem from the testing perspective is that no web service testing tools, currently, will allow us to use our custom binding required for the framework. Thus, we’re forced to instantiate our own service clients in code in order to test services. Some teams are better prepared to deal with this than others. Solving this problem is one of the efforts I’m focused on now.
I’ve had the benefit of working with Architects on both a failed and a successfull agile endeavour. It’s obvious that the integration of architecture within agile is a poorly understood and continually emerging relationship. I still don’t think i’ve witnessed the perfect integration, but in comparing the 2 cases i mentioned, one may find a nugget or 2 of wisdom. It’s worth noting that this exact topic at Agile 2008 in Toronto had only one session of attention directed towards it, although Ron Jefferies made several comments about it in some of his sessions.
Very quickly, in the failed implementation, Architecture was almost comically proud of their ivory tower status. They created Enterprise Architect diagrams for system interfaces that were either already implemented, currently being developed, or nowhere near the next 2 or 3 sprints. Their work never appeared on the Product Backlog, and the PO had no idea why they inhabited the same floor space. The obvious collisions occurred as you might imagine. Interfaces were implemented by devs in complete disregard of an Architecture team that was disengaged from the current product development rhythms, and as a result, Arch spent most of their time trying to force Dev to refactor working stories to ‘fit their models’. Ridiculous? They were hired by the VP, so that gives you some insight into why it was tolerated, and ultimately failed. I almost forgot, they also didn’t code.
Ok, now onto the good stuff. At Logitech we have decided to retain the Arch group because of the complexity of the product, and to drive consistency across 5-8 scrum teams. However, there are very clear guidelines that we impose on them to keep it as ‘Agile’ as possible, and not break too many rules.
1. Archs code too: The only way an Arch will live in harmony with a bunch of crack senior devs, is to possess the same willingness and skill deep inside the codebase.
2. Architect ‘kind-of just in time’: For a complex framework that requires a significant amount of design, it’s probably too ambitious to rely on a bunch of XP Devs to emerge the architecture in a highly refactorable way that doesn’t incur too much waste. So, in other words, the stories ultimately trigger the need for architecture, but we Load Up that first story with as much of this design/prototyping as we can get away with, and still deliver something testable and usable for the user. Future stories in that functional area will always force us to revisit the architecture and the interface design, but at least we have meatier platform to work with. The counter argument here is that we might be inadvertantly creating waste because we don’t know how the arch will emerge throughout the project, but i don’t buy it, because there is always a bunch of ‘core and crud-like’ stuff that needs to get built as part of your overhead cost of creating a robust platform. A good example is error handling. The obvious waste in creating a quick and dirty solution for one story should be avoided, because you know that eventually you’ll have to create a centralized error handling mechanism, which of course will never be a user story.
3. Architects bind to stories: Rather than having them operate as their own scrum team, we have decided to lend them out to the scrum team that has picked up a story that requires arch input. So, in the previous case, an Arch would probably build a ‘starter’ error handling component to be delivered with the first story that required it, and with some quick knowledge transfer, devs could handle the rest. This doesn’t help us derive clean team sprint velocities, but in a Scrum of Scrums setup, we are also concerned with the team’s overall Product burndown. It’s the trade-off in our scenario, but the teams love having the archs work closely with them on stories that require that bridging effort across the platform. That’s what the Archs do, and it help the devs to concentrate on acceptance criteria, TDD, and the ultimate delivery of a story. The primary reason i don’t gravitate completely to XP is because complexity creates specialization. This is a dynamic that is as apparent in human social development as it is in product development. We can’t rely on a model of equally brilliant and capable Devs who can work on anything. Specialization, interspersed with as much knowledge sharing as we can afford, allows Archs to coexist with devs and to platoon from scrum team to scrum team, depending on the stories that require their input.
4. It’s ok to create a ‘Technical Story’: We have negotiated a 15% budget to work on ‘technical’ or Arch related stories. That means that our public velocity is 85% of our average story points capacity. This prevents a lot of discussion around proving that a technical story is required in order to complete what seems like a simple user story.
Once again, i know we have made some un-Agile compromises, but the alternative of a pure Agile approach was untenable. Like everything, we iterate and we retrospect, and through it all, we keep a close eye on the principles and values that keep us honest. The purist in me feels guilty, but i can live with that, because after all, it’s the user that ultimately determines success.
Nuno
Nuno, this is so helpful, thank you! That sounds like a really good way to integrate an architecture team and help the developers with a useful framework.
I’ve been on two major projects with architecture teams. In neither one, the result of this was disastrous, but the structure was a way for the project to ensure that their most experienced people had as little effect as possible. It was wasteful, pure and simple.
When working on an architecture team, I noticed that my focus was almost always on irrelevant problems and that I had a poor estimation of the work I imposed on the teams. In hindsight, my work resulted in little value.
After having had an architecture team, the first project gave system architects the final say insofar they did not create work for people outside the team. The system architects and senior devs got together every week to exchange notes. If someone wanted to influence the organization as a whole, this forum served as an advisory body for guidelines written by the system architects who wanted the change.
Lisa,
I work for a large software company on a web-based enterprise systems management product. We have plugins that are developed by different teams all over the country, and they all have independent release schedules. We’re also supported on several different versions of Windows and different DB servers. We also support clustered server environments. So, it’s a fairly complex environment.
We have what I’d call a “sort of agile” system. We have six week iterations, have a big push to automate as much as possible (all new QA people hired now need to have automation skills), we are story-based and the dev team is pushing hard for unit tests. However, we also are using a heavyweight test management system (HP QualityCenter), we have to do quite a bit of project documentation (MRD, QA Plan, Dev Plan, etc) – this is to fullfill senior-management level decisionmaking process. So, it’s definitely not a perfect process…probably better than some, worse than others.
One thing I do think…is that it has helped our team (I am the automation lead) to think in terms of automating testing instead of just automating tests. So, sometimes our automation resources have been dedicated to automating things like the restoring of test images and the like, because that was causing the most pain.
As far as architecture, I just think our application has too many moving parts for it to be practical to evolve our foundation organically. There is a standard, an API, that we publish and pretty much stick to once we reach Beta (the point products don’t start developing on our product until we are at least in Beta). I think with the type of software we’re building, there have to be some compromises (though at this point I recognize we’re probably making too many).
Jim
It seems like a pragmatic approach for your situation. I like the idea that you address whatever automation-related issues cause the most pain, not just automating tests.
Do programmers have any trouble working within the architectural framework? Does the framework make test automation more difficult, or easier?
Johannes, do you have suggestions for how organizations can make sure the architects do relevant work?
I’d say the architectural framework makes automation easier. Since there is a published, stable API, I can use the API documentation (which is pretty thorough) to know what functionality to call and how to call it, and the amount of re-working – at least at the level of the foundation – is limited.
Our test framework is twofold. Our product is Java/Spring-based, and allows us to build “plugins” (in Java or Groovy) that can be called from a script through http posts and gets. The nice thing is that I am good with Java but our other people aren’t. If the underlying code changes, I can change the undrlying plugin code but keep the http interface the same, and our test scripts (written in Ruby and connecting to the server through an https connection) don’t have to be modified…not too much, anyway.
It’s worked out pretty well for us.
Jim