Our test framework is twofold. Our product is Java/Spring-based, and allows us to build “plugins” (in Java or Groovy) that can be called from a script through http posts and gets. The nice thing is that I am good with Java but our other people aren’t. If the underlying code changes, I can change the undrlying plugin code but keep the http interface the same, and our test scripts (written in Ruby and connecting to the server through an https connection) don’t have to be modified…not too much, anyway.

It’s worked out pretty well for us.

Jim

Do programmers have any trouble working within the architectural framework? Does the framework make test automation more difficult, or easier?

I work for a large software company on a web-based enterprise systems management product. We have plugins that are developed by different teams all over the country, and they all have independent release schedules. We’re also supported on several different versions of Windows and different DB servers. We also support clustered server environments. So, it’s a fairly complex environment.

We have what I’d call a “sort of agile” system. We have six week iterations, have a big push to automate as much as possible (all new QA people hired now need to have automation skills), we are story-based and the dev team is pushing hard for unit tests. However, we also are using a heavyweight test management system (HP QualityCenter), we have to do quite a bit of project documentation (MRD, QA Plan, Dev Plan, etc) – this is to fullfill senior-management level decisionmaking process. So, it’s definitely not a perfect process…probably better than some, worse than others.

One thing I do think…is that it has helped our team (I am the automation lead) to think in terms of automating testing instead of just automating tests. So, sometimes our automation resources have been dedicated to automating things like the restoring of test images and the like, because that was causing the most pain.

As far as architecture, I just think our application has too many moving parts for it to be practical to evolve our foundation organically. There is a standard, an API, that we publish and pretty much stick to once we reach Beta (the point products don’t start developing on our product until we are at least in Beta). I think with the type of software we’re building, there have to be some compromises (though at this point I recognize we’re probably making too many).

Jim

When working on an architecture team, I noticed that my focus was almost always on irrelevant problems and that I had a poor estimation of the work I imposed on the teams. In hindsight, my work resulted in little value.

After having had an architecture team, the first project gave system architects the final say insofar they did not create work for people outside the team. The system architects and senior devs got together every week to exchange notes. If someone wanted to influence the organization as a whole, this forum served as an advisory body for guidelines written by the system architects who wanted the change.

Very quickly, in the failed implementation, Architecture was almost comically proud of their ivory tower status. They created Enterprise Architect diagrams for system interfaces that were either already implemented, currently being developed, or nowhere near the next 2 or 3 sprints. Their work never appeared on the Product Backlog, and the PO had no idea why they inhabited the same floor space. The obvious collisions occurred as you might imagine. Interfaces were implemented by devs in complete disregard of an Architecture team that was disengaged from the current product development rhythms, and as a result, Arch spent most of their time trying to force Dev to refactor working stories to ‘fit their models’. Ridiculous? They were hired by the VP, so that gives you some insight into why it was tolerated, and ultimately failed. I almost forgot, they also didn’t code.

Ok, now onto the good stuff. At Logitech we have decided to retain the Arch group because of the complexity of the product, and to drive consistency across 5-8 scrum teams. However, there are very clear guidelines that we impose on them to keep it as ‘Agile’ as possible, and not break too many rules.

1. Archs code too: The only way an Arch will live in harmony with a bunch of crack senior devs, is to possess the same willingness and skill deep inside the codebase.

2. Architect ‘kind-of just in time’: For a complex framework that requires a significant amount of design, it’s probably too ambitious to rely on a bunch of XP Devs to emerge the architecture in a highly refactorable way that doesn’t incur too much waste. So, in other words, the stories ultimately trigger the need for architecture, but we Load Up that first story with as much of this design/prototyping as we can get away with, and still deliver something testable and usable for the user. Future stories in that functional area will always force us to revisit the architecture and the interface design, but at least we have meatier platform to work with. The counter argument here is that we might be inadvertantly creating waste because we don’t know how the arch will emerge throughout the project, but i don’t buy it, because there is always a bunch of ‘core and crud-like’ stuff that needs to get built as part of your overhead cost of creating a robust platform. A good example is error handling. The obvious waste in creating a quick and dirty solution for one story should be avoided, because you know that eventually you’ll have to create a centralized error handling mechanism, which of course will never be a user story.

3. Architects bind to stories: Rather than having them operate as their own scrum team, we have decided to lend them out to the scrum team that has picked up a story that requires arch input. So, in the previous case, an Arch would probably build a ‘starter’ error handling component to be delivered with the first story that required it, and with some quick knowledge transfer, devs could handle the rest. This doesn’t help us derive clean team sprint velocities, but in a Scrum of Scrums setup, we are also concerned with the team’s overall Product burndown. It’s the trade-off in our scenario, but the teams love having the archs work closely with them on stories that require that bridging effort across the platform. That’s what the Archs do, and it help the devs to concentrate on acceptance criteria, TDD, and the ultimate delivery of a story. The primary reason i don’t gravitate completely to XP is because complexity creates specialization. This is a dynamic that is as apparent in human social development as it is in product development. We can’t rely on a model of equally brilliant and capable Devs who can work on anything. Specialization, interspersed with as much knowledge sharing as we can afford, allows Archs to coexist with devs and to platoon from scrum team to scrum team, depending on the stories that require their input.

4. It’s ok to create a ‘Technical Story’: We have negotiated a 15% budget to work on ‘technical’ or Arch related stories. That means that our public velocity is 85% of our average story points capacity. This prevents a lot of discussion around proving that a technical story is required in order to complete what seems like a simple user story.

Once again, i know we have made some un-Agile compromises, but the alternative of a pure Agile approach was untenable. Like everything, we iterate and we retrospect, and through it all, we keep a close eye on the principles and values that keep us honest. The purist in me feels guilty, but i can live with that, because after all, it’s the user that ultimately determines success.

Nuno

The biggest problem from the testing perspective is that no web service testing tools, currently, will allow us to use our custom binding required for the framework. Thus, we’re forced to instantiate our own service clients in code in order to test services. Some teams are better prepared to deal with this than others. Solving this problem is one of the efforts I’m focused on now.

We don’t have a lot of testing-specific problems with the shared framework – in fact, this is one place where sharing does help us a lot, as we run hundreds of Selenium tests in a fairly large grid using a mechanism shared by both teams.

Our company is a lot smaller than yours though (we are under 50).

I know for fact that in that case that the framework and the game it self are treated as two different products, having there own cycles but that there is still a form of communication where the gaming side asks for new framework features.

Thanks