Successful Web Development Methodologies

Commercial Web development has been around for more than 10 years. As an industry, this one's still fairly young when you consider others that have been around for centuries. But relative youth as an industry is no excuse for not doing better.

Consider the number of sites that are rebuilt for clients every day, and you'll likely agree that there's still much poor quality work being done, which affects us all: it means that clients are more wary and less trustful of Web developers. Anything that tarnishes our industry can tarnish all of us individually.

Having tried, trusted and standardised approach to Web development would go a long way to helping avoid the mistakes we all see over and over again. We need a Web methodology. However, finding a methodology that seems suited to Web development is not easy; making it work in the real world is even harder.

As the Development Manager for a team of 20, in the heady dotcom days, this was exactly the dilemma I faced. This article explores the issues that arose from our lack of a decent methodology, and how we as a team tried to resolve them. The result was the successful adaptation of an existing methodology for Web development.

The Symptoms

A number of factors combined to force the Web development team to make a change in the way we did things.

First and foremost, projects were constantly going over-time. There wasn't a specific reason for this; each project seemed to have its own particular issues. In some cases, the client changed his or her mind. In others, our team interpreted the client's requirements differently from the client's own interpretation. And for others it was a simple underestimation of the work required to complete the project. Regardless of the reason, the end result was the same: projects that go over-time go over-budget unless the client is willing to pay more. And each late project also impacts new projects.

The more frequently this happened, the worse things got, until we had a situation in which almost every project became high risk and suffered scope creep, generated low staff morale, and was expected to have unrealistic deadlines.

The answer was pretty simple: we needed a better way of doing things -- a proven way of delivering projects that met the client's needs on time, and on budget. The solution would also need to fit in with our organisational culture.

We needed a Web methodology.

Adopt, Adapt or Build Your Own

Once the decision had been made to find a better way of doing things, we realised we had three paths to choose from:

• adopt an existing methodology
• adapt from an existing methodology
• build our own methodology

The development team was divided on this issue. Some members believed we should make it up ourselves; others said we should avoid re-inventing the wheel. It was clear we had to do some research to work out which was the best path for us.

At the time, there were no recognised Web methodologies (although, from recent research it appears that the situation hasn't changed much). So, for adopting or adapting, we had little choice but to look at the existing software development methodologies.

Evaluation Criteria

When we started looking for methodologies, we decided that it was important to know what we were looking for. The first step was to decide on the criteria by which we would evaluate them.

Complexity

The solution had to be more than just a simple guide, but at the other end of the scale, if it was too big, there was no way it would work. We needed something that was easy to understand on the surface -- as both staff and clients had to be able to grasp it -- but had sufficient depth to give developers the guidance they needed.

Size

It's hard to get people to read documentation at the best of times, so a thick document explaining the methodology was unlikely to be effective. Chances were that, if we had a 100-page guide with a 10-page summary at the end, most people would use only the summary.

Cost

Anything that cost money would have to be justified; the less money that was required, the better.

Risk

We couldn't afford to get the methodology wrong. It was extremely unlikely that I'd be able to convince people to go through the process of trying a second methodology if the first one didn't work.

Pragmatic

The solution had actually to work, not be based on theory. There had to be real-world examples of projects to which it had been applied successfully, more than once!

Methodologies Evaluated

Rational Unified Process

We had two presentations from RUP representatives. The first was a one-hour session, the second a more detailed two-hour presentation. Both times, I was more confused after the presentation than before. I also spoke to a number of friends that had worked with RUP and had positive things to say about the process.

The scope of RUP is extremely broad. It covers pretty much everything in the entire SDLC, which is both its strength and its weakness. I was recently at a conference and attended a session by Phillipe Krutchen, one of the leading authorities on RUP. His view was that the main problem that arose when people tried using RUP was that they tried to use all of it. This was the same advice that I'd heard from friends who had used RUP. It's large and sophisticated, and the key is to use only those aspects of the process that you need for your project. This makes a lot of sense given that projects often vary and the same approach won't work in every case.

However, given the context of our team, RUP presented a number of issues:

• RUP was large, complex and sophisticated, yet our team was not! We had some team members who were still learning about the SDLC, and to try to introduce something as complex as RUP would require significant additional training.
• Even though RUP was comprehensive, we would have to review it in depth before we could decide which elements we would apply to our projects. We'd then need to trial and refine the process over time.
• The cost associated with the tools that were required to use RUP (eg. rational rose, requisite pro…etc) was high.

Overall, to implement RUP, or a scaled down version of RUP, appeared to be a potentially complex, difficult, time-consuming and expensive process that had a high risk of failure.

Process Mentor

We also received a presentation from a Process Mentor representative. The process itself was more compact than RUP, and therefore easier to get one's head around. In essence it was a Website with a series of steps, forms and templates that could be used to run a project. We felt more comfortable with Process Mentor than RUP because it was less overwhelming, but it still wasn't quite right. There wasn't anything that stood out as a major issue, as had happened with RUP, but regardless, Process Mentor did not feel like the right approach.

In-House Methodologies

Our team included a number of experienced developers who had worked with "home grown" processes from previous jobs with heavyweights such as IBM GSA. We had each of these developers talk about their experiences, explaining what worked, what they liked, and what they'd use again. There were a number of techniques that sounded useful but, when all was said and done, the overall feeling was that we weren't going to be able to "borrow" one of these in-house methodologies from another organisation.

Why Traditional Methodologies didn't Fit

There was no shortage of vendors willing to tout their processes and associated tools yet, after many presentations, we didn't feel any the wiser. Nothing seemed to address our needs. The reasons varied but the underlying problem was that none of the methodologies took into consideration the way things worked in Web development.

In comparison to traditional software development, Web development time frames are often shorter, the experience levels of employees vary dramatically, clients often have a poor understanding of what's possible, the technology changes rapidly and everything comes down to a single user interface (the browser). That's not to say these elements don't exist in traditional software development; however, the limitations are much more pronounced in Web development.

Another issue with traditional methodologies is that they failed to take into consideration the "soft" aspects of software development. A recent study of the most important factors in successful software teams (from Cutter Journal March, 2005) ranked trust as the number one factor and technical expertise as last, out of a total of 17 factors. Nowhere in any of the presentations or literature for either RUP or Process Mentor was trust mentioned; nor were any of the other soft skills that apparently have a huge impact on success.

The short answer was that we couldn't either adopt or adapt a traditional approach. This left us with the unenviable task of trying to create our own.

Going Agile

We were about to start defining our own process when I came across a lightweight methodology that's now known as the "agile" movement. The methodology in this case was Feature Driven Development (FDD); some of the other popular agile methodologies are XP, Scrum, Crystal and DSDM. (The details of Agile Methodologies and FDD are beyond the scope of this article but if you're interested to read more, visit http://www.agile.org and http://www.featuredrivendevelopment.com)

It was pretty clear that FDD was far better suited to Web development than anything else we had seen, so we decided to investigate further to see if we could adopt or at least adapt it to Web development. It didn't take long for the team to agree to give FDD a go. However, we soon realised it wasn't a silver bullet.

An Overview of FDD

FDD is an agile development methodology created by Jeff Deluca to:

For details, review the FDD Overview Presentation .pdf document.

Briefly, FDD consists of 5 clearly defined processes that can be captured in 5 pages. The core of FDD is the concept of a feature which is a clearly defined Client-valued piece of functionality. The processes that make up FDD are structured around defining every element of a project as a feature, then designing and building each feature in an iterative manner.

The high level structure of FDD is captured in the following diagram.

Process 1: Develop an Overall Model

This is an initial project-wide activity with domain and development members under the guidance of an experienced object modeller in the role of Chief Architect.

Process 1 involves the project team creating an object model of the business domain -- a model that's more shape than content. The model is not fully defined with all attributes and methods, as this step is more about capturing correctly the shape of the business domain in an object model -- not capturing every detail.

This is a highly collaborative process in which everyone has to work together to create the overall model. It takes an iterative approach. First, the domain expert explains a part of the business domain. The project team is broken up into groups (preferably 3 per group) to model that part of the domain. The groups then present their models and consensus is reached on which to use. This process is then repeated for each part of the domain until everything has been covered. The end result is an overall model of the entire domain.

Process 2: Build a Features List

This is an initial project-wide activity to identify all the features required to support the project requirements.

Creating the feature list is the task of the Chief Programmers involved in the modelling process. The client and stakeholders don't need to be a part of this process, as they have made their contribution in Process 1. Now's the time to capture the project in a list of features. This does not require collaboration: getting a group of people involved at this stage would not be productive or constructive.

The key to this process lies in defining the project using the language of the business domain. This means that the client will be able to understand and value each feature, but it also enforces a common language across the project team and reduces the risk of miscommunication or assumptions.

Poor communication is the basis of most problems in software and Web development. The language we choose has a significant impact on how effectively we communicate. There are many techniques in FDD that help to provide meaningful communication. The most powerful of these is encapsulated in Process 2: defining the entire project in features using the language of the domain, i.e. using the client's language. This might sound simple and obvious, but it should not be underestimated. The focus that this step brings is incredible and affects the project in many ways.

In FDD, everything is described as a feature, which is, in turn, defined as:

<action> the <result> by/for/to an <object>

For example:

calculate the total for a sale
calculate the total sales for a cashier

A feature is also defined in terms of size (e.g. more than 2 hours' work but less than 2 weeks' work). If a feature takes more than 2 weeks' work, it should be broken into separate features.

Defining everything as a feature avoids the problems that occur whenever the client refers to a concept in one way, the programmer refers to it in another, and the Project Manager has to continually interpret the two. If the Project Manager doesn't get the interpretation exactly right, mistakes occur: the programmers think they're building one thing, the client expects another. Using the same language doesn't mean the problem disappears, but it reduces the risk of confusion considerably.

Process 3: Planning

Process 3 is an initial project-wide activity to produce the development plan.

This process extends the benefits provided by Process 2. It provides the Project Manager with a means of planning the development phase in a meaningful way for both the client and the programmers. It is completed in conjunction with the Development Manager and Chief Programmers, who look, in particular, at the order in which features will be built, balancing load within the team and providing strategies for delivering early results to keep the client happy.

Process 4: Design by Feature

Process 4 involves a per-feature activity to produce the feature design package. This process is broken down into three steps: walkthrough, design and inspection.

In the walkthrough, programmers familiarise themselves with what they're about to build before starting on a detailed design, which is inspected before they start the build. The inspection of the design allows defects to be found and removed before a single line of code is written for that feature.

It might seem like commonsense to design, and inspect that design, before building, but this step is often ignored. In many other industries, the idea of building something before it has been fully defined, designed and planned would be considered negligent, yet it happens all the time in Web development. The first reaction of many programmers, especially those in Web development, is to open their favourite editor and start coding. The amount of risk to which this approach exposes any project is enormous. However, the converse can also be problematic: attempting to design everything up-front can often result in "analysis paralysis".

Exactly how much design should occur up-front is a hotly debated topic amongst advocates of agile methodologies. The approach taken by FDD is clear in the difference between Process 1 and Process 4. As we have seen, Process 1 includes design early in the lifecycle, but it's not a detailed design. The detail is left to Process 4. Putting the detailed design in this later stage ensures that it is considered at the right time: before the code is written. It also breaks the design down into meaningful chunks, feature by feature. This means programmers don't feel like they're spending all their time designing and no time coding; immediately after the design has been completed and inspected, the programmer can start to code.

Process 5: Build by Feature

Process 5 involves a per-feature activity to produce a completed client-valued function (feature).

Process 5 is also broken down into three steps: code, code inspection, and promote to build. As with Process 4, the idea of collaboration and benefits inspections is enforced. What makes Process 5 unique is the final step, "promote to build".

For code to be "promoted to build" it must be finished. The key to this is the definition of "finished". A feature is not finished until there is nothing else to be done. The way a Project Manager can test if a feature is truly finished is simply by asking if the work is finished. If the programmer answers "yes", the Project Manager then asks, "There's nothing else to be done?". This question often elicits a different response. It's not that the programmer is being difficult or misleading, it's just that, given the chance, many programmers will continue to work on code, tweaking, optimising or trying to improve it ad infinitum. If there's time to do this, it's not a problem, but if there's a tight deadline, the Project Manager needs to focus programmers on getting the project completed. This process is a great way to ensure that focus.

The other benefit of this process is that helps the Project Manager see clearly how much of the project has been completed, and how productive each programmer is. According to Gerald Weinberg (Quality Software Management Vol.1), the productivity difference between programmers can be as much as 20 to 1. The issue is how to assess that productivity. Even though features might range in size from 2 hours to 2 weeks of work, it doesn't take long for the Project Manager to assess how much work each programmer is actually producing once the variances in feature size are taken into account.