Working Practices: Bug Reporting

Being able to report bugs accurately and efficiently is an incredibly useful ability, but it is one that many people don't seem to either take seriously, or worry about.

Some developers seem to believe that it is only the testers that should worry about how to report bugs, or even to report bugs at all, but every person on the team should be on the look out for bugs at all stages.

Definition of a "Bug"

Before determining how best to report bugs, we need to understand what one is. A bug is a programming term, but the general meaning of it can be expanded to cover all areas of game development.

For game development purposes a bug is an unintended problem or side effect in a piece of work.

Often, the term "bug" is used to describe anything that the finder thinks is incorrect, regardless of whether this was intentional or not. I've had bug reports saying that something was "too hard", which isn't a bug - it's feedback (incidentally, another bug on the same report mentioned that the same thing was "too easy"). Although opinion based feedback is also valuable, it tends to be easier for most people to give.

Purpose of a Bug Report

A bug report both identifies a problem, and then aids the developer in tracking down and removing that problem. A bug report should be informative and accurate, and avoid ambiguity and assumption. It should provide a map of how to get to the problem, and what to expect when you get. Bug reports also contain other information, like who submitted the bug, who is supposed to be dealing with it, and it's status (not looked at, not done, being worked on, or fixed).

Bug reports provide two main areas of information - impact and definition

Bug Impact

Severity

How significant are the effects of the bug? Ranging from "fatal crash" to "minor glitch".

Bugs can vary in impact wildly, from minor glitches through to machine lock ups, or corruption of data. The severity of a bug is one of the factors that determines how important it is that the bug is fixed before release. Although in an ideal world all software projects should be shipped bug free, the reality is that this is incredibly hard to achieve, and most game go out with some problems, whether minor or severe.

Importance

How important is it to fix the bug before release? Ranging from "vital" to "could do".

Although severity may seem to be the biggest factor in whether to fix a bug, this is not always the case. Some severe bugs happen so rarely, or under such unusual circumstances, whilst others are minor, but will be seen by every player. Since a developer's time is limited, it is important that the time is spent as effectively as possible.

Urgency

How quickly do we need to fix the bug? Ranging from "Immediately" to "no hurry"

Urgency is purely a timescale thing, and effects when a bug should be tackled. If a bug is halting work in another area NOW, it might not have a very severe effect, but it needs to be tackled quickly in order to prevent delays. For example, minor glitches in the level editor can cause a lot of delay, even if the bug itself isn't that severe.

The three above pieces of information allow a developer to organise the bug list that they have. With these factors, they can determine the order in which to fix a bug, and which bugs must be fixed, and which ones aren't vital.

Bug Definition

Development Area

The development area is the first stage in determining who should fix the problem, and the type of bug that is being talked about. This will change as more information is available. For example, an artist may list the development area as Code, and then once a programmer has looked at the problem, may redefine it to Collision Detection.

How to Repeat

This is incredibly important. Giving the developer steps that can be taken to repeat the bug allows it to be identified very quickly. In addition, the first thing someone is going to try to do once they start to fix a bug is try to repeat it. If they can't for whatever reason, a great deal of time can be wasted.

Providing a method of repeating a bug also cements the source. If someone files a bug report the first time there is a problem, but doesn't try to repeat it, then the description of the bug may provide incorrect information. For example, if a bug report said "I walked through a doorway, and the game crashed", and a programmer walked through some doorways and nothing happened, what do they do next?

If a bug can't be repeated, or is intermittent, then this should be written in this section. Although I've stressed the importance of the "how to repeat" section, if you can't repeat a bug, report it anyway, just make sure you mention that you can't repeat it. Even that provides information

Description of Effect

This is a full textual description of the problem. This should be factual, and assumptions should be avoided.

The description should be distilled from the different times that the bug occurred. If you write a full description of the bug based on the first time you spotted it, the report might infer a different problem.

Obviously if the bug only occurred once, then the information has to be based upon that.

The complexity of information required varies wildly from bug to bug. Texture mapping inaccuracies don't need a great deal of description, but need very accurate information on how to find the problem, whilst bugs with game logic or level design might need a different emphasis.

There are programs that ensure that cater for all of this (Testtrack is one I have used and liked), and these are certainly worth looking at for any development team.

Things to Watch Out For

There are a number of common problems that occur in bug reporting, some of which even happen when all the information described earlier is entered.

Ambiguity

Bug reports must be accurate and as unambiguous as possible. Steer away from vague phrases, and try to be as descriptive as you can. As an example, the phrase "so and so doesn't work" is a constant bane to people trying to fix bugs. The less vague a description, the easier it is to track a bug.

Assumptions

Assumptions can be dangerous. When writing a bug report, it can be easy to make an assumption as to the problem. Everything works well if the assumption is proved correct, but if it is incorrect, the bug fixer can be thrown off track, or spend time trying to fix something that isn't broken. The person fixing a bug is likely to have a lot more knowledge in that area than the reporter, so keep assumptions out, and write the facts instead. If the assumption is logical based on the description, the bug fixer is likely to come to the same conclusion. If the reporter wants to put in an assumption, it should be very clear that it is an assumption, not something based on knowledge.

Personal Preferences

This problem usually occurs in the impact area, and involves the reporter colouring severity, importance or urgency in order to alter the order in which bugs are fixed. Although it can be tempting to set a bug to Urgent!. this has an impact on when other, potentially more important, bugs are fixed.

Inadequate Testing

Everyone on a development team is busy most of the time. Sometimes a bug can occur, and due to the lack of time, a report is immediately added that hasn't been tested properly to provide adequate information. This leads to the bug fixer being left with a wholly inadequate report which provides little to no helpful information.

This problem also exists from the fixer's side. Once the work has been done to fix a bug, it must be tested to ensure that the fix both works, and doesn't break anything else. Sometimes a testing team can be asked to check this, but not all companies have such a facility. In the absence of a testing team, the fixer needs to take some responsibility to ensure that the bug truly is fixed.