How do you know if a document really works? Writing and designing is only part of the process for producing effective documents. It is dangerous to rely on impressions about how effective a document is. To be sure, you need to test the document to measure its performance and to find out if the document is being used as you intended.
Following the "golden rules" is no guarantee
People who regularly write and design documents realise that there is no formula. What works for one document may not work for another. Because every document has a different purpose and audience, the language and design must vary accordingly.
Testing will show whether you have successfully produced a document that fulfils its intended purpose and that communicates the message intended by the writer.
What are you missing when you don’t test
Many people don’t test because they think that it is too complicated, expensive and time-consuming. But leaving out the testing component is false economy.
If the document is complex you will probably have a team of writers, editors, designers and policy experts working over many months to come up with a functional document. After employing these resources you will want to make sure that the new document meets your expectations.
There are costly consequences if you are not sure that the document really works, or if you realise that the document is flawed after it has been printed. The incentives to test your document are that testing will show:
When to test your documents
Testing should occur during the entire writing project. This means testing before you write, while you are writing, and when you have finished writing and designing.
Test before you write – Once you have decided to produce a new document, you should test to find out the problems in the old document. You can set criteria and performance goals for the new document based on this testing. Identifying problems in the existing document will give you a clearer picture of how you should approach the new document.
Test during the writing process – To make sure you are on the right track, test drafts with small groups of people to make sure that:
Test at the end of the writing process – When you are satisfied with the results of preliminary testing and you have addressed any problems revealed during this process then run final testing with potential users.
But what if you have limited time and administrative constraints?
The first two stages of the testing process do not have to be expensive or large scale. You can use colleagues or small groups of potential users to discover problems early in the writing and designing process. However, if you are using people from within your organisation, make sure that they have little knowledge of the area – for example, new employees.
Final useability testing should be larger scale and made up of people who will actually use the document. You can recruit potential users yourself or ask an employment agency to recruit people for you. Because final testing is more expensive than preliminary testing, it is in your interest to know about potential problems before the final document is tested to avoid having to repeat the final testing.
How to test your document
Document tests don’t have to be mammoth tasks. But you should be aware of some general principles:
There are two types of measurement you will want to apply to your document, whether it is current or new. Dumas & Redish call these "performance measures" and "subjective measures". Performance measures count the number of times certain types of behaviour happen. This is a good way to locate and quantify errors. Subjective measures record how people react to the document and what they think of it. The results of these measures can be compared to suggest why errors are occurring.
These methods give you an opportunity to observe people’s reactions to the document.
Use a questionnaire that refers to "real life" scenarios. Record the number of questions that were answered incorrectly, what types of errors were made by users and the degree to which these answers were incorrect.
Watch for reactions that tell you that the user in confused or frustrated. Count the number of times the person flicks through pages to find information and how many times they refer to the table of contents. Also, record places in the document where people pause or miss sections. Count each action by the user, but avoid making assumptions as to why the action occurred.
One way of subjectively measuring the way users react to a document is to ask them to "think out loud". This requires users to verbalise their queries, their understanding of the document, their impressions and also their path through the document. To be effective, you will probably have to prompt each person when there is a pause to maintain a steady stream of comments.
The second way to measure a document based on subjective analysis is to interview the person after he or she has finished using the document to answer the questionnaire. This is a opportunity to ask for impressions, opinions and possible reasons why they found the document difficult or easy to use.
The need to review documents after they are in use
Writers and designers should be aware that the process of producing a reader friendly and efficient document does not necessarily mean that the document will be perfect first time. You will meed to edit the document several times after each testing session. You will also need to review the document regularly after it is implemented to see that it still meets your organisation’s needs, because your organisation’s procedures or policy may change. You can address these issues when you review the document.
For more information on the topic see: Dumas & Redish 1993, A Practical Guide to Useability Testing, Ablex Publishing Corporation, Norwood, New Jersey, pp 185, 187.