A two-phase, four-step methodology was used in this survey.

Phase one – technical compliance to standards

A combination of automated testing (step one) and manual verification (step two) was used against the e-government Web Guidelines 2.1, which are based on the international Web Content Accessibility Guidelines (WCAG) (4).

There were two prime criterion that were used to determined if a site was compliant:

• sites received a score for each checkpoint that was assessed by automatic or manual check. For sites to comply they firstly had to exceed a minimum score.
• sites that exceeded that minimum score also had to provide their key accountability documents in an accessible format to comply.

Phase two – check accessibility barriers and user testing

This comprised:

• a technical test of the remaining WCAG level 1 and key e-government and WCAG 2 and 3 guidelines (step three). Sites received a score for each checkpoint/question.
• a user test (step four). The 30 top scoring sites that were tested during phase two were assessed for their accessibility to specific groups of testers with a range of impairments (Deaf, low vision, blind, mobility, reading).

User testing

The functionality and practical accessibility of sites was assessed by user testing. Three teams of user testers were used, where each team had testers with the impairments required, and including experienced as well as new testers. User testers were asked to perform a variety of simple tasks to obtain the information from the site any user might reasonably want.

Each task was timed, and testers were asked to give each task a difficulty rating. As well, testers gave an overall rating for each site.

Testers provided many detailed comments, some of which have been included throughout this survey report.

The majority of user test questions were the same for each site. By using the same tests on each site, it is possible to get a more meaningful comparison between sites.

The common tasks were:

• Find the contact details of the organisation and copy the URL of the contact details page.
• Does the website have a purpose statement? Type no if not found and copy and paste in the URL if found.
• Does the website have an accessibility statement or access keys page? Type no if not found and copy and paste in the URL if found.
• Find and respond to the feedback form on the site.

The other task was specific to the site. It could either be related to finding a piece of information that the general public might want to find or to complete a task (e.g. make an application).

Differences to the 2005 survey

The fundamental approach and methodology was the same as used in the 2005 survey.

The main difference in this survey was in the number of sites tested (127 compared with 150) and the criteria used to advance sites to the second phase of testing.

Only the top 30 compliant sites were user-tested, whereas in the 2005 survey all sites that passed to phase two testing were user tested. In this survey, 68 sites were eligible for phase two testing.