Approach to monitoring data collection and reporting

• Monitoring definition, objectives and scope (addressed in section 4)
• Developing a monitoring framework
• Stakeholder involvement and communication
• Collecting, analysing and reporting data
• Using monitoring data for management action and decision making.

The monitoring adheres to the DRPI method for UNCRPD monitoring, though only one of the three elements of the DRPI framework was implemented after the first year of monitoring (monitoring individual experiences through interviews with disabled people). The other elements (studying legislative frameworks and media analysis) are not included in the CCMG approach. 

Developing a monitoring framework

New Zealand has implemented a monitoring framework under Article 33. The 2010 Cabinet Paper which established the CCMG referred to a monitoring framework for the CCMG’s monitoring. The UNCRPD provides some structure for the monitoring and its reporting, as does the DRPI method.

There is an opportunity to strengthen the monitoring by developing a formal framework to guide the disabled people led component of the monitoring. The framework should include:

• Questions the monitoring will address: The questions the monitoring will seek to ask to monitor New Zealand’s progress against each of the rights included in the Convention
• Priorities: Establishes priority areas for monitoring to focus data collection and reporting which change over time. This does not equate to establishing a hierarchy of outcomes but acknowledges the breadth of the Convention.
• Indicators and measures: Describing what success and failure may look like for each monitoring question. Drawing on the indicators used for other pieces of work may be a useful starting point (such as the Outcomes Framework for the Disability Strategy). Some indicators and measures may be aspirational.
• Data sources: Which data sources will inform reporting on each of the questions and how the data will be collected.

A formal monitoring framework would provide a structure for further adapting the DRPI data collection approach to New Zealand’s needs, prioritising topics, communicating the monitoring approach to stakeholders and determining report topics.

Setting the monitoring agenda should be done in a genuinely targeted way that should clearly involve the DPOs around priorities for measurement. Once that agenda is agreed, the DPOs should stand back and wait for the results. (stakeholder)

Developing a monitoring framework requires specialist skills. Government agencies invest significant resources in developing and maintaining monitoring frameworks for their work. The governance group, drawing on monitoring and research expertise, would lead the development of the monitoring framework.

The development process should involve broad and systematic consultation with stakeholders within the disability sector (including disabled people) and within government. There is an opportunity to align the work of the full IMM and the CCMG through the development of a shared framework. Developing a framework was part of the IMM agenda but the task has not yet been resourced.

International context for national alignment of UNCRPD monitoring

New Zealand:The New Zealand Disability Strategy 2016 - 2026 (the Strategy) was developed by the Office for Disability Issues in consultation with other government agencies and the disability sector. More than 1,130 people attended workshops and there were approximately 770 submissions from individuals and organisations submitted online. The DPOs played an important role in its development. The Strategy will guide the work of government agencies on disability issues from 2016 to 2026.

The Strategy is supported by an Outcomes Framework and a Disability Action Plan. The Outcomes Framework will outline how progress against the Strategy will be monitored.

It is currently being developed (2017). The Outcomes Framework will specify targets and indicators, where the information will come from, how often it will be collected, who is responsible for collecting it, and where proxies are needed and how information gaps will be addressed.

The Disability Action Plan will be the primary vehicle for implementation of the Strategy.  The Disability Action Plan presents priorities for action that advance implementation of the Strategy and the UNCRPD. These priorities emphasise actions requiring government agencies to work together, as well as with disability sector organisations and others. .  These plans also include success indicators and refer to use of an evidence base for measuring progress.

The Strategy, Disability Action Plan, Outcomes Framework and UNCRPD monitoring frameworks provide structure and accountability for work to improve the wellbeing of disabled people in New Zealand. Together, they put New Zealand in a strong position, along with Australia, in comparison to the other countries reviewed during this study.

Australia: Each focus area in the National Disabilities Strategy (2010-2020) aligns with priority areas under the UNCRPD and contains clear outcomes. The Australian National Disability Strategy includes trend data indicators against each of the focus areas. These are reported against every two years. 

Austria: The Council of Europe Disability Strategy sets out a number of cross-cutting themes that align to the UNCRPD, and sets priority areas for Europe as a region.  The intention is that the Council of Europe Disability Strategy Committee will work with government representatives to form implementation plans relevant to each nation.  This is currently a work in progress.

The United Kingdom: There is a lack of clear indicators, measures and monitoring framework to guide the monitoring of implementation of the UNCRPD across the UK.  Monitoring appears to be disparate and lacks a comprehensive framework and plan for ensuring the desired outcomes are being met, as intended by the UNCRPD. There are some robust reports available on specific topic areas, but these do not seem to link to a coordinated and planned monitoring approach.

Canada: A number of Canadian disability bodies (national and state-based) have developed strategies but there is no over-arching federal strategy that provides national direction for addressing disability matters. The Canadian Human Rights Commission and the Canadian Civil Society Parallel Report Group provide progress reports against the requirements of the UNCRPD and its articles. The Group is the closest parallel to the CCMG in New Zealand. It was a temporary coalition of sixteen DPOs with government funding.

The monitoring team

Diagrame description - The monitoring team described in the diagram below includes tier 1: the CCMG core monitoring team, tier 2: a project leader, tier 3: a national coordinator and a researcher) and tier 4: monitors, transcribers and a data analyst (Figure 3). The diagram represents the lines of communication flowing between the various tiers and roles in a hierarchical fashion.

Figure 3. Monitoring team. Lines represent lines of communication.

Project leader: The project leader played a role in data collection and reporting though her focus was on project management and communication.

National coordinator: The national coordinator prepared the monitor material including interview guides, policies and procedures, tracking documents, communications, etc). The national coordinator managed the work of the monitors, including arranging interviews and seeking initial consent for participation. Initially these functions were managed by multiple local coordinators but the role was combined into one national coordinator, which benefited the monitoring by condensing institutional knowledge and simplifying communication.

The role extended longer than planned and was challenged by difficulties obtaining contacts from the DPOs to begin the snowball approach used to sample people to interview.

Researcher: The researcher led data analysis and reporting, bringing expertise in analysis of qualitative data and commitment to the purpose of the monitoring. The researcher managed the analysis of transcribed interview notes and analysed them in NVIVO to inform reporting. The researcher was not included in CCMG meetings and had limited control over decision making and oversight. The researcher noted in the end of study ethics review (unpublished):

Future research may wish to address the role of the researcher regarding research responsibility, the training of contractors who are in roles of responsibility regarding research practices, the targeting of disabled people with a research background for key roles, and a review of operational and governance factors that may have impacted on adverse events. 

The researcher was supported through two of the last three years by a data analyst who coded the transcribed interview data. The researcher drafted reports for input from the project lead.

What I did was come up with drafts. Impossible not to write a draft when you’re writing up the analysis. Then the project lead restructured it but the analysis remained the same. (researcher)

The efficiency of all roles was limited by issues including:

• Lack of continuity and handover following changes in the monitoring team roles
• Sometimes long delays waiting for decisions on operational or methodological issues from the CCMG
• Research decisions made by the CCMG with limited input from the researcher
• Long delays in receiving samples of interview participants from the CCMG
• Slow payments of invoices both for time and expenses, resulting in some monitors accumulating debt to Work and Income and Inland Revenue due to having difficulty in reporting their income
• Long gaps between monitor training and beginning interviews due to delays in identifying interview participants
• Unavailability of essential interview equipment (for example, recorders) due to delays in purchasing additional equipment
• Challenges with scheduling interview travel times.

The monitoring could be strengthened by addressing these difficulties through separating governance and operational functions, as discussed in section 4. However, some challenges such as the monitoring’s variable demand for time and materials will remain and will need to be managed on an ongoing basis. Development of a monitoring framework is likely to help plan and manage time and resource requirements.

Monitors: The monitors interviewed disabled people. They were trained in the DRPI method. There is growing evidence in the literature about the advantages of peer interviewers in establishing rapport and collecting robust information from disabled people.

The monitors generally worked relatively small number of hours sometimes spread over long periods of time. Some monitors were given additional responsibilities in managing interview scheduling and logistics as local coordinators.

Monitors spoke very highly of the training provided by the monitoring team and of benefiting from increased understanding of the UNCRPD and disability rights.

I enjoyed the workshopping type part, talking about the questions and how you would open up the person you were interviewing, how to ask open-ended questions. That was really awesome. You don’t really get that kind of training anywhere else. Everyone was able to be involved in that. It was a very participatory type process. We were very excited and looking forward to the work. (monitor)

Building the understanding of human rights in the disability community through training monitors was seen by many stakeholders as a valuable secondary benefit of the monitoring.

I’d really like to be involved again. The biggest benefit I had, especially for people who are blind and deaf. I know my disability, but their disability, wow, I didn’t even think of that! (monitor)

Completing interviews was often very rewarding work and monitors spoke of their commitment to the work.

The one on one stuff was amazing. The training was excellent. The support throughout for us in Auckland, you always worked in pairs. The project head who gathered reports from us was great. The DRPI stuff was great. (monitor)

However, the sparse and unreliable hours along with the paperwork necessary in managing contracting work and reporting income created challenges for some monitors.

Some monitors invested considerable time in reporting their availability and on logistics to conduct interviews. The monitoring aspires to give as wide a group as possible the opportunity to participate as monitors. However, the cost involved in some arrangements can be substantial. More restrictive requirements in selection of monitors would streamline the process and reduce the overhead costs of transport and support time required. The advantages and disadvantages of such an option would need to be considered by the CCMG. 

The data collection approach

Data collection method

The core components of the DRPI approach used for data collection were:

• Qualitative interviews with people with disabilities using open-ended and semi-structured interview questions focused on negative experiences identified by the participant
• Conducting interviews using monitors who are themselves people with disabilities
• Founded on the human rights principles (dignity, autonomy, participation, inclusion and accessibility, non-discrimination and equality, respect for difference).

The monitoring completed 176 interviews under the most recent three-year contract. While this did not reach the target of 300 interviews (100 per year) it still gave many disabled people a chance to have their views heard and communicated. Interviews were conducted by two monitors, or one monitor and the national coordinator (with few exceptions) using the DRPI interview guide.

Right now, I think the reports are going to give the wider perspective of the disability community than the feedback from one or two individuals at the meetings. That’s the benefit of the interview process to gather the information. (Stakeholder)

The DRPI approach should be considered to provide a starting point for adaptation to the New Zealand context and priorities. There is room to further modify the DRPI method to increase the value of the monitoring.

DRPI is a system in place. Where the monitoring mechanism could do better is more community focus groups. Not everyone wants to be interviewed for a couple of hours but they could learn a lot about what people say through group meetings. I might think education is accessible till I go to a meeting and learn that other people have had it better or that I could have expressed my needs better. (stakeholder)

Opportunities include:

• Revisit data collection tools to increase their focus on issues important to New Zealand, for example cultural wellbeing. Changes should be based on a monitoring framework supported by monitoring stakeholders.
• Include other methods of data collection, for example written or online questionnaires, family/whānau focus groups or other group meetings. New Zealand’s disabled community includes great diversity in both disability and culture. There is no one-size-fits-all data collection approach. Including different methods increases likelihood that people will be able to participate in a way that suits them.

I think it’s easy to send things out online – but sometimes you shouldn’t underestimate the power and peer support and almost educational opportunity of getting people together. (stakeholder)
Transcribed interview recordings resulted in thousands of pages of interview transcripts, which were coded in NVIVO using the DRPI coding scheme.

Sampling approach

The DRPI method uses a snowball method to identify and engage interview participants. The monitoring team sought an initial list of interview participants from the DPOs through the CCMG representatives. Monitors would then grow the sample by asking participants at the end of their interviews to suggest names of two friends who might be interested in being interviewed.

The sampling method posed challenges for the monitoring because there were sometimes long delays in receiving contact details for interview participants to start the snowball rolling from the DPOs. Interview participants were sometimes unwilling to suggest other people to be interviewed.

One other issue – in finishing the interview we were required to ask whether the person knew two other people. As monitors, we would then give those names to the coordinator. Often they did not put two names forward. (monitor)

There was little support for the effectiveness of the snowballing approach amongst the monitoring team, stakeholders or monitors.

Development of a monitoring framework and agreement about measures would provide more clarity on who should be interviewed and why. Options include drawing a sample from government administrative data, using social media and local publications to call for participants and utilising the monitors’ networks. Some people, especially those least able to participate, may need to be reached through personal connections. This is particularly important for people disengaged from DPOs, or who are hard to reach for other reasons. Stakeholders identified these challenges as impacting Māori and Pacific participation in the monitoring.

I think there are the usual suspects. The disabled people who are always invited to contribute and have a voice who are well known. That’s great. But the area we find much harder to connect to is the people who don’t have a voice. People living in residential facilities, that sort of thing. There’s a large section who aren’t being asked to contribute. Who aren’t finding ways to contribute. I think that’s a real issue. I don’t know how to connect those people in to the community. Would love to be able to do that. Those are the people who most at risk, who aren’t connected through other people. (stakeholder)

Using additional methods to sample interview participants could increase the efficiency of data collection while reducing the monitoring team’s reliance on contact details from DPO members.

The people we got, … it felt like we could have got so much more stronger and harder information if we just changed the process a little. When I first started we all have huge networks and we couldn’t use any of them. It was frustrating. (monitor)

The other thing was especially the European side of the people we interviewed, they had homes, they were selling homes and buying others. You don’t get Māori in those situations. We missed that type of questioning. We missed out on a lot. Up here it’s word of mouth that gets you in the door. (monitor)

Challenges in data collection materials and logistics

Data collection was a challenging process, primarily managed by the national coordinator. Challenges encountered are summarised below.

Stakeholder involvement and communication

The monitoring aims to include all disabled people in New Zealand. The monitoring must therefore engage the diversity of the disability sector throughout the monitoring process. The term ‘diversity’ here is used to represent different genders, ages, types of disability, ethnic groups, means of communication and any characteristics seen across people who are considered part of the disability sector.

The importance of sharing the monitoring with the disability community is recognised in the founding memorandum of understanding:

The [CCMG] recognises the importance of sharing its work with the regional and global disability communities.

There is strong support for the monitoring as a means to promoting disability rights in New Zealand and giving voice to the disabled community. These views were evident in discussions with stakeholders and with the monitors employed to conduct interviews with people with disabilities.

Stakeholders with low awareness of the current approach or who felt excluded from the process were more dissatisfied with the monitoring as a whole. This feeling of dissatisfaction contributed to desire for alternative reporting to the UNCRPD.

Hopefully next time around there will be a quicker and more in-depth community consultation by whoever is managing the monitoring. (stakeholder)

The monitoring method adopted limited the involvement of stakeholders. However, modifications to the approach, as suggested by this review, create opportunities to engage with stakeholders through:

• Development of the monitoring framework: Include consultation with disability sector stakeholders (including individuals, disability sector organisations, government agencies and IMM agencies) in developing the monitoring framework. Broad participation in the development of the framework will increase support for the monitoring, promote it in the early stages and strengthen the method. The DPOs play an important role in this process representing the views of their membership.
• Data collection. Include stakeholder consultation in the data collection phase. Collecting data from organisations beyond the DPO group will strengthen the monitoring by increasing understanding of the approach taken, including a wider range of perspectives and consequently increasing support for monitoring reporting. The data collected through stakeholder interviews could complement interview data and give more people and a wider range of people the chance to participate in the monitoring process.
• Reporting: It is difficult for stakeholders to engage with monitoring reports when they are delivered as finished products with no prior opportunity to review and comment. Stakeholders can be engaged through discussion of draft findings and the implications for different groups within the disabled community and identification of any factual issues or contextual information that might help in interpreting qualitative data.
  

Using the monitoring data

Formal reporting

The CCMG have published the following reports for the disabled people led monitoring:

• 2016 – The Education and Employment Report (completed but not yet released)
• 2016 – The Health Report (due to be completed by end of November 2016 but not yet published)
• 2015 – Acceptance in Society (standard and large print)
• 2015 – Participation and Poverty (standard and large print)
• 2014 – The story so far (not published online)
• 2013 – Youth monitoring report (available on the DPA website)
• 2013 – Media report (standard and large print)
• 2012 – Disability Rights in Aotearoa NZ (standard, large print and easy read)
• 2010 - Disability Rights in Aotearoa NZ (standard and easy read).

Stakeholders valued the reports as capturing the independent voice of some of New Zealand’s disability community. Some saw the reports as important resources to draw upon in planning and justifying work in their communities.

For us, the positive thing is the [CCMG] … We have been able to say our reports have been significantly influenced by the voice of disabled people. Whether it can be better is a question but it’s there. (IMM stakeholder)

Feedback also identified opportunities for strengthening the reports including:

• Demonstrating the soundness of the conclusions by providing a more thorough account of the sampling, data collection and analysis methods

• Bringing the voice of disabled people through more strongly

• Report the research findings in other forms so they are more likely to engage a wider audience

I don’t know if they reach out wider. I’m in the know, I know where it is and how to find it. But even with the one on one monitoring, that’s when I realised there are people who aren’t in the loop and they are the ones that could use that information. (Stakeholder)

• Focus on the issues that are either higher priority or more likely to influence change.

Report topics were set in advance and included in monitoring contracts. The former chair of the UNCRPD and New Zealand Country Rapporteur noted New Zealand produced a high number of different reports for the Committee in the last reporting round (2014) relative to its size. For example, the DPOs also produced the shadow report for the Committee in a process separate from the work of the CCMG.  He suggested reducing the number of reports in the UNCRPD process could strengthen the influence of the UNCRPD in New Zealand.

I would encourage disabled groups to put in one report next time. (former UNCRPD chair)

In future, focusing more intensively on fewer issues, which can be selected in the development of the monitoring framework, may make the reports more useful tools for informing the IMM and influencing change. Increasing the engagement of the sector with the CCMG and the monitoring could contribute to more focussed reporting.

I would say what are the main issues you would like the committee to think about. What are the things you want to highlight that you want the government to focus on. Selecting the issues to focus on is very difficult. Doesn’t mean you don’t mention other issues but the question is, how do you want to push the dialogue. The issues selected should balance importance and likelihood to change. (former UNCRPD chair)

Some stakeholders commented the reports did not clearly target an audience.

They tried to theme them but I’ve read every one of them and I’m not sure who they are for. Who the audience is. They could miss everybody. It’s a difficult one. The audience is wide. They should be reaching disabled people. But they also have an academic, monitoring and Government role. (Stakeholder)

People who may be expected to use the reports include:

• The other IMM partners in producing the IMM reports 
• Government stakeholders
• Disabled people
• The wider disability sector including disability service providers and organisations representing disabled people
• Media and the public.

Different audiences respond to different types of reports. For example, the public and media may engage better with summary level reporting while government stakeholders and the IMM may need more detail. Developing a reporting plan with different products for different audiences mapped to the four-yearly UNCRPD Committee cycle may increase the reach and profile of the monitoring.

Regardless of the format of the reporting, making them easy and intuitive to access online is an important part of making them available to the disability community.

Working with government

Government agencies are an important part of the disability sector. The monitoring is led by disabled people for disabled people, but meaningful engagement with the entity being monitored, government, can influence change directly rather than through the UNCRPD Committee’s review of New Zealand. Building good relationships with stakeholders within government does not reduce the independence of the process.

Government stakeholders currently have little involvement and in some cases awareness of the monitoring. Government stakeholders reported they wanted to be able to discuss the findings of the reporting before being presented with the final reports. Engaging with government stakeholders throughout the monitoring process increases the likelihood that government will act on the findings of the monitoring. Government may also be able to provide useful information (for example statistics based on administrative data) to inform analysis and reporting.

Building a relationship of trust between the monitoring and government begins with consistent communication from both sides. Stakeholders noted the expertise needed for the CCMG to navigate government processes and represent the findings to high-level meetings.

The experience of those who attend. There are some represented at the CE level who are comfortable in these situations, but others who don’t have that experience and knowledge of the machinery of government. (DPO)

Working more closely with government stakeholders through the process could contribute to building the capacity of the CCMG members as well as government stakeholders’ understanding.

Understanding the nature and the context of the people representing the DPOs, issues with communication or using sign language interpreters or whatever, chuck that into the mix of their individual communication issues. Chuck in that they’re meeting Government. If you don’t have the right people in place, it’s quite a challenge to ensure they’re heard. Completely understandable. (govt stakeholder)