Monitoring/Evaluation of Policies - Disability Indicators 1/4

Theme: Statistics, Data and Evaluation, and Monitoring
Programme Monitoring and Evaluation; The Disability Perspective in the Context of Development

Previous | Contents | Next

IV. DEVELOPMENT OF "GLOBAL DISABILITY INDICATORS"

A. Information currently available for the development of indicators

1. Basic parameters for indicators established by the World Programme

As with Programme monitoring, there are few detailed specifications for the development of "global disability indicators," indicated in the World Programme. Evaluation indicators are to be selected in consultation with Member States, relevant United Nations agencies and other organizations.[165] The Programme does however call for the development of "social indicators on the education of persons with disabilities to analyze problems and plan programmes accordingly" among its recommended "Research" areas.[166] Finally, General Assembly resolution 50/144, in its operative paragraph eight, reflects the interest of member States in "the development of global disability indicators".

All too often, the concept of indicators is confused with that of monitoring. While both concepts involve making choices, there is a crucial distinction. As noted earlier, monitoring refers to the practice of setting goals and objectives and then establishing evaluation criteria to determine whether the goals and objectives have been achieved. Data can be gathered periodically based on those evaluation criteria, which will provide measures of those evaluation criteria. Indicators are the data elements that are believed to provide the best measures of progress.

An example from the field of labour and employment may help to explain the need for indicators. There are many different ways to calculate unemployment rates. For instance, some unemployment rates include workers who have given up looking for work. These are often referred to as discouraged workers and are often considered to be economically inactive, because, even though they wish to be employed, they are not looking for work. Because some argue it is desirable to calculate unemployment rates based on the economically active population, some other unemployment rates do not include discouraged workers. While both types of unemployment rates are useful measures and many countries may calculate both, most countries choose one method to provide the official estimate of the unemployment rate. The usefulness of one official estimate is that communication is enhanced. When discussions of unemployment ensue, persons will be addressing the same phenomenon, and programmes can be targeted towards addressing that phenomenon. If the various unemployment rates tend to move in similar directions, it is easier to focus people's attention on a single unemployment rate, as opposed to several. Thus, the official estimate can be viewed as the country's indicator of unemployment.

Turning to the concept addressed in the World Programme, the United Nations General Assembly may have envisioned that a set of good indicators on education would enhance communication and focus attention on those aspects of education which should be addressed to promote equality of opportunity. For example, a country may be at a stage of development where it is focused on promoting the completion of primary and secondary education and few attend post-secondary education. In such a situation, a statistic measuring the equality of disabled and non-disabled people in the completion of post-secondary education may be of some interest but it may not be the most useful indicator of the educational situation of persons with disabilities. The country may choose a set of indicators reflecting the situation in primary and secondary education. A different, highly developed country where virtually everyone completes primary education might make the exact opposite choice of an indicator.

Indicators, by definition, are valued. When viewed in this sense, the focus is not on value in terms of goals and objectives but in terms of its usefulness as a tool for monitoring progress towards goals and objectives. Certainly it is true that an unemployment rate may have value in terms that society might view a reduction in the rate as a positive development. However, as an indicator, the official unemployment rate has value in that it has been chosen as that data element that best reflects the unemployment situation.

A set of global disability indicators could have value as those elements that best reflect the situation of persons with disabilities in a country. A question might be raised as follows: should not indicators for persons with disabilities be the same as for the entire population? The answer to this question would be yes if the characteristics of the population with disabilities are the same as for the total population, except for disability. As demographic and some socio-economic characteristics that may be associated with the presence of a disability are significantly different for the disabled population than for the total population, indicators have to be chosen to remove the effect of those characteristics. Note, however, the need for a particular set of indicators for disabled persons does not imply that judgments should be made that equality of opportunity cannot be achieved. If this occurs, the very presence of the indicators becomes a barrier to the aspirations of disabled people. A result of this is that indicators would clearly not be valued by persons with disabilities and their families. Moreover, they should not be valued in that, in the view of the consumers, the indicators would not be the best reflectors of the situation of persons with disabilities.

To have value, disability indicators should have several attributes. First, they must be viewed as useful by disabled persons, their families and other interested constituencies. Second, they should be determined to be useful reflectors of what they purport to measure through standard scientific criteria. Finally, there should be a reasonable, non-arbitrary limit on their number. If the set meets these criteria, they can do their job, which is to provide an indication of the situation.

An additional caveat is warranted. While a set of indicators should have value, it should be viewed as only one tool for monitoring. Ironically, if a set is established and viewed as the only set of measures of progress towards achievement of World Programme goals, its value is actually diminished in several ways. For example, different constituencies demand that the set be endowed with the capacity to reflect all aspects of the disability situation, usually leading to the addition of so many indicators that the ability to focus is lost. This can then result with the indicators becoming a set of mini-goals, rather than measurement of progress towards goals. Moreover, other potentially useful measurement efforts that are not part of the set become devalued. The focus of a set should be on their value as providing indications, not on their goal-related values.

The concept of "global disability indicators" poses some additional problems. There are several types of indicators that can be used. These reflect the different monitoring concepts noted earlier and include (a) statistical indicators of changes over time, (b) measures of programme assessment and performance and (c) measures of the achievement of specific targets. Given different needs, it is difficult to declare one type of indicators as preferable for all countries. Even within these categories, there are some problems. Turning to the example cited earlier, if developed countries wish to focus on achievements at secondary education and higher while developing countries may need to focus on secondary education and lower, the temptation becomes strong to declare the indicator of educational progress to be the cross-tabulation of presence of disability by all levels of educational attainment. While such information is useful for monitoring, establishing it as the educational indicator makes the indicator equivalent to monitoring. In other words, the establishment of indicators has no value beyond monitoring. This then raises the question, given the heterogeneity of countries, is the concept of "global disability indicators" even viable? By reviewing information currently available for their development and taking a case studies approach, it is hoped that the viability of the concept will be demonstrated in this section.

2. Country activities

Few countries have conducted surveys on disability. Thus, the census constitutes the major source of disability data for many countries. Over the past forty years, the inclusion of disability issues on censuses has grown dramatically. In the 1960 round of population and housing censuses, only three censuses included disability questions. In the 1970 and 1980 rounds, this number grew to 18 and 57, respectively. For the 1990 round, 84 censuses have been documented as having questions related to disability.

In assessing the situation in 1980, the World Programme of Action took particular note of the education and employment status of people with disabilities in assessing the equalization of opportunities. Since 1980, at least 12 countries have produced census tabulations on education and 15 have produced tabulations on employment variables. Two countries (Cyprus and Zambia) have data from two censuses on education and employment while Tunisia has data from two censuses on employment.

Many countries have conducted surveys, as well as censuses. Since 1992, at least 36 countries have included questions on disability in a census, obtained information from a survey or obtained estimates. Information was obtained on age and gender (17 sources), education and labour force variables (5 sources), urban and rural residence (9 sources), marital status (4 sources), cause of disability (10 sources) and other topics (4 sources). Of particular interest is the relatively high number of countries collecting data on etiology.[167]

A clear pattern of increasing coverage can be discerned. While it appears that few countries have established clear indicators of progress towards Programme goals, data related to the target areas of the Standard Rules are being collected and are published to a greater extent. While little progress appears related to gathering data for indicators of the environment and specific life areas, the situation for indicators of overall outcomes appears promising.

3. Activities of international organizations

International organizations have taken steps towards monitoring disability among children. WHO and UNICEF reached an agreement on indicators for monitoring the health goals of the World Summit for Children.[168] To monitor progress towards the goal of improved protection of children in especially difficult circumstances, two broad indicators were proposed: (a) the total number of persons with disability of six months duration and (b) the total disability rate per 1,000 children. Disability categories covered are seeing, hearing and speaking, moving, learning and comprehending, behaviour and other disability. Dissagregations would occur by age groups (under 5, 5 to 14, 15 to 19 and 20 years of age and over) by men and women and by urban and rural areas. Another potential indicator considered is recent rehabilitation service coverage. WHO has incorporated these issues into its broad programme indicators.

WHO and the World Bank have also collaborated in efforts to develop quality of life indicators, based on adjustments to life years. This effort has been considered controversial in some quarters, because disability is equated on a scale with death and because the estimates are based on clinical estimates of prevalence, as opposed to information obtained from DISTAT.[169]

The indicators discussed thus far appear to be somewhat more oriented to the goals of prevention and rehabilitation than to equalization of opportunities. While few formal indicators have been established in this area, UNESCO has participated in an OECD process to develop indicators of special needs education, which could provide indicators based on needs as opposed to disability category.

Coverage is an important issue for international organizations to consider when establishing data for indicators for equalization of opportunities. When the Special Rapporteur on Disability first conducted his survey (in late 1994), only 38 Governments submitted data. In the second survey (in late 1996), the number rose to 83, representing a survey coverage of 45 per cent of the member States. In the UNESCO study, as noted earlier, a request for information was sent to 70 countries, of which 52 or 74 per cent responded. Information is available on many issues, but it clearly would be desirable to improve coverage of these surveys further.

As noted earlier, the Statistics Division has made great progress towards the compilation of disability statistics. By establishing topics for tabulation in DISTAT and by drafting recommendations for the collection of data, both of which have a good association with the Standard Rules, the Division has provided a solid foundation of topics from which to select indicators of outcomes.

4. Problems encountered in third review and appraisal

While certain aspects of the data situation are promising, a set of global disability indicators was not available for the third, 1997, review and appraisal of the World Programme of Action. To consider what such a set of indicators might look like, it is useful to return to the example of employment raised in section III.C, above. Indicators for employment could be selected from the examples of specific measures for employment shown in figure 5. This would result in the reduction of the number of measures to a series of indicators, perhaps similar to those presented in figure 6.

Figure 6. EXAMPLES OF INDICATORS FOR A SYSTEMATICSET OF EMPLOYMENT INDICATORS BY MONITORING CONCEPT AND TARGET AREA

Monitoring Concept	Target Area	Example of Indicator
Socio-economic outputs	Increase economic activity	Labour force participation rate by age
	Reduce unemployment	Unemployment rate by age
Environmental variables	Improve access to positions	Number of employers and hiring targets Number of rehabilitation programmes
Target areas for access	Economic self-sufficiency	Income
	Transition	Number of vocational students trained
Programme outputs	Employment and schools	Graduation rates Number of students placed in jobs
Programme inputs	Employment and schools	Educational budget Number of crentialled teachers

It is important to note how indicators may be specifically targeted and, in some cases may serve only as a proxy measure for the phenomenon. For example, if a country has a low level of economic activity for women, it may need to compare labour force and unemployment rates for disabled and all men between 16 and 64 years of age as key indicators for employment outputs. If that country has increasing participation among women in young ages, it might choose to compare the number in school-to-work training programmes for disabled and all women in the school ages as a key indicator of transition opportunities. If a country cannot obtain good information on training for students with disabilities in its schools, it might need to use graduation rates for disabled students as a proxy for those training programmes.

Ideally, indicators would be established at the beginning of a period for review and appraisal, with baselines and specific targets. Actual change over time could be observed with information explored to explain any disparities between specific targets and actual indicators. Such a set of indicators was unavailable for the current review and appraisal of the World Programme. Given the lag time between data collection and publication in a census, it is unlikely that the data required to produce disability indicators from the 2000 round of censuses will be available in 2002, when a fourth review and appraisal is envisaged. Nevertheless, the progress in data development discussed earlier establishes a sound basis for long-range development of indicators.

Notes:

[165] World Programme of Action …, paras 194-195.

[166] Ibid., para 184.

[167] "Review and Appraisal … (A/52/351)" para 45.

[168] World Summit for Children (New York, 29-30 September 1990) [http://www.unicef.org/wsc/index.htm].

[169] Murray, Christopher L. And Lopez, Alan D., The Global Burden of Disease: a comprehensive assessment of mortality and disability from diseases, injuries and risk factors in 1990 projected to 2020 (Cambridge, Harvard University Press, 1996); and Murray, Christopher L. And Lopez, Alan D., "Alternative projections of mortality and disability by cause, 1990-2020: Global Burden of Disease Study", Lancet: 349 (1997) pp 1498-1504.

Previous : Next