Education, New social contract

To measure real progress in education we must include out-of-school children

By Michael Ward, Senior Analyst, Education and Skills Directorate, OECD

In many low- and middle-income countries – including some that have participated in PISA – relatively large proportions of 15-year-olds are not enrolled in school or are not enrolled in PISA’s target grades (grade seven and above) and are thus not covered by the assessment (see figure 1). With an increasing number of low- and middle-income countries participating in PISA, and with 61 million children of lower secondary school age, out of school around the world, this population can no longer remain beyond the reach of programmes that try to evaluate the success of education systems.

Why does this matter?

In contexts where large numbers of youth are out of school, using a sample of in-school eligible students as a proxy for a cohort of 15-year-olds or an entire schooling system can lead to misleading results, such as:

  • Inaccurate cross-country comparisons when using only PISA data, especially when countries have different levels of incomplete access or different proportions of delayed (and ineligible) students.
  • Underestimates of progress ofreal educational improvements over time for countries that have improved access or attainment.
  • Underrating socioeconomic inequalities when sampling focuses on schools and misses out the poorer youth that are out of school or enrolled in grade six and below.
  • Potential perverse incentives for countries to maintain policies of exclusion to inflate performance in international tests.

What does it mean for assessing the efficiency of education systems?

Measuring school achievement through the administration of tests refers to students – not the whole population – and this makes it difficult to assess the efficiency of education systems and whether all children and young people have what they need to reach their full potential, especially in low- and middle-income countries.

To assess the efficiency of an educational system, it is important to see test scores in the context of participation rates and the extent to which the dropout rate in a country has been reduced between any two measurement points. When it comes to evaluating the skills of the population as a whole, it is essential to determine the capacities of those who drop out of school and those that never enrolled. Moreover, dropping out and never enrolling are negatively correlated with socio-economic status. As mentioned above, an assessment that ignores the out-of-school is at risk of perversely encouraging policies of exclusion.

While it is possible to estimate test scores for the whole population (that is, taking into account those who drop out or never enrol) by putting bounds on unobserved scores, this is effectively guesswork and is carried out under weak assumptions.

What are the solutions?

In the context of Sustainable Development Goal 4 (quality education) with its emphasis on leaving no one behind, PISA-D (PISA for Development) is the first attempt in the history of international large-scale assessments to evaluate the skills of the whole population by incorporating the out-of-school population. Through its proven cognitive and contextual data instruments, its effective delivery platform through tablet computers, its approach to sampling and survey operations, the PISA-D pilot has shown that out-of-school youth are no longer beyond the reach of large-scale learning assessment.

Currently, the OECD and its partners are preparing to scale-up the initiative in low-income countries in particular, especially by linking to or integrating the PISA-D out-of-school assessment in national household surveys or household surveys conducted with the support of development agencies.

While it is difficult and fraught with approximations, it is possible to extrapolate, theoretically at least, from the PISA-D out-of-school assessment results to the rest of the world. For example, by matching the results of specific PISA-D participating countries to ‘statistical neighbours’ among the PISA 2018 participants where less than 80% of the population of 15-year-olds were covered by the PISA sample. We hope these low coverage countries will feel encouraged to see themselves in the PISA-D out-of-school assessment results and consider the relevance of this work for their contexts.

Figure 1 Coverage of the national 15-year-old population in PISA 2018 (Coverage Index 3)