► Superficially, one would assume that Korea is making excellent progress toward the urban-related Sustinable Development Goal 11 (SDG 11), which seeks to “make cities and human settlements inclusive, safe, resilient and sustainable”. 

► However, progress is less clear, if closely examined. Some indicators selected by the Korean government suffer from two common statistical faults: 1) aggregation bias and 2) cherry-picking. 

► In the first case, through aggregation bias, researchers can overlook or ignore the worsening conditions for some Koreans, for instance, Airborne particulate matter (PM). 

► In the second case, the Korean government has chosen only one indicator- the proportion of people under the Minimum Housing Standard, while they could have chosen the precise indicator recommended by the UN. This appears that some official national indicators may have been cherry-picked to depict the country in a more positive light. 

Introduction

Superficially, one would assume that the Republic of Korea (hereinafter “Korea) is making excellent progress toward the urban-related Sustainable Development Goal 11 (SDG 11), which seeks to “make cities and human settlements inclusive, safe, resilient and sustainable”. After all, Korea is one of the wealthiest and most urbanized countries in the world, and urban settlements are arguably one of the most sustainable forms of settlement. And superficially, Korea is making progress. The indicators and national-level data officially presented by Korea to the international community all show improvement.

If one digs deeper, however, progress is less clear. Some indicators selected by the Korean government suffer from two common statistical faults: aggregation bias and cherry-picking. Data aggregation at the national level introduces aggregation bias that obscures sub-national variation, including urban areas that have experienced worsening environmental quality. Meanwhile, it appears that data has been selected or ignored to convey positive outcomes, a practice referred to as “cherry picking”. In the following paper, we explore these two faults employing examples from SDG 11 targets, showing that while much progress is being made, caution is advised.

Aggregation bias

Aggregation bias occurs when trends expressed by aggregate data are assumed to apply to individual cases. That is, aggregation bias occurs when aggregate data shows an overall decreasing (or increasing) trend and one assumes that all constituent cases are decreasing (or increasing) over time at similar rates. For example, aggregation bias occurs when data on vegetables shows a ten percent increase over the past year and we assume that prices for each vegetable have gone up roughly ten percent, even though artichoke prices may have increased dramatically while onion prices decrease. In short, aggregation bias overlooks oftentimes meaningful differences between individual cases.

Particulate matter

One example that viscerally demonstrates the importance of avoiding aggregation bias is particulate matter in Korea. Airborn particulate matter (PM) consists of a mixture of minute particles suspended in the air we breathe. When we inhale, PM enters our lungs and can cause significant damage over the long term by damaging lung tissue and causing lung inflammation. In both the short and long term, this can even result in premature death and lower cognitive ability (Sakhvidi et al. 2022; Waidyatillake et al. 2021). PM 2.5, which refers to particulate matter smaller than 2.5 microns, is particularly damaging as it penetrates more deeply into our lungs.

Concern about rising PM levels is integrated into Target 11.6, which calls for “reduc[ing] the adverse per capita environmental impact of cities, including by paying special attention to air quality and municipal and other waste management.” The primary indicator identified by the UN and adopted by Korea is population-weighted annual mean levels of fine particulate matter (e.g., PM2.5 and PM10) in cities by 2030.

The data offered by the Korean government show a steady decline from 26μ g/m^3 in 2015, when data on PM 2.5 was first collected, to 18μ g/m^3 in 2021. This represents an improvement in air quality overall. A closer look at subnational data, however, shows some variation in outcomes. Figure 1 shows regression lines for PM 2.5 levels recorded every month to more clearly illustrate changes in each urban area. The national average is shown by the solid black line, which is closely paralleled by dark grey lines indicating Korea’s major cities (Seoul, Busan, Daegu, Incheon, Gwangju, Daejon, and Ulsan). Together, the downward trends in these cities, which are home to the majority of Korea’s population, represent a meaningful reduction of PM 2.5 by roughly one-third since 2015.

Figure 1. PM 2.5 level trends in urbanized areas of Korea

Source: KOSIS, Table DT_106N_03_0200145.

Though these large cities are consistent with the aggregated trends, some urban areas have performed significantly better and some significantly worse. Those cities represented by dotted black lines in Figure 1 (Jiangsu, Sunchang, Taebaek, and Yeongwol) have reduced PM 2.5 by more than half and as much as two-thirds and now have some of the country’s lowest concentrations. Counteracting these success stories, however, are urban areas indicated by long dashed black lines with worsening PM 2.5 concentrations: Cheongdo, Hapcheon, Uiseong, Uljin, Ulleung-do, Yecheon, Yeongyang, and Yesan. A few of these urban areas remain well below average, while others now exceed the average, but all of them have experienced a decline in environmental quality.

Commentary

Choosing the appropriate level of analysis for any measure is a challenge endemic to all quantitative analysis. Indeed, the very nature of quantitative analysis compels analysts to aggregate information to at least some degree (Latour 1999). Aggregate too little and broad trends are missed. Aggregate too much and important variation is missed. One must always bear in mind both the forest and the trees. This is especially so in cases like the SDGs, as they are intended to increase the quality of life for all people. If we allow large-scale improvements to obscure localized declines, we undermine the spirit of the SDGs.

Of course, in the case of particulate matter, the Korean government has chosen the exact indicator recommended by the UN. So clearly, some responsibility for the risk of aggregation bias can be attributed to the UN, for whose purposes subnational PM concentration data would create assessment difficulties.

Cherry picking

In some cases though, the Korean government seems to have elected to ignore readily available data that the UN recommends collecting. These decisions, whether deliberate or not, are examples of a second statistical fault: cherry-picking. Cherry-picking refers to the practice of choosing only data that reflects the results the analyst desires. That is, if a variable, data set, or statistical method produces undesirable results, the analyst simply chooses a different variable, data set, or method.

Target 11.1 offers clear evidence of this practice in Korea. According to the UN’s SDG indicator metadata, Target 11.1 calls for “ensur[ing] access for all to adequate, safe and affordable housing and basic services and upgrade slums” by 2030. The primary indicator for this target is the “proportion of the urban population living in slums, informal settlements or inadequate housing”. The Korean government has chosen a single national indicator for this target: the proportion of people under the Minimum Housing Standard. This indicator calculates the proportion of the population living in housing units with less than the minimum acceptable number of habitable rooms and floor area for their household composition.

The national level data again look promising. Though the proportion of people living below the Minimum Housing Standard reached its minimum of 4.7 percent in 2014, overall it has dropped from 12.8 percent in 2006 to 6.7 percent in 2017 and 5.5 percent in 2021. This shows that Koreans in aggregate are enjoying larger living spaces than in the past.

The concern is that Korea has chosen only one indicator to address a target that embraces a vast array of possible indicators that inform the fuzzy concepts of adequate, safe, and affordable. Though some of the nine plus possible indicators recommended by the UN are readily available in national data sets, the government has chosen an indicator it recognizes as different from the UN indicator.

Instead, Korea could have chosen the precise indicator recommended by the UN for evaluating “adequate housing”. To capture the importance of affordability in defining adequate housing, the UN recommends calculating the “proportion of households with net monthly expenditure on housing exceeding 30% of the total monthly income of the household”, a widely accepted figure (Stone 1993). The UN indicator metadata documents explicitly suggest that the house price-to-income (HPIR) and the rent-to-monthly household income (RIR) can also be used, two readily available indicators. Though these measures are aggregate measures, Statistics Korea (direct communication) claims they were dismissed because the official indicator calls for the proportion of households.

Figure 2. Housing price-to-income ratio (PIR) for major Korean cities

Source: KOSIS, Price-to-Income Ratio (PIR, 2006-2021):

https://www.index.go.kr/unity/potal/indicator/IndexInfo.do?cdNo=2&clasCd=10&idxCd=F0188

As Figure 2 shows, however, Korea has been failing miserably on this indicator. The UN suggests that an HPIR less than 3.0 indicates housing affordability. All major metropolitan regions and Korea as a whole have risen to above 5.0 and Sejong City even exceeded 9.4 at one point. While a few cities were affordable in the late 2000s (Ulsan, Gwangju, Sejong, and Incheon), not a single city would have been considered affordable when the SDGs were introduced in 2015. And now, all cities except Ulsan are over twice the affordable rate. Korea is moving away from this SDG target rather than toward it, a clear failure.

Commentary

Korea is not alone in suffering from rising housing prices relative to income. Housing prices have been rising globally, causing consternation for most people who do not already own their own homes (The Economist 2023). However, this is not an acceptable rationale for ignoring a recommended and readily available measure of adequate housing. One can only assume that this data was ignored as it would highlight Korea’s difficulty reaching some aspects of SDG Target 11.1 and that the proportion of people living under the Minimum Housing Standard was cherry-picked show progress.

Conclusion

This paper has explored two indicators that illustrate two statistical faults. The discussion indicates that caution is advised for researchers, policymakers, and social movements that analyze the data offered by the Korean government as indicators of progress toward the SDGs. In the first case, aggregation bias can lull researchers into overlooking and ignoring worsening conditions for some Koreans. In the second case, it appears that some official national indicators may have been cherry-picked to depict the country in a positive light, while ignoring very real and growing social problems, like housing affordability.

References

Latour, Bruno. 1999. Pandora’s Hope: Essays on the Reality of Science Studies. Cambridge: Harvard University Press.

Sakhvidi, J.Z., Jun Yang, Emeline Lequy, Jie Chen, et al. 2022. Outdoor air pollution exposure and cognitive performance: findings from the enrolment phase of the CONSTANCES cohort. The Lancet Planetary Health, 6(3), E219-E229. https://doi.org/10.1016/S2542-5196(22)00001-8

Stone, Michael. 1993. Shelter Poverty: New Ideas on Housing Affordability. Philadelphia: Temple University Press.

The Economist. 2023. The Great Escape: Is the global housing slump over? The Economist. Retrieved from https://www.economist.com/finance-and-economics/2023/06/12/is-the-global-housing-slump-over

Waidyatillake, N. T., Campbell, P. T., Vicendese, D., Dharmage, S. C., Curto, A., & Stevenson, M. 2021. Particulate Matter and Premature Mortality: A Bayesian Meta-Analysis. International journal of environmental research and public health, 18(14), 7655. https://doi.org/10.3390/ijerph18147655