Conceptualising and Measuring Democracy Evaluating Alternative ...

Download PDF
28 downloads 222 Views 359KB Size Report
Comment
Conceptualising and Measuring Democracy. Evaluating Alternative Indices. G.L. Munck & J. Verkuilen. Foivos Skalidis. 29/04/2013. Comparative Politics.
Conceptualisation

Measurement

Aggregation

Conceptualising and Measuring Democracy Evaluating Alternative Indices G.L. Munck & J. Verkuilen Foivos Skalidis

29/04/2013

Comparative Politics

Universit` a degli Studi di Milano

Conceptualisation

Measurement

Aggregation

Motivation

Quantitative researchers have overemphasised the problems of causal inference... ...while neglecting the quality of the data on democracy that they analyse.

Comparative Politics

Universit` a degli Studi di Milano

Conceptualisation

Measurement

Aggregation

Motivation

Quantitative researchers have overemphasised the problems of causal inference... ...while neglecting the quality of the data on democracy that they analyse.

Comparative Politics

Universit` a degli Studi di Milano

Conceptualisation

Measurement

Aggregation

Datasets on Democracy

Comparative Politics

Universit` a degli Studi di Milano

Conceptualisation

Measurement

Aggregation

Datasets on Democracy

Comparative Politics

Universit` a degli Studi di Milano

Conceptualisation

Measurement

Aggregation

Aim

Constructing a framework for the analysis of data, discussing three challenges: I

Conceptualisation

I

Measurement

I

Aggregation

Comparative Politics

Universit` a degli Studi di Milano

Conceptualisation

Measurement

Aggregation

Conceptualisation 1. Identification of attributes I

I

Avoid maximalist definitions Decreases the usefulness of a concept Little analytical use Avoid minimalist definitions All cases automatically become instances Difficulty in discriminating among cases

2. Vertical organisation of attributes by level of abstraction I

Comparative Politics

Avoid redundancy and conflation of attributes Directly affects measurement and aggregation

Universit` a degli Studi di Milano

Conceptualisation

Measurement

Aggregation

Conceptualisation 1. Identification of attributes I

I

Avoid maximalist definitions Decreases the usefulness of a concept Little analytical use Avoid minimalist definitions All cases automatically become instances Difficulty in discriminating among cases

2. Vertical organisation of attributes by level of abstraction I

Comparative Politics

Avoid redundancy and conflation of attributes Directly affects measurement and aggregation

Universit` a degli Studi di Milano

Conceptualisation

Measurement

Aggregation

Conceptualisation 1. Identification of attributes I

I

Avoid maximalist definitions Decreases the usefulness of a concept Little analytical use Avoid minimalist definitions All cases automatically become instances Difficulty in discriminating among cases

2. Vertical organisation of attributes by level of abstraction I

Comparative Politics

Avoid redundancy and conflation of attributes Directly affects measurement and aggregation

Universit` a degli Studi di Milano

Conceptualisation

Measurement

Aggregation

Conceptualisation

Figure: The logical structure of concepts

Comparative Politics

Universit` a degli Studi di Milano

Conceptualisation

Measurement

Aggregation

Measurement 1. Selection of indicators I

I

I

Use of multiple indicators Single indicators entail the risk of bias Multiple indicators pose the burden of establishing cross-system equivalence Use of indicators that minimise measurement error Do unbiased historical data exist? Reliability The same data collection process should produce the same data

2. Selection of measurement level I

I

Comparative Politics

Maximise homogeneity within measurement classes with the minimum number of necessary distinctions Too fine-grained, too coarse-grained distinctions Reliability Multiple coders should produce the same codings Universit` a degli Studi di Milano

Conceptualisation

Measurement

Aggregation

Measurement 1. Selection of indicators I

I

I

Use of multiple indicators Single indicators entail the risk of bias Multiple indicators pose the burden of establishing cross-system equivalence Use of indicators that minimise measurement error Do unbiased historical data exist? Reliability The same data collection process should produce the same data

2. Selection of measurement level I

I

Comparative Politics

Maximise homogeneity within measurement classes with the minimum number of necessary distinctions Too fine-grained, too coarse-grained distinctions Reliability Multiple coders should produce the same codings Universit` a degli Studi di Milano

Conceptualisation

Measurement

Aggregation

Measurement 1. Selection of indicators I

I

I

Use of multiple indicators Single indicators entail the risk of bias Multiple indicators pose the burden of establishing cross-system equivalence Use of indicators that minimise measurement error Do unbiased historical data exist? Reliability The same data collection process should produce the same data

2. Selection of measurement level I

I

Comparative Politics

Maximise homogeneity within measurement classes with the minimum number of necessary distinctions Too fine-grained, too coarse-grained distinctions Reliability Multiple coders should produce the same codings Universit` a degli Studi di Milano

Conceptualisation

Measurement

Aggregation

Measurement

3. Recording and publicising of coding rules, coding process and disaggregate data I

Replicability

Comparative Politics

Universit` a degli Studi di Milano

Conceptualisation

Measurement

Aggregation

Aggregation 1. Selection of level of aggregation I

Balance between parsimony and the concern of underlying dimensionality and differentiation High level of aggregation results in loss of validity Low level of aggregation impedes theorising and testing

2. Selection of aggregation rule I

I

Comparative Politics

Ensure the correspondence between the theory of the relationship between attributes and the selected rule of aggregation Attributes could be sufficient, necessary Robustness of aggregate data Changes in the aggregation rule should result in proportionate changes in the aggregate data Universit` a degli Studi di Milano

Conceptualisation

Measurement

Aggregation

Aggregation 1. Selection of level of aggregation I

Balance between parsimony and the concern of underlying dimensionality and differentiation High level of aggregation results in loss of validity Low level of aggregation impedes theorising and testing

2. Selection of aggregation rule I

I

Comparative Politics

Ensure the correspondence between the theory of the relationship between attributes and the selected rule of aggregation Attributes could be sufficient, necessary Robustness of aggregate data Changes in the aggregation rule should result in proportionate changes in the aggregate data Universit` a degli Studi di Milano

Conceptualisation

Measurement

Aggregation

Aggregation 1. Selection of level of aggregation I

Balance between parsimony and the concern of underlying dimensionality and differentiation High level of aggregation results in loss of validity Low level of aggregation impedes theorising and testing

2. Selection of aggregation rule I

I

Comparative Politics

Ensure the correspondence between the theory of the relationship between attributes and the selected rule of aggregation Attributes could be sufficient, necessary Robustness of aggregate data Changes in the aggregation rule should result in proportionate changes in the aggregate data Universit` a degli Studi di Milano

Conceptualisation

Measurement

Aggregation

Aggregation 1. Selection of level of aggregation I

Balance between parsimony and the concern of underlying dimensionality and differentiation High level of aggregation results in loss of validity Low level of aggregation impedes theorising and testing

2. Selection of aggregation rule I

I

Comparative Politics

Ensure the correspondence between the theory of the relationship between attributes and the selected rule of aggregation Attributes could be sufficient, necessary Robustness of aggregate data Changes in the aggregation rule should result in proportionate changes in the aggregate data Universit` a degli Studi di Milano

Conceptualisation

Measurement

Aggregation

Aggregation

3. Recording and publicising of aggregation rules and aggregate data I

Replicability

Comparative Politics

Universit` a degli Studi di Milano

Conceptualisation

Measurement

Aggregation

Conclusions

No single index offers a satisfactory response to all three challenges Comparative Politics

Universit` a degli Studi di Milano

Conceptualisation

Measurement

Aggregation

Conclusions

Interestingly, datasets have a high level of correlation. Why? I

Same sources, same precoded data

I

No sense of validity, only of their reliability

I

Correlation tests have been performed with highly aggregated data

Comparative Politics

Universit` a degli Studi di Milano

Conceptualisation

Measurement

Aggregation

Conclusions

Finally... I

...mathematical statistics often presume that the relationship between theory, data and observations has been well established....

I

...this comprehensive framework for the generation and/or analysis of data, draws attention to issues underpinning causal inference.

Comparative Politics

Universit` a degli Studi di Milano

Conceptualisation

Measurement

Aggregation

Conclusions

Finally... I

...mathematical statistics often presume that the relationship between theory, data and observations has been well established....

I

...this comprehensive framework for the generation and/or analysis of data, draws attention to issues underpinning causal inference.

Comparative Politics

Universit` a degli Studi di Milano

Conceptualisation

Measurement

Aggregation

Conclusions

Finally... I

...mathematical statistics often presume that the relationship between theory, data and observations has been well established....

I

...this comprehensive framework for the generation and/or analysis of data, draws attention to issues underpinning causal inference.

Comparative Politics

Universit` a degli Studi di Milano