Are charter schools effective?  Philadelphia Inquirer columnist Joel Naroff wrote about the data we might use to decide whether charter schools are doing well. He describes the pitfalls of using achievement test scores to measure school effectivness. His article uses the language of multiple regression and internal validity to describe how difficult this process is.

Specifically, Naroff mentions the following about charter schools, discussing

….what researchers call "selection bias." Parents have to choose to send
their children to a charter school, meaning that charters start with
students who have high parental involvement.

a) To translate his comment into our own research methods terms:  Imagine a study that compared achievement test scores in a city's charter schools with the city's traditional public schools. Sketch a simple bar graph showing that charter schools score higher in achievement than traditional schools. Does this result mean that charter schools cause better school performance?  What are some internal validity issues here?  Apply the term  "selection bias" or "selection effect," as the author is doing above.

This article also states that

Unfortunately, we don't adjust test scores for differences in factors
such as intelligence, income, parental involvement, or school
facilities. Instead, we use misleading, unadjusted test-score
comparisons. That means some schools that test well might not be doing
all they can, while those with failing scores might be making
outstanding progress. We are largely clueless about what is happening.

In this paragraph, the author seems to be advocating for using multiple regression to evaluate the effectiveness of charter schools. 

b) Why would multiple regression help us rule out the selection bias problem the author mentioned earlier?

c) Sketch a possible table of multiple regression results, as the author suggests above: Assume that achievement test score is the dependent variable of interest. What would be your predictor variables? What results would show that charter schools really are more effective than traditional schools, even controlling for these selection effect variables?  What results would show that charter schools are not more effective than traditional schools?

d) Do you agree that "we are largely clueless" about what is going on? Do you think the regression approach he mentions is a good idea? Are there any downsides to the regression approach?

 

Suggested answers to question c:

Below is a possible table of (fabricated) results that would show that charter schools really are more effective than traditional schools, even controlling for the selection effect variables the author mentioned.

DV: Achievement test scores

Predictor                                                                                    Beta

Traditional vs. Chater (0 = traditional, 1 = charter)       0.15**
Entering achievement test score                                         0.59*
IQ of child                                                                                0.20**
Child's family income                                                            0.11**
Child's parental involvement                                               0.18*
School's level of facilities                                                       0.08

Note: Data are fabricated for teaching purposes.

* represents a statistically significant effect. This table suggests that charter schools are associated with better achievement test scores, even when controlling for earlier scores, IQ of child, income, parental involvement, and school facilities.

 

Now, below is a possible table of (fabricated) results that would show that charter schools are not more effective than traditional schools, even controlling for these selection effect variables