Survey Development and Methodology

Survey Question Development

The survey contained 118 questions, including thirty-three testing the respondents’ civic knowledge, thirty-nine gauging their public philosophy, twenty-nine measuring their civic behavior, and sixteen on demographics. One tested knowledge of popular culture. Drs. Ken Dautrich, Richard Brake, and Gary Scott coordinated development of the questions through a rigorous process of independent consultation, validity analyses, and scholarly review.

Thirteen of the thirty-three civic knowledge questions were taken from previous ISI surveys developed by ISI faculty advisors from universities around the country. Nine were taken from the U.S. Department of Education’s 12th grade NAEP test and six from the U.S. Naturalization exam. Two new civic knowledge questions were developed by ISI especially for this survey.

Interview Technique and Sample Size

2,508 American adults were interviewed by telephone from April 17 to May 10, 2008. The margin of error for the full sample is +/- 2.0 percentage points at the 95% level of confidence. The margin of error for subgroups (e.g., men, women, college graduates, etc.) is higher. The sampling and interview methodology was designed by Dr. Ken Dautrich of the University of Connecticut.

Survey Population

The telephone survey data can be taken to represent a probability sample of all individuals who live in households with residential telephone service in the United States.

Randomized Sample Selection

A Random Digit Dial (RDD) methodology was used to generate random samples of telephone households in the United States. Within each household, one respondent was randomly selected utilizing the modified Trodahl/Carter in-house selection technique. Braun Research was commissioned to conduct the telephone data collection.

Weighting

A standard weighting process was applied to the data to adjust for error inherent in the sampling methodology. The frame of the general population was aligned to the national population, as taken from the 2006 American Community Survey, and a weight was applied based on age, gender, education, and race.

Opinion Questions for Identifying Respondents’ Public Philosophy

The thirty-nine public philosophy propositions were developed for this survey to empirically test for possible relationships between a person’s knowledge of America’s history and institutions, opinions on public affairs, and civic participation. Propositions were kept succinct to maximize the questions that could be asked in a telephone interview. ISI researchers refined and validated the propositions with assistance from a panel of scholars with advanced training in various disciplines. A focus group of non-scholars was used to identify unnecessarily vague or problematic language in the phrasing of propositions.

Analyses and Report Writing

Analyses of the raw data matrix, including statistical inferences based upon multiple regression analyses, were independently conducted and then jointly corroborated by Dr. Ken Dautrich at the University of Connecticut and Dr. Gary Scott at ISI. ISI visiting fellow Terence Jeffrey provided the technical writing for this report, along with additional analyses and writing from Dr. Richard Brake and Patrick Ford.

Technique for Identifying the Impact of College and Civic Knowledge on Opinion

Multivariate regression analyses were employed to distinguish the unique impact of college from the impact of additional civic knowledge on respondents’ opinions. Peter Kennedy has written, “Knowing that something is what you say it is… is a mathematical (as opposed to statistical) problem” (A Guide to Econometrics, MIT Press, 1993, p. 153). Equation one serves as the theoretical specification for this analytical task of identification:

(1) S_i = β₁ + β₂X_i + β₃H_i + β₄B_i + β₅M_i + β₆P_i + µ_i

where:

S_i = ith respondent’s rating or scoring of an opinion proposition

β_i = regression parameters (j = 1 ….m)

X_i = non-education characteristics for ith respondent

H_i = 1, if ith person’s terminal degree is a high school degree (otherwise = zero)

B_i = 1, if ith person’s terminal degree is a four-year baccalaureate (otherwise = zero)

M_i = 1, if ith person’s terminal degree is a masters degree (otherwise = zero)

P_i = 1, if ith person’s terminal degree is a Ph.D (otherwise = zero)

µ_i = stochastic error term for ith observation

Each respondent earns the actual or equivalent of pre-requisite degrees in order to earn the ensuing degree. Each respondent offered only one response in the actual survey mapping to his terminus education level. The following dichotomous variables account for both the person’s terminus degree and necessary or implicit preceding degrees.

h_i = 1; if: H_i = 1 or B_i = 1 or M_i = 1 or P_i = 1; otherwise zero (91% meet criteria)

b_i = 1; if: B_i = 1 or M_i = 1 or P_i = 1; otherwise zero (25%)

m_i = 1; if: M_i = 1 or P_i = 1; otherwise zero (8.6%)

p_i = 1; if: P_i = 1; otherwise zero (1.1%)

Notice that every dichotomous variable equals one for that person possessing a Ph.D., whereas in the previous coding, only the last variable (p_i) was equal to one. Replacing these new dichotomous degree variables into equation one results in equation two:

(2) S_i = β₁ + β₂X_i + β₃h_i + β₄b_i + β₅m_i + β₆p_i + µ_i

The impact of the baccalaureate degree can now be obtained analytically by taking the first derivative of equation two with respect to b_i (baccalaureate), as shown in expression three:

(3) dS_i/db_i = β₄

This method allows for the statistical identification of the average impact of the baccalaureate on each public opinion item, even from among those respondents possessing an MA or Ph.D. The residual and omitted category of “some high school or less” avoids perfect multi-collinearity, thereby permitting the computing of estimated parameters. The impact of high school and graduate degrees is similarly obtained:

(4) dS_i/dh_i = β₃

(5) dS_i/dm_i = β₅

(6) dS_i/dp_i = β₆

An analogous method was employed to distinguish the impact of the baccalaureate on civic knowledge from other independent variables. Arthur Goldberger further elaborates on the identification method and construction of binary variables in Introductory Econometrics (1998).