Survey Development and Methodology

Survey Question Development

The total survey consisted of 118 questions. Thirty-three of these questions tested for respondents’ civic knowledge, while the remaining questions secured information on their public philosophy (39 questions), civic behavior (29), and demographics (16). One question tested for knowledge of popular culture. Drs. Ken Dautrich, Richard Brake, and Gary Scott coordinated the development of these questions through a rigorous process of independent consultation, validity analyses, and scholarly review.

Thirteen of the 33 knowledge questions were taken from previous ISI surveys developed by ISI faculty advisors from universities around the country, and who are listed on page 30. Nine of the civic knowledge questions were taken from the U.S. Department of Education’s 12th grade NAEP test, six from the U.S. Naturalization exam. Two new knowledge questions were developed especially for this new survey.

Additional consultation and scholarly review concerning survey question development and final formulation were secured from members of the National Civic Literacy Board appearing on page 31, and members of ISI’s professional staff.

Interview Technique and Sample Size

The research approach that was used to conduct this survey features a national random-digit-dial (RDD) sampling design. A total of 2,508 American adults were included in the sampling. They were interviewed by telephone from April 17 to May 10, 2008. The margin of error for the sample of 2,508 adults is +/- 2.0 percentage points at the 95% level of confidence. When reporting on subgroups of adults (e.g., men, women, college graduates, etc.), the sampling error is higher. The sampling and interview methodology was designed by Dr. Ken Dautrich at the University of Connecticut.

Survey Population

The telephone survey data can be taken to represent a probability sample of all individuals who reside in households with residential telephone service in the United States.

Randomized Sample Selection

The telephone component utilized a Random Digit Dial (RDD) methodology to generate random samples of telephone households in the United States. Within each telephone household, one respondent was randomly selected utilizing the modified Trodahl/Carter in-house selection technique. We asked for the youngest male first, then if not available, the youngest female. This technique removes the control of the person answering the telephone from deciding who participates in the survey.

For this study, Braun Research Incorporated was commissioned to conduct the telephone data collection, utilizing RDD sampling through Survey Sampling International (SSI). Using SSI’s standard RDD methodology, a sample is drawn with a sample size equal to the number of completed interviews.

Weighting

In order to make appropriate projections to the survey population, weight has been applied to the data. Weight represents a compound probability that adjusts a sample to match the population characteristics of the civilian non-institutionalized population of the United States. The weighting process adjusts for error inherent in the sampling methodology. The frame of the general population was aligned to the national population, as taken from the 2006 American Community Survey, and a weight was applied based on age, gender, education, and race.

Participation Question for Identifying Respondents’ Civic and Political Experience

Twenty-nine of the survey questions covered respondents’ civic behavior, ranging from peer discussions, to religious attendance, to political party identity, to media viewing habits. Among these twenty-nine items, thirteen identify respondents’ civic and political activity and experience by answering this question:

Which of the following activities have you participated in at least once in your life (check all that apply):

a.____

Volunteer community service

b.____

Contributed money to a charity

c.____

Active or reserve military

d.____

Registered to vote

e.____

Voted in an election

f.____

Tried to influence how others vote

g.____

Attended political meetings, rallies, fund-raisers, or similar events

h.____

Worked on a political campaign

i.____

Gave money to help a political campaign

j.____

Contacted a public official

k.____

Signed a petition about a political issue

l.____

Published a letter to the editor

m.____

Successfully elected to a government office

Knowing these civic and political participation items help to advance the education research approach of measuring the three traditional effects of formal education interventions: cognitive, participatory, and affective. A prominent study initiated in 1994 by the U.S. Department of Education resulted in the final report, What Democracy Means to Ninth-Graders: U.S. Results from the International IEA Civic Education Study (U.S. Government Printing Office, 2001;http://nces.ed.gov/pubs2001/2001096.pdf). Apart from their civic knowledge and their political opinion and attitude questions, this survey asked students to rate the frequency of their newspaper reading, discussing current events, and their expected future political activities such as voting.

Analyses and Report Writing

The various diagnostics and statistical analyses of the raw data matrix, including statistical inferences based upon multiple regression analyses, were independently conducted and then jointly corroborated by Dr. Ken Dautrich at the University of Connecticut and Dr. Gary Scott at ISI. Dr. Richard Brake of ISI included additional analyses and performed the technical writing, with further assistance with illustrations and layout from Holly Feldheim and Sandy August.

Technique for Identifying the Impact of College and Knowledge On Civic Engagement

Multivariate regression analyses were employed to distinguish the unique impact of college from the impact of additional civic knowledge on respondents’ civic and political engagement. Peter Kennedy defines the identification problem as, “Knowing that something is what you say it is…[it] is a mathematical (as opposed to statistical) problem” (A Guide to Econometrics, The MIT Press, 3rd edition, 1993, 153).

Equation one serves as the theoretical specification for this analytical task of identification:

where:

S_i =

ith respondent’s proportion of seven political activities performed

β_i =

regression parameters (j = 1 … m)

X_i =

on-education characteristics for ith respondent plus civic knowledge score

H_i =

1, if ith person’s terminal degree is a high school degree (otherwise = zero)

B_i =

1, if ith person’s terminal degree is a four-year baccalaureate (otherwise = zero)

M_i =

1, if ith person’s terminal degree is a masters degree (otherwise = zero)

P_i =

1, if ith person’s terminal degree is a Ph.D (otherwise = zero)

µ_i =

stochastic error term for ith observation

Each respondent earns the actual or equivalent of pre-requisite degrees in order to earn the ensuing degree. Each respondent offered only one response in the actual survey mapping to his terminus education level. The following dichotomous variables account for both the person’s terminus degree and necessary preceding degrees.

h_i = 1;

if: H_i = 1 or B_i = 1 or M_i = 1 or P_i = 1; otherwise zero (91% meet criteria)

b_i = 1;

if: B_i = 1 or M_i = 1 or P_i = 1; otherwise zero (25%)

m_i = 1;

if: M_i = 1 or P_i = 1; otherwise zero (8.6%)

p_i = 1;

if: P_i = 1; otherwise zero (1.1%)

Notice that every dichotomous variable equals one for that person possessing a Ph.D., whereas in the previous coding, only the last variable (pi) was equal to one. Replacing these new dichotomous degree variables into equation one results in equation two:

The impact of the baccalaureate degree can now be obtained analytically by taking the first derivative of equation two with respect to bi (baccalaureate), as shown in expression three:

This process allows for the statistical identification of the average impact of the baccalaureate on civic engagement, even from among those respondents possessing an M.A. or Ph.D. The residual and omitted categories of “some high school” or “some college” avoid perfect multi-collinearity, thereby permitting the computing of estimated parameters. The unique and estimated impact of each graduate degree (equation five and six below), as well as a high school diploma (equation four), is now successfully distributed among these three various stages of formal education:

An analogous method was employed to estimate the unique value-added or civic learning arising from the baccalaureate, in the absence of longitudinal data, and to distinguish the impact of the baccalaureate from other influences such as family background, respondent’s individual characteristics, and that of the high school, or graduate degrees. The experiences of being a professor or K–12 teacher were additionally controlled in order to identify the unique and independent influence of the baccalaureate degree and civic knowledge on civic participation. For further reading on the identification problem and the construction of binary variables for applied research, see for example, Introductory Econometrics, by Arthur S. Goldberger, (Harvard University Press, 1998), pp. 121–123.