Survey Research in Education
Survey research aims to provide a comprehensive, representative summary of specific characteristics, beliefs, attitudes, opinions, or behavior patterns of a population. There are a number of different methods of collecting survey information, such as inperson, phone, and email questionnaires and interviews. Surveys are not simply exhaustive collections of statistics about specific traits of a population. Surveys are always conducted in response to particular research questions, generally qualitative in nature, and thus aim to collect only information that might be relevant to the study at hand. Survey study design includes the articulation of the research question, of the scope of the study, and of the targeted population, the subdivision into themes or sub-questions of the original problem, the selection of survey items, pilot testing, and finally, administration, collection of information, and data analysis. Survey research methodology is the most commonly used approach in the social sciences, accounting for about 70% of all studies.
ACADEMIC TOPIC OVERVIEWS
Research in Education: Survey Research
Introduction to Survey Research
Survey research aims to provide a comprehensive, representative summary of specific characteristics, beliefs, attitudes, opinions, or behavior patterns of a population. There are a number of different methods of collecting survey information, such as in-person, phone, and e-mail questionnaires and interviews. Surveys have been used for over two thousand years (Di Iorio, 2005); the first recorded use was around 150 BCE, when Hipparchus surveyed the night sky and recorded the positions of stars as well as their relative brightness as measured by a six-point scale. Today, surveys may be found in all aspects of public life: they are used to collect information about voting practices and political preferences, about satisfaction with a particular product or experience, and about personal habits and behaviors, to name but a few. Survey research methodology is the most commonly used approach in the social sciences, accounting for about 70% of all studies (Lodico et al, 2006).
Surveys are not simply exhaustive collections of statistics about specific traits of a population. Surveys are always conducted in response to particular research questions, generally qualitative in nature, and thus aim to collect only information that might be relevant to the study at hand. In educational research, surveys have been used to gather information on test scores in order to identify patterns of low achievement, to form impressions of new teachers' attitudes toward teaching, and to identify trends in student interests. Mellard, Patterson, and Prewett (2007) used survey research to collect information about adult students' reading patterns and about students' traits in order to identify ways to encourage adults to read. Richardson, Slater, and Wilson (2007) used survey research to collect information on university students in the UK.
Use in Quantitative Research
Surveys, though primarily quantitative in design and implementation, are always developed in response to a qualitative inquiry, unlike other forms of quantitative research (Lodico et al, 2006). Surveys are not experimental, so data is not collected to test a hypothesis, but rather, to describe--both qualitatively and quantitatively, but primarily quantitatively--existing conditions and attitudes. Surveys can take the form of questionnaires or of interviews, though neither format implies specific types of questions: interviews might require survey participants to choose their answers between previously determined categories, while questionnaires may ask for open ended responses. Di Iorio (2005) suggests using open ended questions for gathering information of a personal or sensitive nature, and multiple choice questions for easily quantifiable information. Sometimes both questionnaires and interviews are used within a single survey study. For example, Mamlock-Naamam et al (2007) evaluated a workshop designed to encourage science teachers to create their own curricula through questionnaires--to get a rough quantitative approximation to teacher beliefs--as well as through interviews--to gauge teachers' initial reactions and to probe quantitative answers in more detail.
Designing a Survey
The first step in the design of a survey study is the articulation of the research question, of the scope of the study, and of the targeted population. Next, the research question is subdivided into themes or sub-questions that reflect different dimensions of the problem. Survey items are then formulated and selected using scaling, a method of generating questions and statements that adequately measure--quantitatively--the qualitative aspects of the attitude or belief under investigation.
After survey items are selected, and after the questionnaire or interview format are completed, a survey must undergo pilot testing before it can be used to collect and generalize information about the targeted population.
Because surveys answer qualitative questions through an analysis of correlations between quantitative measurements, scaling--the process of transforming qualitative statements into meaningful quantitative measures--is critical to the design of a survey study. Surveys may employ open-ended questions; however, these are rarely used outside of allowances for "additional" or "other" comments at the end, as these types of responses are not easily processed or analyzed (Di Iorio, 2005).
Scaling is not simply a process of assigning numerical values to qualitative statements, such as a score of 5 for "highly agree" or of 1 for "highly disagree." Scaling is the intricate, nuanced technique of developing qualitative statements that gauge respondents' beliefs about a particular issue--thus scaling research always begins with the clear identification of a research question or aim. Experimental research is then conducted to determine appropriate qualitative statements for use on surveys that adequately gauge the desired features in the population of interest. There are several formalized conceptual and experimental frameworks commonly used in scaling research; the most relevant to educational researchers are:
* Thurstone Scaling,
* Likert Scaling, and
* Guttman Scaling.
All of these methods are unidimensional--they only measure one aspect, or dimension, of a variable. For example, if the study aims to gauge the attitudes of various constituencies in a particular school district toward school choice initiatives, a unidimensional scale might be used to measure attitudes as (very and somewhat) favorable or unfavorable (Lodico et al, 2006).
Louis Leon Thurstone (1887-1955) was a pioneer in psychometrics, the discipline concerned with measurement of psychological traits such as intelligence, personality, values, and attitudes. Thurstone developed three scaling methods: those of equalappearing, paired comparisons, and successive intervals. While Thurstone scaling is rare today (Di Iorio, 2005), it presented a revolutionary breakthrough in measurement in the early 1900s and forms the foundation for thinking about all other methods of scaling (Allen, 1994).
After the identification of the research question or aim, Thurstone scaling calls for the generation of statements that gauge beliefs about the content matter of the study; a large pool, of about 70+ statements, is appropriate. The questions generated should be suited in tone, language, and content for the targeted population, and should be similar in structure (Lodico et al, 2006).
A team of experts in the subject matter of the survey should then be called upon to rate these items on a scale that may have any number of "points"--a five point scale, for example, would allow one to rate a statement on a scale from 1 to 5. A Thurstone scale may have an odd or even number of points; an odd number of points allows for a "neutral" answer, while an even number of points does not (see Figure 1).
At this phase of the scaling, the experts do not rate the items to reflect their own opinions, but on how much each statement reflects a favorable or unfavorable attitude. For example, in a study to gauge attitudes on school choice, generated statements might include "School vouchers would give each child an opportunity to find an environment s/he can best learn in" and "School choice would lead to large discrepancies between those who have the resources to locate, apply to, and travel to another school and those who do not." The first of these items would be rated as favorable (very, moderately, somewhat), while the second would be rated as unfavorable.
After each item has been rated by several content-matter experts, scores of all statements are tallied, tabulated, and ordered. The score of a statement is composed of three parts: the median and the first and third quartile. The median is the rating below which 50% of responses and above which 50% of responses fall. The first quartile is the rating below which 25% of responses fall, while the third quartile is the rating below which 75% fall. For each item, the difference between the inter-quartile range, or the difference between the first and third quartile ratings, is computed (see Figure 2).
Statement School vouchers would give each child an opportunity to find an environment s/he can best learn in. Ratings 3 4 4 4 5 5 5 5 First Quartile 3 4 4 4 5 5 5 5 25% of scores fall below the value 4 Median 3 4 4 4 5 5 5 5 50% of scores fall below 5; 50% of scores fall above 4; therefore, the average between the two gives the median: 4.5 Second Quartile 3 4 4 4 5 5 5 5 75% of scores fall below the value 5
Thurstone scoring for the statement "School vouchers give each child an opportunity to find an environment s/he can best learn in." If there are 8 judges, and they rate this statement as shown above, the statement will have a score of: 4 for the first quartile, 4.5 for the median, 5 for the second quartile, and 1 for the interquartile range.
Statements for the survey are then chosen; items picked should equally represent all medians, so that the survey consists of an equal number of statements that reflect favorable and unfavorable attitudes. When electing between items with the same median score, the ones with the smallest inter-quartile range should be chosen, as these reflect statements that have the least variability between respondents and are thus the most reliable.
The Thurstone method described above is that of equal-appearing scaling, however Thurstone also proposed others that are commonly used in the social sciences: paired comparison and successive interval scaling. In the previous example, the subject matter experts rate each item independently (hence, each is "equal-appearing"), while in paired comparison (Allen, 1994) and successive interval (Adams & Messick, 1958) scaling items are compared with each other before...
(The entire section is 4915 words.)