2.2 Data collection techniques

Scientific norms and rules govern how to collect data. Text books and professional guides detail how each of these techniques should be used. The respective strengths and weaknesses of each technique selected must be understood and recognized.

Experimental

Experiment is a technique used to test for causation or association between X (independent or explanatory) and Y (dependent or response) variables.

Used mostly in the field of experimental psychology, it is considered one of the most intrusive and reactive techniques.

There are dozens of experimental designs. The classic design is a randomized comparative experiment in a highly controlled setting. It typically involves randomly assigning subjects to groups, usually an experimental group and a control group, to measure for differential reactions to an administered treatment. A pre-test/post-test design is also classic.

Only repeated trials of the most rigorous and controlled experiments can provide standard proof of a cause-effect relationship. It took over 60 years and 300+ experiments to finally “prove” that cigarette smoking “causes” lung cancer. For this reason, do not assume that any single experiment can prove a cause-effect relationship.

There are also quasi-experimental techniques that are less rigorous and controlled than so-called true experiments. They are not designed to determine statistical significance or causation. Instead, quasi-experiments are used to better understand changes as a response to controlled/uncontrollable treatments, treatments that cannot be prearranged. Of the many possible designs, one popular one is the natural experiment, in which the researcher measures changes in a post-hoc (after-the-fact) test for differences. Childhood malnutrition is a typical example of when a treatment (i.e., imposed starvation) cannot be administered and then tested.

Quantitative survey

Surveys are generic for a range of techniques for questioning people: ranging from questioning a large number of people, as in a census or poll, to unstructured one-on-one interviews.

Quantitative surveys gather aggregate data on opinions, beliefs, habits and knowledge, using a questionnaire. Since the aim is to obtain a large quantity of data that can be uniformly input using software, forced-response categories are the norm. The data are then usually grouped into tables and graphs and number summaries. Statistical analyses such as chi-square, correlation coefficients and t-tests can also be conducted on data drawn using a quantitative survey technique.

Focus group

A focus group is essentially a group interview of approximately 5 to 10 people. It is one of several survey method techniques for gathering data by questioning people. Quantitative or qualitative data can be derived from this technique. Popular with marketing and polling firms, it can be used to ascertain the needs and concerns of consumers with regard to a trial product or a new political candidate or policy.

Qualitative interview

Conducted mostly with open-ended questions and probes in a semi- or unstructured way, it aims to produce detailed explanations and rich descriptions, usually from a small number of individuals. The responses are usually transcribed from a recorded tape of the interview into a verbatim (word-for-word) written transcript. It is one of several survey method techniques for gathering data by questioning people. Qualitative interviews are often used in conjunction with other techniques such as case studies. They provide descriptive accounts that are rich in detail and particular to the person being surveyed.

Non-participant observation

This technique involves unobtrusive observation in a natural setting. Since the research subjects have no knowledge of being observed, there is no interviewer bias. Ethical rules restrict such observations from taking place anywhere other than open, well-populated public places.

Field research

Field research or fieldwork is a very involved, multi-method design. The researcher becomes immersed in a natural field setting over a period of time and uses any number of data gathering techniques, ranging from field interviews to unobtrusive observation of behaviour. The written account of the research is often referred to as ethnography, a rich descriptive account of the field experiences typically incorporating photographic images and diagrams, maps, stories and so on.

One famous field study involved pool room hustling and was conducted by Ned Polsky in the 60s. Most commonly used in anthropology, it is also widely used in sociology and non-experimental psychology.

Historical-comparative

 “Doing history” generally involves the analysis of a collection of primary and/or secondary data relative to particular periods, people or events in the past. Historical data can be quantitative, as in census data, for instance, or qualitative, as in diaries. Historical quantitative data can be analyzed by either tracking trends or comparing cases over time or by identifying or associating distinct patterns in aggregate data sets.

It also involves appreciating historical data for what it offers: often incomplete, selective artifacts or accounts from which very limited inferences can be drawn. It requires an evaluation of the previous research questions and the tentative answers generated, because knowing the source of the information and understanding how to evaluate its veracity is as important as the content of the data itself. As humble and limited as the analysis appears, historical analysis demands careful and guarded inferences, from the specific to the general, and an appreciation of the multiple factors involved in analyzing history. Comparing one period to another or one place to another provides some weight to the analysis.

Case study

The case study technique involves focusing on a particular individual, organization or group, such as a corporation, non-profit agency or religious cult, to name a few. The goal of a case study is limited to descriptively detailing how a particular case operates or develops within the specific parameters of the environment. A number of techniques are typically used to collect the data/information, ranging from available data and information to interviews with key stakeholders or informal field studies involving attendance at events, ceremonies or meetings.

Physical traces

“Physical traces” is a roundabout way of gathering data about human behaviour. Instead of asking people to tell you what they do or how they do it, it takes note of a collection (accretion) or erosion of materials as “readings” of actual behaviour. This is particularly valuable when measuring behaviour that is not reliably measured using the survey technique. Survey respondents are prone to provide unreliable responses to socially displeasing behaviours, such as poor eating habits or smoking and drinking. Instead of asking respondents whether they make healthy eating choices, an analysis of the contents of the garbage might register more reliable findings. 

Content analysis

Content analysis refers to the systematic analysis of communications, whatever form they may take, whether in print form, such as news articles or advertising, music form, such as lyrics, or electronic form, such as Twitter feeds. Categorized as an unobtrusive, non-reactive way of gathering data, this is very popular among student researchers interested in understanding media messages.

Measures are usually focused on identifying techniques of persuasion or cultural characteristics, message traits, inherent bias or defining features of the particular “communication community” being analyzed. Content analysis has been instructive in studying bias in news media articles with regard to such controversial subjects as the Middle East conflict or global warming, or in revealing the political positions in pamphlets of little-known groups such as sex workers.

Though this is called a type of analysis, which it is to a certain extent, it is categorized as a method or technique. It is difficult to separate the technique of obtaining the data from the actual process of analyzing it.

Document analysis

This involves interpreting documents such as court rulings or letters in order to gain insight into the thought processes or ideas surrounding a particular issue, person or event. The “reading” of the document involves a series of critical questions such as the purpose, audience, authenticity and significance of the author and the document. Code sheets for categorizing the data derived from a careful “reading” are sometimes designed for this technique. It is used mainly by historians using primary sources and is often used in conjunction with other data collection techniques.

Though this is called a type of analysis, which it is to a certain extent, it is categorized as a method or technique. It is difficult to separate the technique of obtaining the data from the actual process of analyzing it.

Secondary analysis

Referred to as “desk research” and popular in college and university student circles, it involves using the primary research of others found mostly in published books, peer-reviewed journals and monographs to address a research question. It is also a technique that few studies can do without. Though it can stand alone as a single technique, it is regularly used to provide context and corroboration for almost every other technique. For instance, a case study or a document analysis requires the input of secondary analysis to contextualize or situate the case or document.

Be sure to read up on its strengths and weaknesses.

Though this is called a type of analysis, which it is to a certain extent, it is categorized as a method or technique. It is difficult to separate the technique of obtaining the data from the actual process of analyzing it.

Meta-analysis

Also known systematic review or best-evidence synthesis and very closely related to secondary analysis, this technique involves pooling a collection of high-quality published studies and systematically reviewing the combined evidence. The aim is to identify patterns, problems or inconsistencies regarding the state of knowledge in a particular area.

Though this is called a type of analysis, which it is to a certain extent, it is categorized as a method or technique. It is difficult to separate the technique of obtaining the data from the actual process of analyzing it.