Print

Field Tests and Pilot Studies

In the following, we will discuss field tests and pilot studies, which are required by many dissertations, considering what they are and how they differ.

Objectives

In this document, we will work on field tests and pilot studies. When you have finished reviewing this document, you will be able to:

• Identify what a field test is and when it is needed,
• Identify what a pilot study is and when it is needed,
• Discuss the differences between them , and
• Prepare for IRB requirements regarding pilot studies

Let's begin.

What is a Field Test?

Many dissertations use researcher-created questionnaires, surveys, interview protocols, and the like. As you know, all data collection tools must be valid and reliable.

Published instruments such as tests or inventories typically have been studied and analyzed carefully, after which their validity and reliability coefficients have been published.

Researcher-created instruments must be reviewed for validity as well. Such a review is called a field test.

What's in a Name?

Many writers use the terms field test and pilot study interchangeably. At Capella, however, the two names mean different things. A field test is an expert panel's review of any researcher-created data collection instrument. These can include:

• Questionnaires.
• Internet surveys.
• Other surveys.
• Interview protocols (guiding or structured questions).
• Other prepared data collection tools.

The emphasis is on data collection tools. If you write a set of questions for an interview, a set of items for a survey of any kind, or any kind of instrument that will be used with participants to collect information, it needs a field test.

Field tests apply to both qualitative and quantitative research.

What is Reviewed?

A field test panel reviews the instrument to determine that the items:

• Have face validity.
• Are readable at the appropriate reading level.
• Are well-constructed and accurately worded to ensure understanding.
• Represent the best way to ask questions that will avoid distressing participants.
• Provide for the safety of the participants.
• Are appropriate to the knowledge level and sophistication of the participants.

Who Makes Up the Field Test Panel?

A field test panel is a group of experts in the topic area who are able to review the items or questions knowledgeably. Many field test panels are outside consultants known in the field for their expertise on the topic. Occasionally, this is the dissertation committee if they have such expertise. An expert is someone who has published, taught, or practiced in the topic area for a reasonable period of time. Field test panels can be any size, but usually three to five.

The process can work in any way that is acceptable to the researcher and the mentor, but ordinarily the panel is:

• Given the document to be reviewed and asked to respond regarding specific issues, which are put in writing by the researcher.
• Their feedback is written and sent to the researcher.
• The researcher revises the document according to the feedback.
• The researcher reports the process in the proposal section discussing instruments or interview protocols.
• The researcher also reports the results of the field test in a document to the IRB.

How does a pilot study differ from a field test?

What is a Pilot Study?

Pilot studies:

• Review any or all elements of the study that affect participants.
• Use a small subgroup of live (real) participants.
• Require IRB review and approval before they can be started.
• Require a total of two IRB reviews: One for the pilot, one for the actual study. In practice, this usually means that the pilot study is included in the full IRB application and the study is approved on condition that the pilot study is conducted and any revisions reported to the IRB. When this happens, the IRB then issues full approval and the actual study begins.

Why and When Conduct a Pilot Study?

Conduct a pilot study to address the areas of concern or uncertainty if, for any reason, a researcher is:

• Unsure if a procedure will work as planned.
• Unsure if participants will be able to do what is asked.
• Unsure if participants can understand test items.
• Unsure of any element of the design affecting participants.

The key is that participants should be involved to resolve the uncertainty.

Consulting experts will not suffice, because real people must undergo the procedure to clear up confusion. For example, experts may be able to identify that test items are readable at a given grade level, but, if we are unsure whether the actual participants will be able to read them, we need to find people like the participants and confirm the items' readability.

How Does a Pilot Study Work?

The researcher:

• Includes a provision for a pilot study in the proposal to the dissertation committee, who approve.
• Includes the pilot study in the IRB Application. The IRB will approve the full study provisionally, on condition that the pilot study is completed and any required revisions made. IRB approves doing the pilot study.
• Recruits a small subset of real participants meeting inclusion criteria for the actual study.

She obtains full informed consent from these participants. Then she conducts the full study (or study of whatever uncertain elements were approved by IRB) with the subset, then notes what works and what doesn't.

Afterwards, she:

• Interviews (or otherwise learns from) the participants about the experience, paying particular attention to their reports of difficulties.
• Revises the procedures based on the participants' experience.
• Submits a Modification Form listing the revisions to the IRB for their final approval. Only when this final approval is received can the full study be carried out.

Summing Up

Learner-generated interview questions, survey questions, or other information collection tools without validity data must be field-tested; no IRB approval is needed. Here are questions to ask:

• Is this about the tool itself?
• Is this only about the validity or quality of the tool?

Conduct a field test to validate the tool.

Any participant-facing procedure calling for a "live" test requires a pilot study. IRB approval is required. Here are some questions to ask:

• Is this about how something works with the participants in any way?
• Is it about some procedure's effect on participants?
• Is it about whether the participants can do something I need them to do?
• Is it about whether some procedure will actually accomplish what it's designed to do?

Conduct a pilot study to address the areas of concern or uncertainty.

---

Doc. reference: phd_t3_u03s2_testpilot.html