Language Learning & Technology
Vol. 5, No. 2, May 2001, pp. 34-37
REVIEW OF TEST PILOT
Paginated PDF version
Hardware and software requirements
Institution: A Web server (Test Pilot is compatible with most standard server software; a list is available online); 100% Java compliant Virtual Machine.Hosting services are available for a monthly rate of $25 (educational users) or $100 (commercial users) for up to 20 megabytes total storage.
Test takers: Internet connection and a standard Web browser.
200 users, $500; 500 users, $1000; unlimited number of users, $6000 (all listed prices are for government and academic users; business users pay more)
Unlimited for one month; 12 calls per year; unlimited support available for $300 a year
Educational institutions and businesses
Reviewed by Charlene Polio, Michigan State University
Test Pilot is an application that allows users to author tests or surveys and administer them from a Web server and is similar to products such as Question Mark and Examiner. None of these applications is designed specifically for language testing, and, in fact, none includes any examples of language tests among the various sample tests available to potential customers on their Web sites. Although all of these products' Web sites allow users to download or use free trial versions, ClearLearning has the most informative site and is the only one that includes specific price information.
Brown (1997) discusses the pros and cons of using computers for assessment. To summarize, the pros include faster scoring, easier generation of reports, uniform administration in a variety of locations at a variety of times, item-banking capabilities, and computer-adaptive testing capabilities. Among the disadvantages are the lack of availability and reliability of computers, lack of availability and reliability of Internet connections, small screen size, questionable test security (if done at a variety of sites), and reliability and validity issues (e.g., test-taker familiarity with computers, differences in completing a task on a computer screen as opposed to on paper).
This review is based on the online trial version of Test Pilot and information available on ClearLearning's Web site. There is a version that allows offline authoring which was not evaluated because, according to the Web site, it lacks some of the features of the online version. This review will attempt to highlight the most important features and information, but potential customers are encouraged to go to the ClearLearning Web site and try the product themselves.
Among the attractive features of Test Pilot are (a) that it requires no programming skills or knowledge of HTML, (b) the authoring can be done online or offline, (c) test takers need only an Internet connection and a Web browser, and (d) it supports multimedia (particularly important for language testing). Oddly,
ClearLearning does not highlight the fact that one can implement computer-adaptive testing (i.e., tailoring the test to individual test-takers based on their responses) with their product.
The authoring manual is available online without having to access the trial version or buying the product. From this manual and the examples available on the Web site, one can see the range of question types that can be created. It seems all major question types used in language tests are included. For example, the product allows for the creation of multiple choice, fill-in-the-blank, essay, and matching questions. The steps that one goes through to actually create the test are shown in Figure 1.
Figure 1. Steps in test creation
Although data is stored on ClearLearning's Web site with the trial version of Test Pilot, the normal installation of the full version of Test Pilot requires institutions to have a Web server on which to store tests and surveys. However, ClearLearning does offer hosting services for a monthly fee to those who do not want to or cannot manage their own server.
Test creators can either work online, using a Web browser (version 4.x or better), or work offline using the stand-alone database that runs on both Macintosh and Windows platforms. The test takers need only an Internet connection and a standard Web browser. However, if the test creator uses multimedia plug-ins, the test-taker will need to have those as well.
There is no doubt that Test Pilot is a useful product and allows for all the advantages of computer-based testing; it can be used for computer-assisted testing, item banking, and computer-adaptive testing. It appears that the test creator has control over everything that may be important: the format, question type, and choice of questions. In Figure 2, one can see some of the choices that the test creator can make when authoring, for example, multiple-choice questions. Some of the page layout and feedback options are shown in Figure 3.
Figure 2. Creating a multiple choice question.
Figure 3. Page layout options
In addition to control over the tests and surveys themselves, information on responses can be reported in most ways that an administrator or researcher may need. Data can viewed by user or by question (with the relevant item and reliability statistics). For examples of how assessment data can be displayed, see http://www.clearLearning.com/using/admin/viewing.html. Furthermore, the data can be downloaded to a spreadsheet for additional analysis.
The major problem with this product lies not in the technical details, but in the cost. For government and academic users, the cost is $500 for 200 users (i.e., test or survey takers) and $1000 for 500 users. An unlimited number of users would cost $6000. One has to compare this price to the cost (opportunity or actual) for someone to program a test of this sort to put online. Furthermore, this product is not cost effective for teachers who want to create classroom activities when free products such as Hot Potatoes (reviewed this issue) are available for this purpose.
Free support consists of unlimited calls for only the first month, and, following that, users are allowed 12 calls per year. Unlimited support is available for $300 a year. I tried to call the support number listed, I did not get through for about 30 minutes. This is definitely a cause for concern for a product's support service. E-mail, on the other hand, was answered instantaneously on three occasions. The company representative who finally did answer the phone was very helpful and directed me to a Spanish test not linked to the Web site. It was useful to see a completed language test, and anyone considering creating a language test that includes listening items might want to contact the company to view language test demonstrations.
Aside from cost, other issues in evaluating Test Pilot are the reliability of the product and ease of use for someone without technical expertise. Without constructing a full test and piloting that test, it is difficult to know how bug-free the software is. Because there are many sample tests and testimonials from various clients, one might be able to contact those people who have used this product on this matter.
I tested the ease of use of this product by trying to construct placement tests, an area in which I have experience, with my very limited technical expertise. On one hand, I found that I could create a simple multiple-choice grammar test very quickly. On the other, to create a language test like the sample Spanish placement test would have taken me a long time and would have likely not been successfully completed without consulting technical support staff. My suggestion is that anyone with limited technical expertise get online and try to create a sample test to see whether the product falls within their range of expertise.
One final note, Test Pilot won't work well for languages using non-Roman character sets. It is possible, however, for one to save a Japanese reading passage, for example, as an image and incorporate that image into the test. This may be a concern if an institution wants to use one product for language tests in several different languages that do not use Roman character sets.
ABOUT THE REVIEWER
Charlene Polio is an associate professor at Michigan State University where she directs the MA TESOL Program and gives workshops for foreign language teachers. She has published research on second language acquisition, classroom discourse, and L2 writing. Her current research focuses on second language writing research methodology.
Brown, J. D. (1997). Computers in language testing: Present research and future directions. Language Learning and Technology, 1(1), 44-59. Retrieved from the World Wide Web April 6, 2001: http://llt.msu.edu/vol1num1/brown/default.html.