How NOT to Use Instructional Computers
R.O. Walker
Police Training Institute
University of Illinois at Urbana-ChampaignJ. Janssen
Illinois Law Enforcement Training and Standards BoardR.A. Avner
University of Illinois at Urbana-Champaign(Prepared for delivery at the Meeting of the Academy of Criminal Justice Sciences in
Orlando, Florida, March 11, 1999)
Abstract
Experience during the past 25 years of Computer Based Education (CBE) has amply demonstrated the difficulty of finding techniques guaranteed to produce effective instructional material for the medium. On the other hand, new CBE authors appear to have no difficulty in rediscovering, at substantial cost to their progress, a number of approaches which virtually insure production of poor material. This paper: (1) states ten such techniques that have been thoroughly tested (and found wanting); and (2) gives answers to important questions about CBE asked of the authors on the Internet during the past year at: http://www.staff.uiuc.edu/~royw.
Paper Begins
Experience during the last 25 years of Computer Based Education (CBE) has amply demonstrated the difficulty of finding techniques guaranteed to produce effective instructional material for this medium. On the other hand, new CBE authors appear to have no difficulty in rediscovering, at substantial cost to their progress, a number of approaches which virtually insure production of poor material. One of the authors of this paper described five of these pitfalls many years ago (Avner, 1974). We now advance the state of this dismal art by adding another five pitfalls. The ten self-destructive techniques described below have all been thoroughly tested. All have been found thoroughly wanting.
1. Art for the Sake of Art. The watchword here is "make full use of the medium." From your title frame (at least three minutes of dazzling op-art), your major objective should be the entertainment of yourself and your audience through a virtuoso display of your command of the CBE equipment and language. Games of skill and mind-bending graphic displays are especially appropriate. Assume that anything that can be done by CBE is automatically worth doing. Moreover, assume that anything that attracts attention should be programmed.
2. Do Your Thing. Impress yourself and your colleagues by applying CBE to your current research interest. The application should be as abstruse as possible. Ideally, the topic and presentation should leave all but your most able colleagues with a feeling of personal inadequacy. Above all, do not limit yourself by the needs, interests or abilities of prospective students. In fact, do not even think about where the material might be used until after it is completed. Research shows (Taylor, 1970) that the degree to which the material impresses your peers will generally be inversely related to its teaching value.
3. The Procrustean Bed. Assume that all instructional media are, or should be, about the same. Find a convenient programmed text or filmstrip and make an exact translation of it for CBE. If you feel more creative, produce new material while strictly adhering to the limitations of some other medium and ignoring any unique capabilities of CBE. Your objective is material demonstrating all of the disadvantages of both media and none of the advantages of either.
The Procrustean approach is a deserved favorite of graduate students needing a quick thesis topic. Few readers of the thesis are likely to note that the "well-controlled research design" has eliminated most major differences in the experimental treatments. Thus, by studying a new medium the degree candidate satisfies the interests of the more innovative members of his committee and, by finding no significant improvement over other media, he satisfies the biases of the nonbelievers.
4. "Damn the Torpedoes." Pick a good solid goal (say, creating a one-semester course on CBE) and do not let anything distract you until it is reached. Do not worry about possible alternative approaches. Any competent teacher ought to be able to write the final version on the first try. Do not waste time trying preliminary versions on students. After all, if they knew what they needed, they would not be students. Do not show your material to other teachers. They will only suggest unneeded frills. Vigorously defend the finished product. Assume that any comments other than words of awed admiration are intended as slurs on your professional ability and ancestry. Eventually everyone will see things your way (you can tell by the fact that they will stop making suggestions). You may be surprised at the stupidity of some of your students who will be unable to comprehend the most obvious (though perhaps not explicitly stated) information. Unfortunately, large numbers of students may never see cunningly devised remedial sequences designed at great cost in time and effort to aid deficiencies that no one seems to have. Other students will no doubt complain bitterly about being bored by having to plow through material that they claim they already know or could have learned faster on their own. Accept these problems as normal drawbacks to use of programmed instructional approaches. You may be assured that you are using one of the most efficient techniques for production of ineffective material known (Markle, 1967).
5. Hitch a Ride on a Bandwagon. You just cannot go wrong by keeping up with the latest trends in instructional design. Do not worry too much about how the current fashion should be implemented or what possible limitations it might have. Ignore any relationships with more established techniques or your own experience. The important thing is to use the right terminology and those features of the technique that are most convenient to apply. If you are either wildly successful or unsuccessful you will have an automatic and easy publication to your credit. If you have been careful to be rather casual in your interpretation of the technique, you should also be assured of many citations to your article by those writing rejoinders. Even if you find no significant effects, your cocktail party presence will be enhanced by the fact that you will be considered a worker on the frontiers of science. Most important, your students will have had the opportunity to serve as a part of the Noble Effort to Further Mankind's Knowledge. Ignore those students who complain that they were more interested in furthering their own knowledge.
6. Promise to have the system serving a thousand students by next month. Your manager or funding agency will be impressed by your positive attitude, and your name will be mentioned with reverence for at least a few weeks. Skilled practitioners can stretch their moment of glory out for several months beyond the promised deadline by the simple expedient of writing lots of memos claiming that you would be right on schedule if it were not for unexpected problems with (pick one or more of the following): (1) hardware delivery, (2) hardware design, (3) hardware reliability, (4) staff cuts, (5) staff increases, (6) need for more staff, (7) software delivery, (8) software design, (9) software reliability.
7. Budget only for hardware. Everyone knows that new technology is really only hardware. Support this view by asking only for enough to get that hardware into your building. Naturally, once the hardware is there it will become evident that a great deal of added effort will be needed before it can be used effectively. When your ungrateful colleagues decline the opportunity to leap to your rescue by volunteering to do that work in their spare time, begin writing memos indicating that the hardware has arrived on time, but that a lack of vision on the part of your co-workers is hindering its glorious application (see also 6. above).
8. Assume the great majority of the CBE work is done by computer programmers. If they call it Computer Based Education, then computers are obviously the most important component. And who knows more about computers than computer programmers? Why complicate life by trying to manage a team of subject-matter experts, instructional design professionals, graphics designers, and evaluation specialists in addition to computer programmers? In fact, everyone knows that any 13-year-old knows all about computers and computer programming. If it were not for those outrageous child labor laws, we could get everything done by kids at a couple dollars per hour. And the resulting materials would be good for students. After all, in real life you have to learn from sources of information that are often inaccurate and presented in a fashion that is sometimes incomprehensible. Why should students have it easier?
Some Conclusions Based on Items One Through Eight. A few common threads run through most of the above techniques. One is a lack of appropriate attention to the capabilities of CBE for instruction. Either ignoring these capabilities or making them ends in themselves is a mistake.
A second error is a lack of attention to the current status and future needs of students. The needs of different students are not likely to be met by material which ignores differences or bases individualization solely on "educated" guesses. In the absence of reliable prescriptions for effective instruction, we must rely heavily on empirical cut-and-try approaches. Fortunately, CBE provides several features (ease in data gathering and revision of material) which aid such approaches. Unless maximum use is made of these special features as well as those features which make CBE attractive as a medium, CBE will continue to be simply a rather expensive experimental technique.
A third error is in assuming that the hardware is what is really important in educational technology. Hardware is a great tool, but it works in education only when supported by a lot of skilled human effort. The real teaching is not done by computer or even by people who know only how to talk to computers, but rather by humans skilled in the nuances of the subject matter being taught, and in the techniques of communicating those nuances to other humans. In the development and implementation of CBE, you probably already plan milestones for all major events and assign projected dates for their completion, but underestimating the amount of human effort required and voicing or declaring too much confidence in the date on which a future milestone will be achieved will almost certainly result in a loss of credibility or in termination of support. Though it will be difficult to budget for the human resources required to implement CBE, those needs are real and must be estimated if an implementation is to succeed. In a common practical situation, where funding is limited, consider estimating the amount of money available and work backwards from that figure to a level of specificity of deliverables acceptable to your institution. Rather than attempting to gain approval for new funds down the road, it may also be easier to change funds from one category to another when the CBE system becomes operational.
A final similarity among all of the poor approaches described above is the fact that all are used to some extent by virtually every CBE author. It will always be difficult for humans to avoid being entranced by a new gadget, to avoid testing the limits of their own abilities, to avoid (particularly under time pressures) using familiar rather than unfamiliar approaches and snap judgments rather than time-consuming alternatives. Nevertheless, they all are poor approaches and wide use or the weaknesses of the species do not alter that fact.
Topics Generated Via the Author's Faculty Internet Hot-Line
As over 20,000 of our graduates can tell you, we are convinced that computers have a vital role to play in cost-effective criminal justice training. Our current pioneering state of the audience microcomputer lessons in Appendix A and arrangement of those lessons by module in Appendix B are based on instructional techniques proven effective in training over 20,000 sworn police officers of city, town, county, state and federal agencies.
Over the years some solutions to important education and training problems have been developed by the authors and staff associates of PTI and the Illinois Law Enforcement Training and Standards Board. For almost three years, we have been responding to questions about CBE on the Internet (http://www.staff.uiuc.edu/~royw) that could be answered based on the author's collective experience.
If you have heeded the sage advice in items one through eight as presented in this paper and implemented it carefully in your program, you are well on the way to ensuring the demise of your CBE effort, but Internet inquiries to the authors suggest two more approaches to disaster.
9. Computer teaching lessons should be treated just like books. It is not appropriate to view computer lessons as books. They are more like individual tutors. By bringing the Internet and World Wide Web (WWW) into the picture as first introduced in the Police Training Institute at the University of Illinois by two of the authors (R.W. and A.A.) in May 1996 (Walker, Janssen, & Walker, 1997), we can involve the benefits of currency, branching, tailored feedback, required responses, repetition, and simulation that would be present in the best tutoring environment to complement the traditional classroom instruction. We can also extend the instruction to any number of locations simultaneously, or even replace the physical classroom. Students can interact with the instructor outside of class hours, view materials from libraries and other virtual resources around the world, run interactive software, complete homework assignments, consult subject matter experts, collaborate with other students, and communicate in other ways consistent with active learning. This is the virtual classroom approach as described by Ball (1998), and Walker, Janssen, and Avner (1998).
10. Intimidate your inexperienced users. There are many members of the instructional staff who have little experience with instructional computers. They can be easily intimidated. Instead of making the computer seem more of threat than a helper, start with an aspect of good instruction that they are already familiar with. Good instruction historically has been based upon questioning. By bringing computer technology into the picture as first introduced in the Police Training Institute at the University of Illinois by Education Professor J. Richard Dennis (Dennis, Flammang, & Walker, 1979), we can involve the benefits of currency, tailored feedback, required responses, repetition, and multi-media as would be present in the best tutoring environment. In optimizing the learning of concepts, principles, and factual information one must be aware of several important principles.
A. A single question correctly answered once carries a relatively low probability of signifying learning.
B. A single question, correctly answered several times, increases the probability that learning has occurred, but also increases the probability that such learning is rote memorization.
C. Correctly responding to several similar (but not identical) questions on an objective increases the probability that learning has occurred, and minimizes the chance that such learning is other than desirable.
D. Questions eliciting incorrect responses must be revisited more frequently and a greater number of times to insure correction of the misconceptions.
E. Incorrect responding requires very specific and appropriate feedback to facilitate correction of a misconception.
An instructor, alone, cannot affect the level of instructional complexity implied by the preceding principles. Neither do the conventional instructional media. Some type of automated individualization and management assistance is required. This is an appropriate instructional task for a computer system. Most instructors already have the knowledge component of their program cast in terms of objectives and sub-objectives. For each objective, a pool (set) of questions is created representing the knowledge and behavior associated with the objective. This pool contains questions causing the learner to address the objective from several points of view. These questions are entered into a computer-controlled question bank that preserves the clustering by objective and sub-objective.
Such a facility may be used in several ways. For initial instruction, objectives and sub-objectives may be grouped to form self-study facilities. Included with such an organization would be assignments to multi-media facilities, readings, etc. The computerized questioning facility would follow an instructor-planned curriculum. Keeping records on the student about successes and weaknesses allows the study principles mentioned above to be invoked.
This mode of computer application is referred to as computer-managed instruction (or CMI). Studies of similar instructional systems in other settings suggest that each student will encounter two to three times as many questions in a unit of time as in conventional instructional environments. The more important attribute, however, is that when questioning is responsive to each individual, learning rates increase.
A second way to use a computerized question bank is to provide the student a means for self-initiated review. In this mode, the student would receive questioning on objectives or sub-objectives of his/her choice. These could be confined to a single objective, a cluster of objectives, or a comprehensive survey over all the objectives in preparation for an examination.
Still another use of an automated question bank is to administer evaluation activities. The reader may have pause at this suggestion, thinking that using questions for both study and testing in some way destroys the credibility of the testing. This is not a concern when the set of items for each objective is more than a few questions. As a practical matter, in most of the areas of Police Science, a pool of eight has been found to be more than enough. At this level, the probability of having study activities identical to testing activities drops to nearly zero.
The authors can be contacted at the Internet address http://www.staff.uiuc.edu/~royw for reviews, an e-mail box for asking questions including how to get started in developing your own courseware, and running computer based police training demonstrations live via the Internet.
APPENDIX A
Listing of Lessons by Hour
Equivalent Classroom
LESSONS Hour (ECH)1
Arson Drill and Practice Review . . . . . . . . . . . . . . . . . . . . . . . 2
Child Sex Exploitation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Child Sex Exploitation Drill and Practice
Communication in the Police Environment . . . . . . . . . . . . . . . 6
Communication in the Police Environment Final Exam
Community Policing: Police Community Relations. . . . . . . . . 4
Community Policing Myths and Case Study Analysis
Crisis Intervention. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10
Crisis Intervention Final Exam
Crowd-Mob-Riot Behavior. . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Crowd-Mob-Riot Behavior Drill and Practice
Emotional Stability for Police Officers . . . . . . . . . . . . . . . . . . . 10
Evidence Potential, Packaging, and Transmittal. . . . . . . . . . . . 2
Feedback, TA and Personality Traits Drill and Practice . . . . . 1
Interviewing and Interrogation . . . . . . . . . . . . . . . . . . . . . . . . . 10
Interviewing and Interrogation Final Exam
Jail Climate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Jail Climate Final ExamLeadership Styles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Motor Vehicle Theft . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Observation and Perception: Street Communication . . . . . . . . 6
Packaging and Transmittal of Evidence (Part One) . . . . . . . . . 2
Packaging and Transmittal of Evidence (Part Two) . . . . . . . . . 2
Passenger Vehicle Identification . . . . . . . . . . . . . . . . . . . . . . . . 4
Police Management: Problem Personnel. . . . . . . . . . . . . . . . . . 4
Police Management: Problem Personnel Case Study Analysis
Police Use of Deadly Force . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Police Use of Deadly Force Final Exam and Case Study Analysis
Report Writing Drill and Practice . . . . . . . . . . . . . . . . . . . . . . . 1
Sexual Harassment in Policing . . . . . . . . . . . . . . . . . . . . . . . . . 6
Stabilizing the Traffic Accident Scene . . . . . . . . . . . . . . . . . . . 10
Stress Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10
Stress Test (Drill and Practice) . . . . . . . . . . . . . . . . . . . . . . . . . 2
Test Your Evidence Potential I.Q. . . . . . . . . . . . . . . . . . . . . . . 2
The Arson Crime Scene Search . . . . . . . . . . . . . . . . . . . . . . . . 6
The Functional Components of Interrogation . . . . . . . . . . . . . 2
The Police and the Underprotected Child . . . . . . . . . . . . . . . . 20
The Police and the Underprotected Child Drill and Practice
Variant Behavior . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Variant Behavior Final Exam
Pretest and Posttest (Variations by Lessons and Modules) . . 2
Total 154
Drill and Practice (17 Basic topics, a total of 897 items) . . . . 15-30
TOTAL 169-184
1
The hours listed above are Equivalent Classroom Hours (ECHs) which represent an estimate of the time it would take to teach each Lesson in the traditional classroom setting.
APPENDIX B
SUGGESTED ARRANGEMENT OF LESSONS BY MODULE
CRISIS MANAGEMENT
LEADERSHIP MODULE
This microcomputer module is designed to provide the police officer with a variety of competencies intended to insure appropriate development of effective crisis management leadership skills involving critical operational problems. The purpose of the pretest and posttest is to verify that the students have the prerequisites for instructional modules, and when completed, that they have demonstrated the required level of learning.
Equivalent Classroom Hour (ECH)
LESSONS
Sexual Harassment in Policing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Leadership Styles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Police Management: Problem Personnel . . . . . . . . . . . . . . . . . . . . . . . . . 4
Stress Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Community Policing: Police Community Relations . . . . . . . . . . . . . . . . . 4
Crowd-Mob-Riot Behavior . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Police Use of Deadly Force . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Crisis Intervention . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Pretest and Posttest . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 ____
Total 48
____________________
Note 1. The hours listed are Equivalent Classroom Hours (ECHs) which represent an estimate of the time it would take to teach each Lesson in the traditional classroom setting.
Note 2. Student study guides have been developed for almost all Lessons with the exception of the drills.
EMOTIONAL STABILITY, STRESS,
AND VARIANT BEHAVIOR MODULE
Street-level oriented, the instruction is intended to address selected major operational problems on-the-street, e.g.: emotional instability due to a mismatch among emotions, expectations, and behaviors; failure to understand what is happening under physical and psychological stress; and on-scene operational ineffectiveness in responding to disturbed persons calls. The purpose of the pretest and posttest is to verify that the students have the prerequisites for instructional modules, and when completed, that they have demonstrated the required level of learning.
Equivalent Classroom Hour (ECH)
LESSONS
Emotional Stability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Stress Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Variant Behavior . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Pretest and Posttest . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
____Total 24
INTERVIEWING AND
INTERROGATION MODULE
The instruction is intended to address selected major operational problems on-the-street, e.g.: failure to obtain necessary information in situations calling for precise application of interview and interrogation skills; and general street operational difficulties due to a lack of communication skills. The purpose of the pretest and posttest is to verify that the students have the prerequisites for instructional modules, and when completed, that they have demonstrated the required level of learning.
Equivalent Classroom Hour (ECH)
LESSONS
Observation and Perception: Street Communication . . . . . . . . . . . . . . . . 6
Communication in the Police Environment . . . . . . . . . . . . . . . . . . . . . . . 6
Interviewing and Interrogation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
The Functional Components of Interrogation . . . . . . . . . . . . . . . . . . . . . 2
Pretest and Posttest . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
____Total 26
INVESTIGATION PROCEDURES MODULE
With an on-the-street orientation, the instruction is intended to address selected major operational problems, e.g.: expecting too much from the forensic science/crime laboratory due to overestimating the relative value of physical evidence submitted and making improper submissions; inaccurate decoding and reporting of vehicle identification numbers (VINs); improper protection of the crime scene resulting in the contamination, loss, or unnecessary movement of physical evidence items, any one of which likely to render the evidence useless; failure to prepare clear, concise, accurate and effective investigator's reports; and forgetting or ignoring proper procedures when faced with complex traffic accidents and time pressures. The purpose of the pretest and posttest is to verify that the students have the prerequisites for instructional modules, and when completed, that they have demonstrated the required level of learning.
Equivalent Classroom Hour (ECH)
LESSONS
Test Your Evidence Potential I.Q. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Packaging and Transmittal of Evidence (Part One) . . . . . . . . . . . . . . . . . 2
Packaging and Transmittal of Evidence (Part Two). . . . . . . . . . . . . . . . . 2
Crime Scene Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Passenger Vehicle Identification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Stabilizing the Traffic Accident Scene . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Pretest and Posttest . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
____Total 24
CHILD ABUSE/NEGLECT INVESTIGATION MODULE
Street-level oriented, the instruction is intended to address the problem of oversimplified perceptions of a sophisticated multi-dimensional social phenomena, coupled with the need for a wider focus during the initial stages of the investigation of child abuse/neglect allegations. The purpose of the pretest and posttest is to verify that the students have the prerequisites for instructional modules, and when completed, that they have demonstrated the required level of learning.
Equivalent Classroom Hour (ECH)
LESSONS
The Police and the Underprotected Child . . . . . . . . . . . . . . . . . . . . . . . . 20
Child Sex Exploitation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Pretest and Posttest . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
____Total 26
SPECIAL PROGRAMS
Equivalent Classroom Hour (ECH)
MISCELLANEOUS LESSONS
Report Writing Drill and Practice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Illinois Domestic Violence Act Drill and Practice . . . . . . . . . . . . . . . . . 2
Feedback, TA and Personality Traits Drill and Practice . . . . . . . . . . . . . 1
Jail Climate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Pretest and Posttest (Variations by Lessons and Modules) . . . . . . . . . . 2
Drill and Practice (897 items with 1,794 feedbacks) . . . . . . . . . . . . . . . . . 15-30
_____Total 24-39
MOTOR VEHICLE THEFT
INVESTIGATION MODULE
LESSONS
Motor Vehicle Theft . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Passenger Vehicle identification*. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Evidence Potential, Packaging, and Transmittal* . . . . . . . . . . . . . . . . . . . 2
The Functional Components of Interrogation* . . . . . . . . . . . . . . . . . . . . 2
Pretest and Posttest . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2
____Total 12
__________________
Note 3. The asterisks
"*" identify a Lesson tailored for more than one module. The modules listed are optional. Curricula are available for the modules with the exception of Arson Investigation and Special Programs. The curricula feature a seminar component with practical activities.
ARSON INVESTIGATION
MODULE
Equivalent Classroom Hour (ECH)
LESSONS
The Arson Crime Scene Search* . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
The Functional Components of Interrogation* . . . . . . . . . . . . . . . . . . . . . . . 2
Passenger Vehicle Identification* . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Stress Management* . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
Arson Drill and Practice Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
Pretest and Posttest . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
____Total 26
LIST OF REFERENCES
Avner, Allen. (1992, July 30). PTI CBE learning certification module assessment. (CERL Evaluation Report, University of Illinois at Urbana-Champaign USA, Computer-based Education Research Laboratory, Educational Applications Research Group, Evaluation
Research and Support Office). (Available from Allen Avner, e-mail a-avner@uiuc.edu ,or R.O. Walker, e-mail rowalker@hotmail.com)
Avner, R.A. (1974). How to produce ineffective CAI material. Educational Technology, 14(8).
Avner, Allen, & Friedman, H. George. (1981). Interacting with computer users: Design considerations. In J.L. Divilbiss (Ed.), Public access to library automation. Urbana, Illinois USA: Board of Trustees, University of Illinois, 8-19.
Ball, Catherine N. (1998, May). [Internet Based Program] Managing the virtual classroom. Washington, D.C. USA: Georgetown University. (Available at the Internet address: http://www.georgetown.edu/cball/cball.html)
Broadbent, Donald A. (1975). The magic number seven after fifteen years. In Alan Kennedy and Alan Wilkes (Eds.), Studies in long term memory. New York: Wiley, 1975, 3-18.
Elmer-DeWitt, Philip. (1994, September 26). How the Internet was tamed. Time, p. 60.
Handbook of the United Nations: Crime and justice information network. (1997). [Internet:http://www.ifs.univie.ac.at/~uncjin/manual/uncjin.hdb]. Vienna, Austria: UNCJIN Systems Operator, Crime Prevention and Criminal Justice Branch, United Nations Office at Vienna, P.O. Box 500, A-1400 Vienna, Austria (telephone: +43-1-21345-4272; telefacsimile: +43-1-21345-5898: e-mail: evetere@cpcjb.un.or.at).
Johnson, B., Van Meter, C.W., and Walker, R.O. (1995). The effectiveness of computer-based education in criminal justice undergraduate curricula: An evaluation. Police Computer Review, 4(3), 11-18. (Available from Brian Johnson, Grand Valley State University, School of Criminal Justice, 239 Mackinac Hall, Allendale, Michigan 49401 USA)
Maney, Kevin. (1997, January 2). Worm turns at Apple in 97. USA TODAY, p. 2B.
Markle, D.G. (1967). On the control of runaway programmers. NSPI Journal, 6 (7).
Minatel, Jim. (1995). Easy World Wide Web with Netscape. Indianapolis, Indiana USA: Que Corporation.
Roblyer, M.D. (1989). Technology in criminal justice training: A review of the literature. (The Florida A & M University, College of Education Contract No. 00363.) Tallahassee, Florida USA: Florida Department of Law Enforcement, Division of Criminal Justice
Standards and Training, Bureau of Training. (The author can be contacted at Florida A&M University, College of Education, Tallahassee, Florida 32307)
Savola, Tom. (1995). The most complete reference: Special edition using HTML. Indianapolis, Indiana USA: Que Corporation.
Shafran, Andy. (1995). Creating your own Netscape Web pages. Indianapolis, Indiana USA: Que Corporation.
TenCORE Language Authoring System, Version 8.5. [Computer teaching software, including Runtime for Internet delivery]. (1999). (Available from Computer Teaching Corporation, 1713 S. State Street, Champaign, Illinois 61820 USA, e-mail pt@tencore.com) UNESCO statistical yearbook. (1995). Lanham, Maryland USA: Bernam Press, 6-11.
Walker, R.O. (1992). Computers in law enforcement. In Gary G. Bitter (Ed.), Macmillan Encyclopedia of Computers. New York USA: Macmillan Publishing Company, 590-597.
Walker, Roy O. (1997, July/August). Impulse control and cue words. American Jails, XI(3), 76-79. (Available from the American Jail Association, 2053 Day Road, Suite 100, Hagerstown, Maryland 21740-9795 USA)
Walker, R. O. (1999). [Internet Based Education and Training Review] Computer based police training: Individualized on-campus and distance learning applications, Fourth Edition. Urbana-Champaign, Illinois USA: The University of Illinois. (Available at the faculty Internet address: http://www.staff.uiuc.edu/~royw/)
Walker, Roy O., Charles, Michael T., and Avner, Allen. (1994). Network managed computer based education for police. Police Computer Review, 3(2), 19-26 (First presented by the authors at the Academy of Criminal Justice Sciences Meetings in Kansas City, Missouri USA as a paper in 1993, and in a workshop in Chicago, Illinois USA in 1994.) (Available from Dr. R.O. Walker, e-mail rowalker@hotmail.com)
Walker, R., Janssen, J, and Avner, A. (1997, March 13). Early experiences in the utilization of the Internet: Distance learning. (A paper presented at the Meeting of the Academy of Criminal Justice Sciences, Louisville, Kentucky USA.) (Available from Dr. R.O. Walker, e-mail rowalker@hotmail.com)
Walker, Roy, Janssen, John, & Avner, Allen. (1997, May/June). [Internet Based Newsletter] Mary Michaels (Ed.), Distance learning: One solution to increasing quality of instruction while lowering cost. Instructional Microcomputing. Urbana-Champaign, Illinois USA: The University of Illinois, Educational Technologies Assistance Group (ETAG). (Available on-line at the University of Illinois, ETAG Internet address: http://www.oir.uiuc.edu/etag/news/issue/may97.html)
Walker, R., Janssen, J, and Avner, A. (1998, March 12). Creating effective distance learning environments on the World Wide Web for police and allied criminal justice personnel. (A paper presented at the Meeting of the Academy of Criminal Justice Sciences, Louisville, Albuquerque, New Mexico USA.) (Available at the Criminal Justice Distance Learning Consortium, Internet address: http://www.cjcentral.com/cjdlc/walker.htm)
World almanac and book of facts. (1997). Mahwah, New Jersey USA: World Almanac Books, 297.