Running Head: VIOLENCE PREVENTION

Violence Prevention: The Development of Internet-Delivered,

Experimentally-Evaluated, Psychological-Education Curricula

John J. Horan

Arizona State University

March 26, 2002

Abstract

Violence prevention requires a broad focus. Several multimedia-enhanced, psychological education courses that are capable of Internet delivery have been developed at Arizona State University and at KnowConflict, LLC. These interactive interventions have evolved from earlier work on simulations and computer-based learning; however, they incorporate recent technological advances and current psychological research in areas such as differential diagnosis, observational learning, and cognitive restructuring. They specifically address changing the irrational beliefs that mediate low self-esteem and occupational stereotyping, educating parents on practices affecting the career outcomes of their children, and altering attributions relevant to academic motivation and performance. Although all of these psychological education curricula are relevant to violence prevention, and all have been subjected to continuing conceptual and empirical scrutiny, the most recent project deals directly with enhancing the conflict management skills of our nation’s youth.

Violence Prevention: The Development of Internet-Delivered,

Experimentally-Evaluated, Psychological-Education Curricula

The implications of the Internet for knowledge dissemination are profound. Not only can information be published virtually as it happens, but the number of people having immediate access has increased exponentially. In the fields of Education, Counseling, and Psychology, researchers are now able to publish studies in electronic journals or indeed on their own web sites with almost no time lag. A manuscript sent to the Journal of School Violence, for example, may be reviewed by many referees and published on the Internet within a few weeks from the date of submission. Web publishing also permits authors to include complex (even interactive) charts and graphs that would not be possible in a paper article because of space or technological limitations. Moreover, authors can provide hypertext links within their articles to the cited research of other researchers and to their own raw data, thus making everything available to other scientists for reconsideration and reanalysis (Behrens, Dugan, & Franz, 1997; Horan, 1999).

Even more profound from the standpoint of clinical practice, the Internet permits immediate publication of any psychological education curriculum per se as well as the research describing and evaluating it. In the past, production and distribution costs have impeded widespread dissemination of treatment manuals and self-help materials. Programs designed to run off of hard drives or CD-ROMs face similar barriers as well as prohibitive cost-benefit considerations when revisions become desirable. Practitioners are thus confronted with complex purchasing decisions involving not only the applicability of a given program to their client population but also the prioritizing of different programs addressing different problems in living. Consequently, many individuals who might profit from a particular program will never experience it.

Conversely, empirically effective Internet-based interventions are easily exportable. For example, they are capable of being revised, edited, or improved on the fly; new editions are immediately accessible. Moreover, Internet servers can be configured so that consumers are charged on a per-use basis; publishers in effect trade large profits from a few for smaller profits from many. From the consumer's standpoint, per-use costs are trivial in comparison to the outright purchase of desktop software, thus education and mental health organizations face essentially no barriers to adoption.

More importantly, the Internet permits the interactive delivery of intervention materials in this same low-cost format capable of widespread adoption. The benefits of interactivity are profound when compared, for example, to the usual self-help book that requires individuals to wade through all chapters in linear fashion regardless of their relevance to the actual concern. Although bulky and costly computer programs have long been able to target only those areas in need of remediation, consumer budget decisions have curbed the development of interactive technology. The unlimited audience of the Internet, however, transforms the economics of scale from a liability to an asset.

Paradoxically, the Internet enables both interactivity and standardization; the latter is a sine qua non for evaluation. The purported efficacy of a given drug, for example, presumes a standard quantity and frequency of administration. Similarly, the empirical stature of a counseling protocol presumes that counselors adhere to the critical components of the intervention. Counselors differ, of course, in knowledge and skill; even those functioning at the highest levels may not have access to accurate information or cognitive modeling scripts in response to whatever individuals might present. Consistent interactivity, on the other hand, is a raison d'etre for Internet programs.

Despite the justified excitement, we must exercise caution during this era of innovation and development. The public now has access to unlimited information, programs, and products. Ethically and scientifically sound material coexists on the Internet with shoddy work that has not been evaluated at any level. So in our zest to join and contribute to the virtual community, we have an obligation to ensure the integrity of our work before we make it available. In fact, however, psychological education curricula delivered on the Internet are no more prone to ethical shenanigans than programs delivered in a conventional manner. Indeed, the interactive and standardized nature of Internet assessments and interventions engenders hope for significantly enhancing their reliability, validity, and clinical utility.

This article presents a brief overview of current Internet applications that have been developed and evaluated at Arizona State University and more recently at KnowConflict, LLC (http://www.knowconflict.com/). I open with a discussion of how these interactive interventions have evolved from earlier work on simulations and computer-based learning; however, the programs described here incorporate recent technological advances and current psychological research in the areas of differential diagnosis, observational learning, and cognitive restructuring. I then move on to describing their individual foci, namely, changing the irrational beliefs that mediate low self-esteem and occupational stereotyping, educating parents on practices affecting the career outcomes of their children, and altering attributions relevant to academic motivation and performance. More recently, we have attempted to improve directly the conflict-management skills of our nation’s youth.

When originally conceived, each program addressed only its immediate objectives. The self-esteem program, for example, derived from correlational research linking poor self-concepts to the holding of specific irrational beliefs. So our outcome criteria reflected only improvements in rationality and self-esteem. Of course, low self-esteem has been linked in the research literature and popular press to everything from alienation to xenophobia but the cannons of evaluation require a sharp focus in experimental research. We needed to know if the self-esteem program improved self-esteem.

Undoubtedly there is a reciprocal relationship between many of our individual programs. Enhanced self-esteem may contribute to improved academic performance and vice-versa, but explorations of such generalization phenomena did not concern us until our association with KnowConflict, LLC. This corporation emerged from the ashes of Columbine with an idea that the conflict management skills of our nation's youth could be enhanced through deliberate psychological education delivered over the Internet.

Violence prevention requires a broad focus, so the KnowConflict curriculum includes, for example, training modules on the conflict-resolution paradigm developed by the Harvard Negotiation Project, anger management and rational thinking, listening and assertiveness (as opposed to aggressiveness), and decision-making skills. Additional training materials are made available to parents and teachers in the spirit of increasing students’ connectedness to schools as way to reduce their propensity for violence (c.f., Crawford & Bodine, 1996). Other elements from our previous psychological-education curricula will be incorporated into future editions of the KnowConflict program as it expands downward to the junior high and elementary school levels and deeper to include more advanced conflict-management skills.

The Evolution of Our Work

Our work has its origins in simulation and traditional computer-based interventions onto which we have annealed improvements derived from current scientific and technological advances. Simulations provide a special kind of personal knowledge extremely relevant to career decision-making. Several decades ago John Krumboltz developed the Job Experience Kits (1970), a series of out-of-the-box exercises that enabled adolescents to try out a number of possible occupations prior to selecting a career. Although these kits survived subsequent experimental scrutiny (e.g., Krumboltz & Baker, 1973), studies on other early simulations such as The Life Career Game (Boocock, 1967) reported both positive (Varenhorst, 1969) and null effects (Johnson & Euler, 1972; Munsen, Horan, Miano, & Stone, 1976). To be sure, evaluative studies of simulations can in turn be criticized on the basis of problematic assessment devices, low participant interest, and so forth.

In any event, subsequent computer-based simulation experiences have had a heavy impact in the training of pilots and medical personnel. The learning experiences provided are inexpensive and life like. More importantly, they are not life threatening to those being trained or served. One law enforcement program brought to our attention, for example, simulates a house occupied by a gun-wielding felon. If police cadets deviate from prescribed search procedures, the sound of a gunshot accompanied by a forceful blast of air provides an unforgettable one-trial learning experience!

Krumboltz is currently adapting his job experience kits for the computer. He demonstrated the first of these at APA in 1997 (Krumboltz, 1997). It provides a rich audio- and video-intensive exposure to actual job experiences. Although Krumboltz's first kit runs off of a CD-ROM, there is no reason why similar interactive experiences cannot be developed for the Internet.

Well, why hasn't that happened yet? Although the Internet has existed for many years, graphic-intensive browsers capable of delivering interactive audio and visual stimuli are still in their infancy. Indeed, "automatic" audio activated by embedded html tags was not supported by Netscape (Netscape Navigator 2.0, 1996) or the Microsoft Internet Explorer (Internet Explorer 2.0, 1995) until the fall of 1996. And anyone who has ever tried to fill a swimming pool with a garden hose understands the problem of bandwidth; no one likes to wait days for the page or pool to fill. Although web sites using virtual reality modeling have been around for a while, their effects are still a bit cartoonish, thus providing little incentive for incurring labor-intensive developmental costs.

Nevertheless, cable modems and satellite downloading are now at hand; and compression schemes are getting better every day. In our own work we have been using all of the available plug-ins that enable audio- and video-display to occur in real time at no cost to the consumer. In the near future it is reasonable to expect that the Internet will provide interactive programming on par with the production values of current high-quality CD-ROMs. Indeed, everyone will be able to watch a digital quality movies over the Internet at their own convenience -- no annoying midnight excursions to return the rented tape.

Interactive Interventions Involving Differential Diagnosis

Traditional self-help intervention programs (with or without counselor supervision) require all clients to progress through various lessons in linear order regardless of the relevance of such materials to their particular experience of a clinical problem. For example, stimulus-control applications to obesity inevitably involve training objectives for everyone such as setting the fork down between bites; even clients with normatively low consumption rates are taught to slow down their eating pace.

Computer- and Internet-based programs, on the other hand, are capable of assessing the unique pattern of knowledge and skill deficits inherent in each individual so that appropriate intervention components can be delivered to only those areas in need of remediation. Differential assessment and treatment essentially involves the theoretically coherent linking of assessment categories and intervention strategies. Computer-based assessment and canned behavioral interventions such as desensitization have been with us for decades; their vices and virtues are well known and have been reviewed elsewhere (e.g., Newman, Consoli, & Taylor, 1997). Putting tests on the Internet is, technologically, a no-brainer. Although validation of the online version is necessary, any of the tried and true vocational assessment devices can easily be adapted for Internet delivery. Indeed, procedures for billing and protocol encryption to ensure confidentiality are already in widespread use.

Interactive Internet treatments linked to assessment categories, on the other hand, are brand new. Our own research team has been exploring the effects of Internet-based cognitive restructuring on a wide variety of counseling problems. We began this work, frankly, without the Internet in mind. Rather we were focused on harnessing the exponentially growing power of multimedia desktop computers. Our first desktop program (Horan, 1996) in compressed form, for example, occupied 121 megabytes of hard-drive space. Even sharing this program with students and colleagues required Zip drives or the burning of a CD-ROM. Although CD-ROM production and duplication costs have declined in recent years from staggering to trivial, in our minds the advantages of Internet delivery soon became self-evident. Indeed, we recently returned to that original self-esteem program and adapted it for the Internet (Dannenbaum-Daubney & Horan, 1999).

Internet-Based Cognitive Restructuring

Although normally associated with traditional educational content areas such as math and reading, computer and Internet treatments are applicable to fostering a much larger array of life skills including rational thought. Cognitive restructuring, which derives from areas of agreement among rational-emotive and behavior therapies, is perhaps the modal counseling procedure for addressing dysfunctional thinking. But despite its well-documented effects, cognitive restructuring is labor intensive and requires a high level of expertise to deliver. Even thoroughly trained professionals will find it difficult to remember the appropriate assessment questions pertaining to the irrational covariates of various clinical problems, let alone generate multiple impromptu logical analyses and cognitive modeling scripts.

Internet-based treatments are immune to such difficulties. They ensure a high level of expertise in the delivery of replicable responses to varying patterns of client irrationality. Many of the interventions described here initially determine if participants hold any of a number of specific irrational thoughts previously linked to a specific counseling problem; each program then provides individually tailored instruction. I will now move on to specific examples of our research and development work.

A Career Intervention for Young Women

Theresa Kovalski has written an Internet program entitled "Believe It: A Career Development Intervention For Young Women." Her program is designed to change four irrational career-beliefs common among middle-school-aged girls, namely:

• Children should be dependent on adults for their career choices.

• There is only one vocation in the world that will lead to (my) happiness.

• Choosing a vocation involves making final decisions at specific points in time.

• Certain jobs are more appropriate for men, while other jobs are better suited to women.

The interface is adapted from Horan's (1996) program for enhancing self-esteem. In Theresa's program, participants are assessed repeatedly as they progress through the program; and depending on the tenacity of each belief held, the program provides a variety of cognitive restructuring responses.

Theresa began writing her program and recording her scripts the year before Netscape supported automatic audio, a sine qua non, for street-appeal. Theresa was experimenting with several cumbersome plug-ins when Netscape released Version 3.01 which allowed anyone anywhere in the world to experience the audio elements of her program absolutely free and with no technical hurdles to leap. Although our future work in this area will routinely incorporate video clips, the technology available back then made cartoons caricatures representing a rainbow of ethnicity the best choice for visual aids.

A paper detailing the theoretical base of Theresa's program and reporting outcome details from an experimental trial was presented at APA 1998 and subsequently published (Kovalski & Horan, 1998, 1999). In a nutshell, adolescent girls were stratified on the basis of ethnicity and randomly assigned to either the career program or to a control computer treatment focused on a different subject area. Pre-post scores on four measures reflecting irrational career beliefs and stereotyping were subjected to treatment by ethnicity by repeated-measures analyses of variance. A triple interaction on self-stereotyping suggested that the program had an impact on Whites but not on Mexican Americans. Self-stereotyping essentially involves asking participants what they would like to be when they grew up and then noting any disparity to a follow-up question about what they would like to be when they grew up if they were male.

Theresa's findings are encouraging but by no means convincing. There are some obvious factors both within and beyond our control that might have inflated error and washed out treatment effects that otherwise might have emerged on some measures. For example, a student's unexpected tragic death occurred within the time frame that our study was run, and had a generally numbing and distracting influence on both experimental and control participants. And although we were careful to portray different ethnic groups in our stimulus materials, we did not anticipate the highly variable level of computer literacy among participants in our study. Some students had to learn how to click a mouse while they were receiving the treatment, a distracter that undoubtedly undermined the treatment's efficacy. Although the main effect for treatment was not significant, we believe that computer pre-training might replace the interaction between treatment and ethnicity with uniform treatment effects.

We are also carefully evaluating the feasibility of making certain structural changes in programs like this per se. One of the hallmarks of computer learning is the delivery of instruction only where needed. Differential diagnosis and treatment of this sort reduces boredom, speeds up student progress, and generally makes the most effective use of human and machine resources. We need to track student progress through our modules more effectively to determine if the students are opting out prior to fully developing their rational thinking skills.

Helping Parents Facilitate Their Children's Career Development

Ginger Clark has developed an interactive, web-based program designed to help educate parents on their children's career development process. The program, called "Kids and Careers," includes information about the common myths surrounding career development and career choice (e.g., parents know what is best for their children's career choices, or I never attended college so how can I expect my children to). It provides research-based information on motivation and career satisfaction. As parents move through the program, a framework is constructed outlining what they can expect at each stage of their children's career development, and how they can be instrumental in the process. The program contains information about resources that could be helpful to children in identifying interests, values, and abilities. It also includes links to other relevant web sites and programs for children.

In a recent experimental evaluation of her program (Clark & Horan, 2000), the parents' level of knowledge about career myths, career development, and career resources were measured before and after the intervention using measures of career-beliefs, knowledge of career-development, and knowledge/use of career assessment resources. During the program, parents are given feedback according to the level of knowledge they display in each subject area. If their responses to test items indicate information deficits, they are shunted to a page that provides more detail. If their response pattern indicates mastery of a particular area, they receive a reinforcement page with a brief summary and are then able to move on to the next item. The parents also have the option to choose which part of the program they would like to experience. For instance, if parents know that career choices should be based on interests, values, and abilities, but are unsure of where they might find the resources to assess these factors, they can simply click the module on resources and information and bypass the module on career choice and career satisfaction. The program is designed to provide parents with the tools necessary to access relevant information, to provide necessary support, and to encourage their children to explore as many opportunities in their career development as possible. In a study of the program's impact, parents receiving the Internet curriculum showed significantly greater improvement on several major outcome measures in comparison to those in the control condition.

Attribution Retraining for Academic Motivation and Performance

Amy Tompkins-Bjorkman's program is entitled "Just Think It: Internet-Based Attribution Retraining for Academic Motivation and Performance." Like cognitive restructuring, attribution retraining rests on the assumption that thoughts determine action; thus, changing the way a person thinks should result in behavior changes. Maladaptive causal attributions are replaced with functional attributions (Weiner, 1985); specifically, internal and stable attributions that increase expectancy of success and enhance motivation to in engage in academic tasks take the place of external and unstable attributions (Perry, Hechter, Menec, & Weinberg, 1993). If a person were to attribute success to ability rather than to luck, she or he would more likely expect success in the future because ability is a stable attribution that conveys a message of competence. Similarly, attribution retraining might focus on changing certain stable attributions to unstable ones, to promote the realization that a negative outcome can be changed. For example, if a person were to attribute failure to lack of effort rather than lack of ability, she or he would expect a different outcome in the future because effort is a variable factor that can be altered by the individual.

Many studies involving attribution retraining have focused on changing effort-attributions and report improved performance. Other studies have enhanced performance by successfully re-attributing failure to inappropriate strategies. Still others have shown improved academic performance using attribution retraining without actually manipulating attributions; instead, given certain antecedent information, participants inferred that their reasons for academic failure were due to unstable causes. Particularly relevant to the Tompkins-Bjorkman program are those studies that have focused on improving academic performance by combining a study skills program with attribution retraining (Borkowski, Weyhing, & Carr, 1988; Van Overwalle & DeMetsenaere, 1990; Van Overwalle, Segebarth, & Goldchstein, 1989). A pilot experimental evaluation of the Tompkins-Bjorkman program completed last year involved blocking participants on gender and ethnicity and randomly assigning them to experimental and control conditions. All students received standard study-skills training; experimental participants received Internet-based attribution retraining. The assessment battery included devices reflecting attributions, study skills, achievement, performance, and experimental demand. A formal experimental evaluation is planned.

Cognitive Restructuring and Low Self-Esteem

Our earliest, but continuing work in this area focused on the modification of low self-esteem via cognitive restructuring. The logic for this line of inquiry proceeds as follows. Many ex post facto studies have linked specific irrational beliefs to various psychological dysfunctions such as anger, anxiety, depression, and low self-esteem (see Horan, 1996; Nielsen et al, 1996). For example, Daly and Burton (1983) found that among college students, four subscales of the Irrational Beliefs Test (IBT, Jones, 1996) were significant predictors of low self-esteem; the other subscales were not. McLennan (1987) also reported that these same irrational beliefs forecasted low self-esteem assessed by a different measure.

Erickson, Horan, and Hackett (1991) noted that two of these irrational beliefs continued to predict low self-esteem with a younger population. From a construct-validity standpoint, logical divergent relationships appeared as well. For example, neither belief was related to theoretically remote control measures (extraversion, facilitative anxiety, and grade-point-average); conversely, other irrational beliefs correlated with these control measures were not associated with low self-esteem.

Subsequently, Horan (1996) developed an interactive multimedia computer program to foster rational thinking in belief areas previously associated with low self-esteem. In contrast to a control condition, computer-based cognitive restructuring produced improvements on a battery of devices assessing both rationality and self-esteem.

A later study (Dannenbaum-Daubney & Horan, 1999) adapted this program for Internet delivery and addressed additional challenges. In the first place, the relationship between specific irrationality and low self-esteem along the life span was not clear. The Daly and Burton and McLennan studies involved college students; the Erickson, et al and Nielsen et al projects were conducted on high school students. The relationships between rationality and self-esteem were not perfectly consistent between these populations; further differences in younger subjects would undermine the logical basis of cognitive restructuring for these particular beliefs. Although the earlier version of the program (Horan, 1996) proved efficacious with high school juniors and seniors, it was an open question whether similar results might occur with seventh- and eighth-grade adolescents. Moreover, the Horan (1996) study was remedial in nature; only subjects with low self-esteem were included. It thus remained to be seen whether cognitive restructuring for these specific beliefs could foster improvements in rationality and self-esteem among normal populations.

The conversion of the program to an Internet format presented several opportunities for improving it. For example, we increased the program's visual appeal by adding benevolent graphical "mentors" (Jones, Valdez, Nowakowski, & Rasmussen, 1995) and visual metaphors (Fenk, 1994; Mayer, 1994; Gyselinck & Tardieu, 1994; McDaniel & Waddill, 1994) to aid in the cognitive restructuring. We also incorporated the script text into the display graphics to enhance learning and/or allow for hearing impaired participants (Weidenmann, 1994). Finally, we used three-dimensional figures and added both auditory and visual reinforcers for rational responses prior to exiting a module.

The Dannenbaum-Daubney and Horan (1999) study involved sixty-five seventh- and eighth-grade students (blocked on gender, ethnicity, and grade-level), randomly assigned to experimental and control conditions, and assessed before and after treatment on a battery of devices reflecting irrationality and self-esteem. The primary outcome analyses did not confirm the program's earlier overall success with the new sample and conditions. Exploratory analyses, however, indicated that self-esteem benefits were evident among eighth grade but not seventh grade participants

Conflict-Management Education

KnowConflict, LLC has undertaken the most ambitious project ever conceived in this area. They are an education company creating Internet courseware for K-12 schools. Their first course contains eight lessons designed as a package to foster conflict management skills among our nation's youth. The company is well managed and has sufficient resources to deliver this curriculum and others like it in school systems throughout the country. All of the previous projects could be construed as simply "proof of concept;" the KnowConflict curriculum, on the other hand, is the first such product ever to be positioned for widespread implementation.

Violence prevention requires a broad focus. Tobin (2001) has described the theoretical basis of the curriculum. In a nutshell, the program has gone through three revisions. Lesson 1 in the current edition provides an overview of conflict and illustrates possible hard, soft, and principled responses. It also contains an advance organizer for the lessons to follow. Lesson 2 delves deeply into the conflict-resolution paradigm developed by the Harvard Negotiation Project, popularly known as the "Win-Win" approach; learning how to separate positions from interests is a major lesson focus. Lessons 3 and 4 are concentrated on anger management and rational thinking. Irrational thoughts underlying rage and low self-esteem, for example, are exposed in standard cognitive behavioral fashion, while coping skills and rational trains of thought are substituted in their stead. The major ingredients of communication -- listening and assertiveness (as opposed to aggressiveness) -- are fostered in lessons 5 and 6, followed by decision-making skills in Lesson 7. Practice scenarios putting it all together along with a call for action occur in Lesson 8.

The conflict-management curriculum derives from strategies well known and validated in the professional and scientific literatures, and the lessons themselves make liberal use of audio and video modeling along with interactive elements that build toward mastery of the requisite skills. Additional training materials are made available to parents and teachers in the spirit of increasing students’ connectedness to schools as way to reduce their propensity for violence (c.f., Crawford & Bodine, 1996). Other elements from our previous psychological-education curricula will be incorporated into future editions of the KnowConflict program as it expands downward to the junior high and elementary school levels and deeper to include advanced conflict management skills. Experimental evaluations of the current version are being conducted. Readers are welcome to preview the course at the KnowConflict website: http://www.knowconflict.com/

In this article I have discussed the power of the Internet and emphasized the need to ensure that Internet applications survive empirical scrutiny. I have also provided detailed descriptions of Internet programs being developed and evaluated at Arizona State University and KnowConflict, LLC. These include interactive interventions that incorporate differential diagnosis, observational learning, and cognitive restructuring. They focusing on changing the irrational beliefs that mediate low self-esteem and occupational stereotyping, educating parents on practices affecting the career outcomes of their children, altering attributions relevant to the academic motivation and performance, and enhancing the conflict-management skills of our nation's youth. As we stand on the threshold of the new millennium, these are indeed exciting times for those in our field who have strong interests in the exportability of empirically validated interventions.

References

Behrens, J. T., Dugan, J. G., & Franz, S. (1997, August). Improving the reporting of research results using the World Wide Web. In R. W. Lent (Chair), Information highways, byways, and cul de sacs in counseling psychology. Symposium conducted at the meeting of the American Psychological Association, Chicago.

Boocock, S. S. (1967). The Life Career Game. Personnel and Guidance Journal, 46, 328-334.

Borkowski, J., Weyhing, R., & Carr, M. (1988). Effects of attributional retraining on strategy-based reading comprehension in learning-disabled students. Journal of Educational Psychology, 80, 46-53.

Clark, G. & Horan, J. J. (August, 2000). The effects of Internet-based parent education on the career development of gifted adolescent girls. Paper presented at the annual meeting of the American Psychological Association, Washington, D.C.

Crawford, D., & Bodine, R. (1996). Conflict resolution education: A guide to implementing programs in schools, youth-serving organizations, and community and juvenile justice settings. Program Report, Office of Juvenile Justice and Delinquency Prevention, U.S. Department of Justice.

Daly, M.J., & Burton, R.L. (1983). Self-esteem and irrational beliefs: An exploratory investigation with implications for counseling. Journal of Counseling Psychology, 30, 361-366.

Dannenbaum-Daubney, S., & Horan, J. J. (August, 1999). The effects of Internet-based cognitive restructuring on adolescent self-esteem. Paper presented at the annual meeting of the American Psychological Association, Boston.

Erickson, C.D., Horan, J. J., Hackett, G. (1991, August). On thinking and feeling bad: Do client problems derive from a common irrationality or specific irrational belief? Paper presented at the annual meeting of the American Psychological Association, San Francisco, CA.

Fenk, A. (1994). Spatial Metaphor and Logical Pictures. In W. Schnotz & R.W. Kulhavy (Eds), Comprehension of Graphics.(pp. 43-62). Amsterdam, The Netherlands: North Holland.

Gyselinck, V. & Tardieu, H. (1994). Illustrations, Mental Models, and Comprehension of Instructional Text. In W. Schnotz & R.W. Kulhavy (Eds), Comprehension of Graphics.(pp. 139-152). Amsterdam, The Netherlands: North Holland.

Horan, J. J. (1996). Effects of computer-based cognitive restructuring on rationally mediated self-esteem. Journal of Counseling Psychology, 43(4), 371-382.

Horan, J. J. (1999, August). Methodological diversity and outcome research in counseling psychology. In G. Hackett & J. W. Lichtenberg (Co-chairs), Issues related to methodological diversity in counseling psychology research. Symposium conducted at the meeting of the American Psychological Association, Boston. Retrieved May 15, 1999 from the World Wide Web: http://horan.asu.edu/d-method-apa99.htm

Internet Explorer 2.0 [Computer Software]. (1995). Redmond, WA: Microsoft Corporation.

Job Experience Kits. (1970). Chicago: Science Research Associates.

Johnson, R. H., & Euler, D. E. (1972). Effect of the Life Career Game on the learning and retention of educational-occupational information. The School Counselor, 19, 155-159.

Jones, B.F., Valdez, G., Nowakowski, J., & Rasmussen, C. (1995). Plugging In: choosing and using educational technology. EdTalk [Online], pp.1-11. Available: http://www.ncrel.org.sdrs/edtalk/newtimes.htm.

Jones, R.G. (1969). A factored measure of Ellis' irrational belief system, with personality and maladjustment correlates (Doctoral Dissertation, University of Missouri, Columbia, 1968). Dissertation Abstracts International, 29(11B), 4379-4380.

Kovalski, T., & Horan, J. J. (1998, August). Internet-cognitive-restructuring's impact on career beliefs of adolescent girls. Paper presented at the meeting of the American Psychological Association, San Francisco.

Kovalski, T. M., & Horan, J. J. (1999). The effects of Internet-based cognitive restructuring on the irrational career beliefs of adolescent girls. Journal of Cognitive Psychotherapy, 13(2), 145-152.

Krumboltz, J. D. (1997). Virtual job experience. In R.W. Lent (chair), Information highways, byways, and cul de sacs in counseling psychology. Symposium conducted at the meeting of the American Psychological Association, Chicago.

Krumboltz, J. D., & Baker, R. D. (1973). Behavioral counseling for vocational decisions. In H. Borrow (Ed.), Career guidance for a new age (pp.235-284). Boston: Houghton Mifflin.

Mayer, R.E. (1994). Visual Aids to Knowledge Construction: Building Mental Representations form Pictures and Words. In W. Schnotz & R.W. Kulhavy (Eds), Comprehension of Graphics.(pp. 125-138). Amsterdam, The Netherlands: North Holland.

McDaniel, M.A., & Waddill, P.J. (1994). The Mnemonic Benefit of Pictures in Test: Selective Enrichment for Differentially Skilled Readers. In W. Schnotz & R.W. Kulhavy (Eds), Comprehension of Graphics.(pp. 165-184). Amsterdam, The Netherlands: North Holland.

McLennan, J.P. (1987). Irrational beliefs in relation to self-esteem and depression. Journal of Clinical Psychology, 43, 89-91.

Munsen, W. W., Horan, J. J., Miano, L., & Stone, C. I. (1976). Another look at the Life Career Game. Pennsylvania Personnel and Guidance Association Journal, 4, 36-38.

Netscape Navigator 2.0 [Computer Software]. (1996). Mountainview, CA: Netscape Communications Corporation.

Newman, M. G., Consoli, A., & Taylor, C. B. (1997). Computers in assessment and cognitive behavioral treatment of clinical disorders: Anxiety as a case in point. Behavior Therapy, 28, 211-235.

Nielsen, D.M., Horan, J.J., Keen, B., St. Peter, C.C., Ceperich, S.D., & Ostlund, D., (1996) An attempt to improve Self-esteem by modifying specific irrational beliefs. Journal of Cognitive Psychotherapy, 10, 137-149.

Perry, R., Hechter, F., Menec, V., & Weinberg, L. (1993). Enhancing achievement motivation and performance in college students: An attributional retraining perspective. Research in Higher Education, 34, 687-723.

Tobin, J. (2001). Conflict management education: Essential skills for living. White Paper. Phoenix, AZ: KnowConflict, LTD. http://www.knowconflict.com/cl_papers_conflict_management.html

Van Overwalle, F., & DeMetsenaere, M. (1990). The effects of attribution-based intervention and study-strategy-training on academic achievement in college freshmen. British Journal of Educational Psychology, 60, 299-311.

Van Overwalle, F., Segebarth, K., & Goldchstein, M. (1989). Improving performance of freshmen through attributional testimonies from fellow students. British Journal of Educational Psychology, 59, 75-85.

Varenhorst, B. (1969). Learning the consequences of life's decisions. In J. D. Krumboltz & C. E. Thoresen (Eds.) Behavioral counseling: Cases and techniques (pp.306-319). New York: Holt, Rinehart, and Winston.

Weidenmann, B.(1994). Codes of Instructional Pictures. In W. Schnotz & R.W. Kulhavy (Eds), Comprehension of Graphics. (pp. 29-42). Amsterdam, The Netherlands: North Holland

Weiner, B. (1985). An attributional theory of achievement motivation and emotion. Psychological Review, 92, 548-573.