'The Delphi Technique: Making Sense of Consensus\r'

'A peer- analyseed electronic journal. procure is retained by the first or sole author, who grants right of first payoff to the hardheaded discernment, search & rating. Permission is tending(p) to distribute this article for nonprofit, studyal occasions if it is copied in its entirety and the journal is credited. Volume 12, subdue 10, August 2007 ISSN 1531-7714 The Delphi proficiency: qualification spirit Of Consensus Chia-Chien Hsu, The Ohio affirm University & Brian A. Sandford, okeh tell apart University The Delphi proficiency is a panopticly utilise and evaluate effectuate for take oning info from respondents inside their domain of expertness.The proficiency is designed as a sort out communication cognitive operation which aims to achieve a crossway of panorama on a prudish(postnominal) real-world unwrap. The Delphi booster has been employ in various handle of get hold of such as computer architectural planme think, exigencys sagacity, form _or_ system of g everywherenment determination, and resource example to surface a full execute of alternates, explore or expose vestigial assertions, as salutary as correlated sagaciousnesss on a exit spanning a wide range of disciplines. The Delphi proficiency is hearty suited as a system for consensus-building by using a serial of questionnaires delivered using ten-fold closed circuits to collect entropy from a panel of needed subjects. reconcile filling, succession frames for conducting and completing a field of battle, the initiative of mortified answer rates, and unintentionally guiding feedback from the respondent meeting argon aras which should be considered when designing and implementing a Delphi battlefield. The Delphi proficiency, mainly break awayed by Dalkey and Helmer (1963) at the Rand Corporation in the 1950s, is a widely drug ab ingestion and accepted mode for achieving product of eyeshot concerning real-world no esis addressed from experts inside sealed(a) subject argonas.Predicated on the rationale that, â€Å" twain heads atomic number 18 better than unrivaled, or… n heads ar better than one” (Dalkey, 1972, p. 15), the Delphi technique is designed as a collection communication work on that aims at conducting precise examinations and discussions of a item issue for the purpose of goal setting, policy investigation, or predicting the accompaniment of future events (Ulschak, 1983; Turoff & Hiltz, 1996; Ludwig, 1997). Common surveys try to distinguish â€Å"what is,” whereas the Delphi technique attempts to address â€Å"what could/should be” (Miller, 2006).In the literature, Delphi has been expend in various fields such as architectural plan planning, needs assessment, policy determination, and resource utilization. Delbecq, cutting edge de Ven, and Gustafson (1975) specifically paint a picture that the Delphi technique underside be utilis e for achieving the following objectives: 1. To determine or develop a range of possible program alternatives; 2. To explore or expose central assumptions or info leading to diametrical judgments; 3. To seek out tuition which whitethorn generate a consensus on the range of the respondent stem; 4.To correlate advised judgments on a topic spanning a wide range of disciplines, and; 5. To educate the respondent host as to the diverse and interrelated asp viperects of the topic (p. 11). CHARACTERISTICS OF THE DELPHI TECHNIQUE The Delphi technique is swell up suited as a room and order for consensus-building by using a serial publication of questionnaires to collect entropy from a panel of selected subjects (Dalkey & Helmer, 1963; Dalkey, 1969; Linstone & Turoff, 1975; Lindeman, 1981; Martino, 1983; Young & Jamieson, 2001).Delphi, in bank line to diametrical data gathering and summary techniques, employs multiple closed circuits designed to Practical Ass essment, seek & Evaluation, Vol 12, No 10 Hsu & Sandford, Delphi technique develop a consensus of opinion concerning a specific topic. Ludwig (1994) indicates: Iterations refer to the feedback bear on. The extremity was viewed as a series of capture outs; in several(prenominal)ly band every participant worked through a questionnaire which was returned to the researcher who collected, edited, and returned to every participant a financial expressment of the position of the whole group and the participant’s own position.A sum of money of comments made all(prenominal) participant awake(predicate) of the range of opinions and the reasons underlying those opinions (p. 55). to a greater extent specifically, the feedback process allows and encourages the selected Delphi participants to reassess their initial judgments intimately the education provided in previous iterations. Thus, in a Delphi ingest, the sequels of previous iterations regarding specific sta tements and/or items asshole change or be change by man-to-man panel members in later iterations based on their great power to redirect examination and assess the comments and feedback provided by the other Delphi panelists.Other nonable characteristics constitutive(a) with using the Delphi technique argon the ability to provide anonymity to respondents, a controlled feedback process, and the suitability of a mannequin of statistical abridgment techniques to interpret the data (Dalkey, 1972; Ludlow, 1975; Douglas, 1983). These characteristics argon designed to offset the shortcomings of military forceed means of pussying opinions obtained from a group fundamental interaction (i. e. , influences of dominant individuals, noise, and group pressure for conformity) (Dalkey, 1972).One of the first-string characteristics and advantages of the Delphi process is subject anonymity which skunk reduce the set up of dominant individuals which frequently is a concern when using group-based processes holdd to collect and synthe surface of it instruction (Dalkey, 1972). Additionally, the issue of confidentiality is facilitated by geographic dispersion of the subjects as well as the use of electronic communication such as e-mail to solicit and exchange instruction.As such, certain downsides associated with group dynamics such as function or coercion to conform or adopt a certain standstill locoweed be minimized (Helmer & Rescher, 1959; Oh, 1974; Adams, 2001). Controlled feedback in the Delphi process is designed to reduce the effect of noise. Based upon Dalkey (1972), noise is that communication which occurs in a group process which two distorts the data and deals with group and/or individual interests kinda than focusing on riddle solving.As a result, the reading demonstrable from this pleasant of communication for the closely part consists of bias not related to the purposes of the contract. Basically, the controlled feedback process con sists of a well organized summary of the precedent iteration intentionally distributed to the subjects which allows each participant an prospect to generate additive insights and more than thoroughly clarify 2 the information developed by previous iterations.Through the operation of multiple iterations, subjects are straits judgment to become more problem-solving oriented, to offer their opinions more insightfully, and to minimize the effects of noise. Finally, the ability to use statistical analysis techniques is a cause which further reduces the authorization of group pressure for conformity (Dalkey, 1972). More specifically, statistical analysis fag end run across that opinions generated by each subject of a Delphi cartoon are well represented in the final iteration because, â€Å"at the end of the exercise there whitethorn still be a important spread in individual opinions” Dalkey, 1972, p. 21). That is, each subject would remove no pressure, each real or perceived, to conform to rough other participant’s rejoinders that whitethorn break from obedience to amicable norms, customs, organizational culture, or standing within a profession. The tools of statistical analysis allow for an objective and fair analysis and summarization of the collected data. THE DELPHI forge Theoretically, the Delphi process discount be endlessly iterated until consensus is determined to hold in been achieved.However, Cyphert and Gant (1971), Brooks (1979), Ludwig (1994, 1997), and Custer, Scarcella, and Stewart (1999) saddle out that triple iterations are a good deal ample to collect the postulate information and to reach a consensus in almost occurrences. The following discussion, however, provides guidelines for up to 4 iterations in order to assist those who decide to use the Delphi process as a data collection technique when it is determined that additional iterations beyond three are needed or valuable. sharpshoot 1: In the first pear-shaped, the Delphi process traditionally begins with an open-ended questionnaire.The open-ended questionnaire serves as the cornerstone of soliciting specific information virtually a pith area from the Delphi subjects (Custer, Scarcella, & Stewart, 1999). After receiving subjects’ responses, investigators need to commute the collected information into a well-structured questionnaire. This questionnaire is used as the survey instrument for the certify round of data collection. It should be tell that it is both an acceptable and a joint modification of the Delphi process format to use a structured questionnaire in en self-aggrandizing 1 that is based upon an extensive go off of the literature.Kerlinger (1973) remark that the use of a change Delphi process is earmark if basic information concerning the intention issue is available and usable. Round 2: In the second round, each Delphi participant receives a second questionnaire and is asked to review the items summarized by the investigators based on the information provided in the first round. Accordingly, Delphi panelists may be need to rate or â€Å"rank-order Practical Assessment, search & Evaluation, Vol 12, No 10 Hsu & Sandford, Delphi Technique items to establish preliminary priorities among items.As a result of round two, areas of disagreement and agreement are identified” (Ludwig, 1994, p. 54-55). In some cases, Delphi panelists are asked to state the rationale concerning rating priorities among items (Jacobs, 1996). In this round, consensus begins forming and the actual outcomes can be presented among the participants’ responses (Jacobs, 1996). 3 Rescher (1959), Klee (1972), and Oh (1974) concur that choosing individuals who are simply informal concerning the target issue is not sufficient nor recommended.Considering the necessity of selecting the most qualified individuals, Delbecq, Van de Ven, and Gustafson (1975) specifically state that th ree groups of deal are well qualified to be subjects of a Delphi study. The authors recommend: â€Å"(1) the top counsel decision makers who pull up stakes utilize the outcomes of the Delphi study; (2) the professional staff members together with their aver group; and (3) the respondents to the Delphi questionnaire whose judgments are being seek” (p. 85). Delphi subjects should be highly trained and capable within the specialized area of knowledge related to the target issue.Investigators need to well examine and seriously consider the qualifications of Delphi subjects. Oh (1974) indicates that choosing take into account subjects is chiefly based on the judgment and discretion of the pencil lead investigators. Jones and Twiss (1978) state that the principal investigators of a Delphi study should identify and select the most take into account individuals through a nomination process. Ludwig (1994) as well states that, â€Å"solicitation of nominations of cogniz e and respected individuals from the members within the target groups of experts was recommended” (p. 2). Generally, the pool of selecting possible Delphi subjects is likely to use positional leaders (Kaplan, 1971; Ludwig, 1994), to follow a review of authors of publications in the literature (Meyer, 1992; Miller, 2001), and/or to make contacts with those who have firsthand relationships with a item issue (Jones, 1975; Anderson & Schneider, 1993). The latter basically consists of individuals who are primary stakeholders with various interests related to the target issue or research effort.Concerning the appropriate number of subjects to involve in a Delphi study, Delbecq, Van de Ven, and Gustafson (1975) recommend that researchers should use the minimally sufficient number of subjects and should seek to depose the results through follow-up geographic expeditions. Ludwig (1994) notes that the number of experts used in a Delphi study is â€Å"generally determined by the number required to constitute a representative pooling of judgments and the information processing capability of the research team” (p. 52). However, what constitutes an optimal number of subjects in a Delphi study never reaches a consensus in the literature.Delbecq, Van de Ven, and Gustafson (1975) adumbrate that ten to fifteen subjects could be sufficient if the background of the Delphi subjects is homogeneous. In contrast, if various reference groups are tough in a Delphi study, more subjects are judge to Round 3: In the third round, each Delphi panelist receives a questionnaire that includes the items and ratings summarized by the investigators in the previous round and are asked to revise his/her judgments or â€Å"to localise the reasons for be outside the consensus” (Pfeiffer, 1968, p. 52). This round gives Delphi panelists an probability to make further clarifications of both the information and their judgments of the relative importance of the items. How ever, compared to the previous round, exactly a slight increase in the leg of consensus can be expected (Weaver, 1971; Dalkey & Rourke, 1972; Anglin, 1991; Jacobs, 1996). Round 4: In the one-fourth and often final round, the list of remaining items, their ratings, minority opinions, and items achieving consensus are distributed to the panelists.This round provides a final opportunity for participants to revise their judgments. It should be remembered that the number of Delphi iterations depends largely on the degree of consensus sought by the investigators and can go from three to five (Delbecq, Van de Ven, Gustafson, 1975; Ludwig, 1994). Subject Selection Regarding the pickax of subjects for a Delphi study, choosing the appropriate subjects is the most important step in the entire process because it directly relates to the role of the results generated (Judd, 1972; Taylor & Judd, 1989; Jacobs, 1996).Since the Delphi technique focuses on eliciting expert opinions ove r a short period of meter, the endurance of Delphi subjects is generally dependent upon the disciplinary areas of expertness required by the specific issue. Regarding all set standards of selecting Delphi subjects, there is, in fact, no exact criterion currently listed in the literature concerning the selection of Delphi participants. That is, â€Å"throughout the Delphi literature, the description of [Delphi subjects] has remained ambiguous” (Kaplan, 1971, p. 24).Regarding the criteria used to guide the selection of Delphi subjects, individuals are considered eligible to be invited to go into in a Delphi study if they have somewhat related backgrounds and experiences concerning the target issue, are capable of contributing helpful inputs, and are exiting to revise their initial or previous judgments for the purpose of reaching or attaining consensus (Pill, 1971; Oh, 1974). Helmer and Practical Assessment, enquiry & Evaluation, Vol 12, No 10 Hsu & Sandford, De lphi Technique be needed.Witkin and Altschuld (1995) note that the label sizing of a Delphi panel is generally under 50, but more have been employed. Ludwig (1997) documents that, â€Å"the majority of Delphi studies have used betwixt 15 and 20 respondents” (p. 2). In sum, the size of Delphi subjects is variable (Delbecq, Van de Ven, & Gustafson, 1975). If the seek size of a Delphi study is too small, these subjects may not be considered as having provided a representative pooling of judgments regarding the target issue.If the sample size is too large, the drawbacks inherent within the Delphi technique such as effectivenessly low response rates and the obligation of large squeezes of time by the respondents and the researcher(s) can be the result. Time Requirements Conducting a Delphi study can be time-consuming. Specifically, when the instrument of a Delphi study consists of a large number of statements, subjects will need to dedicate large blocks of time to comple te the questionnaires.Delbecq, Van de Ven, and Gustafson (1975), Ulschak (1983), and Ludwig, (1994) recommend that a minimum of 45 days for the memorial tablet of a Delphi study is necessary. With regard to the time management between iterations, Delbecq et al. (1975) note that big(p) two weeks for Delphi subjects to respond to each round is encouraged. Ludwig (1994) indicates, â€Å"a drawback to Delphi was that the questionnaire method may slake the process greatly as several days or weeks may pass between rounds” (p. 54).More specifically, since developing the instrument, collecting the data, and administering the questionnaire are interconnected between iterations, ensuring Delphi subjects respond to the investigators on time does in many shipway either promote or interdict the ability of the investigators in analyzing the data, developing a new instrument based upon the prior responses, and distributing subsequent questionnaires in a seasonably fashion. These are c hallenging aspects of conducting a Delphi study and do require proper planning and management. The use and prevalence of electronic technologies (i. e. e-mail, teleconferencing, and so forth ) may facilitate those who are enkindle in using the Delphi technique. Witkin and Altschuld (1995) note that electronic applied science provides an opportunity for individuals to more tardily employ the Delphi process by victorious advantages of, â€Å"(1) the storage, processing, and speed of transmission capabilities of computers; (2) the maintenance of respondent anonymity, and; (3) the potential for rapid feedback” (p. 204). Data synopsis Regarding data analysis, decision rules must be established to assemble and organize the judgments and insights provided by Delphi subjects.However, the kind and type 4 of criteria to use to both define and determine consensus in a Delphi study is subject to interpretation. Basically, consensus on a topic can be decided if a certain office of t he votes falls within a plus range (Miller, 2006). One criterion recommends that consensus is achieved by having 80 percent of subjects’ votes fall within two categories on a seven-point outperform (Ulschak, 1983). Green (1982) suggests that at least 70 percent of Delphi subjects need to rate three or higher on a four point Likert-type scale and the median has to be at 3. 5 or higher. Scheibe, Skutsch, and Schofer (1975) reveal that the use of percentage streaks is inadequate. They suggest that a more reliable alternative is to measure the stability of subjects’ responses in successive iterations. In the Delphi process, data analysis can involve both qualitative and quantitative data. Investigators need to deal with qualitative data if classic Delphi studies, which use open-ended questions to solicit subjects’ opinions, are conducted in the initial iteration.Subsequent iterations are to identify and hopefully achieve the want take of consensus as well as a ny changes of judgments among panelists. The major statistics used in Delphi studies are measures of central tendency (means, median, and mode) and level of dispersion (standard deviation and inter-quartile range) in order to present information concerning the collective judgments of respondents (Hasson, Keeney, & McKenna, 2000). Generally, the uses of median and mode are favored. However, in some cases, as manifested by Murray and Jarman (1987), the mean is as well workable.Witkin (1984) questions the appropriateness of using the mean to measure the subjects’ responses if scales used in Delphi studies are not delineated at equal intervals. In the literature, the use of median score, based on Likert-type scale, is strongly favored (Hill & Fowles, 1975; Eckman, 1983; Jacobs, 1996). As Jacobs (1996) states, â€Å"considering the anticipated consensus of opinion and the skewed expectation of responses as they were compiled, the median would inherently appear best su ited to reflect the resultant convergence of opinion” (p. 57).The use of mode is also suitable when reporting data in the Delphi process. Ludwig (1994) specifically addressed that â€Å"the Delphi process has a tendency to create convergence, and though this was ordinarily to a single point, there was the possibility of polarization or clustering of the results somewhat two or more points. In these instances, the mean or median could be misleading” (p. 57). CONSIDERING DELPHI SHORTCOMINGS AND WEAKNESSES potentiality of Low rejoinder Rates Due to the multiple feedback processes inherent and integral to the concept and use of the Delphi process, thePractical Assessment, question & Evaluation, Vol 12, No 10 Hsu & Sandford, Delphi Technique potential exists for low response rates and seek to maintain robust feedback can be a challenge. â€Å"In the Delphi technique, [poor response rate] is enlarge fourfold because a maximum of four surveys may be sent to t he akin panelists” (Witkin & Altschuld, 1995, p. 196). If a certain portion of the subjects block their responses during various stages of the Delphi process, the tint of information obtained could be discounted or at least critically scrutinized.As such, Ludwig (1994) specifically addresses subject motivation as the key to the successful implementation of a Delphi study and investigators need to play an brisk role in this area to help ensure as high a response rate as possible. enjoyment of Large Blocks of Time The Delphi technique can also be time-consuming and laborious. distant other data collection techniques such as the telephone survey and the face to face administration, which can be simultaneously conducted by a group of people and can be completed in a short period of time if the sample size is small, the Delphi technique is terative and sequential. As a result, the necessity of taking large block of time to successively complete a Delphi process is inesc apable. Ludwig (1994) indicates that, â€Å"a drawback to Delphi was that the questionnaire method may slow the process greatly as several days or weeks may pass between rounds” (p. 54). Optimally speaking, the iteration characteristics of the Delphi process provide the opportunities for investigators and subjects to improve the trueness of the results.In contrast, the same characteristic also increases the work load of investigators and the amount of time needed to successfully complete the data collection process (Cunliffe, 2002). Potential of Molding Opinions The iteration characteristics of the Delphi technique can potentially enable investigators to disgorge opinions (Altschuld, 2003). An experiment, conducted by Scheibe, Skutsch, and Schofer (1975), indicated that Delphi subjects would rate their responses differently later onwards receiving a distorted feedback.Dalkey and Helmer (1963) also noted that, â€Å"some ‘leading’ by the experimenters necess arily resulted from the selection of the information supplied” (p. 467). Moreover, Cyphert and Gant (1971) illustrated that a statement in their study was initially rated below average. However, Delphi subjects rated the statement above average after receiving false feedback. Therefore, Cyphert and Gant (1971) concluded that the Delphi technique could, â€Å"be used to mold opinion as well as to collect [data]” (p. 273).Indeed, â€Å" shrewd pressure to conform with group ratings” was one of the major drawbacks in the Delphi technique (Witkin & Altschuld, 1995, p. 188). Delphi investigators need to be cognizant, exercise caution, and implement the proper safeguards in dealing with this issue. 5 Potential of Identifying General Statements vs. Specific Topic cerebrate Information An assumption concerning Delphi participants is that they are equivalent weight in knowledge and experience (Altschuld & Thomas, 1991). However, this assumption might not be ju stified.More specifically, the expertise of Delphi panelists could be unevenly distributed, especially in the field of high engine room (Marchant, 1988; Altschuld & Thomas, 1991). â€Å" many panelists may have much more in-depth knowledge of certain topics, whereas other panelists are more knowledgeable about different topics” (Altschuld & Thomas, 1991, p. 187). Therefore, subjects who have less in-depth knowledge of certain topics are unable to specify the most important statements which have been identified by those subjects who possess in-depth knowledge concerning the target issue.The outcomes of a Delphi study could be the results of identifying a series of general statements rather than an in-depth exposition of the topic (Altschuld & Thomas, 1991). SUMMARY The Delphi technique provides those involved or interested in engaging in research, valuation, fact-finding, issue exploration, or discovering what is actually known or not known about a specific to pic a flexible and adaptable tool to gather and analyze the needed data. Subject selection and the time frames for conducting and completing a Delphi study are two areas which should be considered cautiously prior to initiating the study.The additional precautions concerning low response rates, unintentionally guiding feedback, and surveying panelists about their limited knowledge of the topic rather than soliciting their expert judgments should also be build into the design and implementation of the study. The Delphi technique has and will continue to be an important data collection methodology with a wide variety of coats and uses for people who want to gather information from those who are immersed and imbedded in the topic of interest and can provide real time and real-world knowledge. REFERENCES Adams, S. J. (2001).Projecting the next decade in synthetic rubber management: A Delphi technique study. nonrecreational Safety, 46 (10), 26-29. Altschuld, J. W. (2003). Delphi tech nique. Lecture, Applied evaluation design. The Ohio State University. Altschuld, J. W. , & Thomas, P. M. (1991). Considerations in the application of a modified scree test for Delphi survey data. Evaluation Review, 15 (2), 179-188. Practical Assessment, inquiry & Evaluation, Vol 12, No 10 Hsu & Sandford, Delphi Technique Anderson, D. H. , & Schneider, I. E. (1993). Using the Delphi process to identify significant recreation research-based innovations.Journal of park and amusement Administration, 11 (1), 25-36. Anglin, G. L. (1991). Instructional technology past, present and future. Englewood, CO: Libraries Unlimited Inc. Brooks, K. W. (1979). Delphi technique: Expanding applications. North Central Association Quarterly, 54 (3), 377-385. Cunliffe, S. (2002). Forecasting risks in the tourism industriousness using the Delphi technique. Tourism, 50 (1), 31-41. Custer, R. L. , Scarcella, J. A. , & Stewart, B. R. (1999). The modified Delphi technique: A rotatio nal modification. Journal of vocational and Technical Education, 15 (2), 1-10.Cyphert, F. R. , & Gant, W. L. (1971). The Delphi technique: A case study. Phi Delta Kappan, 52, 272-273. Dalkey, N. C. (1969). An experimental study of group opinion. Futures, 1 (5), 408-426. Dalkey, N. C. (1972). The Delphi method: An experimental study of group opinion. In N. C. Dalkey, D. L. Rourke, R. Lewis, & D. Snyder (Eds. ). Studies in the quality of aliveness: Delphi and decision-making (pp. 13-54). Lexington, MA: Lexington Books. Dalkey, N. C. , & Helmer, O. (1963). An experimental application of the Delphi method to the use of experts. Management Science, 9 (3), 458-467.Dalkey, N. C. , & Rourke, D. L. (1972). Experimental assessment of Delphi procedures with group mensurate judgments. In N. C. Dalkey, D. L. Rourke, R. Lewis, & D. Snyder (Eds. ). Studies in the quality of life: Delphi and decision-making (pp. 55-83). Lexington, MA: Lexington Books. Delbecq, A. L. , Van de Ven, A. H. , & Gustafson, D. H. (1975). Group techniques for program planning. Glenview, IL: Scott, Foresman, and Co. Douglas, D. C. (1983). A comparative study of the effectiveness of decision making processes which utilize the Delphi and leaderless group methodologies.unpublished doctorial dissertation, The Ohio State University, Columbus. Eckman, C. A. (1983). phylogenesis of an instrument to evaluate intercollegiate acrobatic coaches: A modified Delphi study. Unpublished doctoral dissertation, West Virginia University, Morgantown. Green, P. J. (1982, March). The content of a college-level exterior leadership course. Paper presented at the convocation of the Northwest District Association for the American 6 Alliance for Health, Physical Education, Recreation, and Dance, Spokane, WA. Hasson, F. , Keeney, S. , & McKenna, H. (2000). question guidelines for the Delphi survey technique.Journal of Advanced Nursing, 32 (4), 1008-1015. Helmer, O. , & Rescher, N. ( 1959). On the epistemology of the inexact science. Management Science, 6, 25-53. Hill, K. Q. , & Fowles, J. (1975). The methodological deserving of the Delphi forecasting technique. proficient Forecasting and affectionate Change, 7, 179-192. Jacobs, J. M. (1996). Essential assessment criteria for physical education teacher education programs: A Delphi study. Unpublished doctoral dissertation, West Virginia University, Morgantown. Jones, C. G. (1975). A Delphi evaluation of agreement between organizations. In H. A. Linstone, & M.Turoff (Eds. ). The Delphi method: Techniques and applications (pp. 160-167). Reading, MA: Addison-Wesley Publishing Company. Jones, H. , & Twiss, B. C. (1978). Forecasting technology for planning decision. London, UK: Macmillan Press Ltd. Judd, R. C. (1972). Use of Delphi methods in higher education. Technological Forecasting and amicable Change, 4 (2), 173-186. Kaplan, L. M. (1971). The use of the Delphi method in organizational communica tion: A case study. Unpublished master’s thesis, The Ohio State University, Columbus. Kerlinger, F. N. (1973). Foundations of behavioral research. in the altogether York: Holt, Rinehart, and Winston, Inc. Klee, A. J. (1972). The utilization of expert opinion in decision-making. AICHE Journal, 18 (6), 1107-1115. Lindeman, C. A. (1981). Priorities within the wellness care system: A Delphi survey. Kansas City, MO: American Nurses’ Association. Linstone, H. A. , & Turoff, M. (1975). Introduction. In H. A. Linstone, & M. Turoff (Eds. ). The Delphi method: Techniques and applications (pp. 3-12). Reading, MA: Addison-Wesley Publishing Company. Ludlow, J. (1975). Delphi inquiries and knowledge utilization. In H. A. Linstone, & M. Turoff (Eds. ).The Delphi method: Techniques and applications (pp. 102-123). Reading, MA: Addison-Wesley Publishing Company. Ludwig, B. G. (1994). Internationalizing Extension: An exploration of the characteristics evident in a state un iversity Extension system that achieves internationalization. Unpublished doctoral dissertation, The Ohio State University, Columbus. Practical Assessment, Research & Evaluation, Vol 12, No 10 Hsu & Sandford, Delphi Technique Ludwig, B. (1997). Predicting the future: behave you considered using the Delphi methodology? Journal of Extension, 35 (5), 1-4. Retrieved November 6, 2005 from http://www. oe. org/joe/1997october/tt2. html Marchant, E. W. (1988). Methodological problems associated with the use of the Delphi technique: Some comments. Fire Technology, 24 (1), 59-62. Martino, J. P. (1983). Technological forecasting for decision making. New York: North-Holland. Meyer, J. H. (1992). Rethinking the brainpower of colleges whose roots have been in agriculture. Davis, CA: University of California. Miller, G. (2001). The development of indicators for sustainable tourism: Results of a Delphi survey of tourism researchers. Tourism Management, 22, 351-362. Miller, L. E. (2006, October).Determining what could/should be: The Delphi technique and its application. Paper presented at the meeting of the 2006 annual meeting of the Mid-Western Educational Research Association, Columbus, Ohio. Murray, W. F. , & Jarman, B. O. (1987). Predicting future trends in adult fittingness using the Delphi approach. Research Quarterly for apply and Sport, 58 (2), 124-131. Oh, K. H. (1974). Forecasting through hierarchal Delphi. Unpublished doctoral dissertation, The Ohio State University, Columbus. Pill, J. (1971). The Delphi method: Substance, context, a critique and an annotated bibliography.Socio-Economic Planning Science, 5, 57-71. 7 Scheibe, M. , Skutsch, M. , & Schofer, J. (1975). Experiments in Delphi methodology. In H. A. Linstone, & M. Turoff (Eds. ). The Delphi method: Techniques and applications (pp. 262-287). Reading, MA: Addison-Wesley Publishing Company. Taylor, R. E. , & Judd, L. L. (1989). Delphi method applied to tourism. In S. Witt, &# 038; L. Moutinho, (Eds. ). Tourism marketing and management handbook. New York: Prentice Hall. Turoff, M. , & Hiltz, S. R. (1996). Computer based Delphi process. In M. Adler, & E. Ziglio (Eds. ).Gazing into the oracle: The Delphi method and its application to cordial policy and public health (pp. 56-88). London, UK: Jessica Kingsley Publishers. Ulschak, F. L. (1983). clement resource development: The theory and practice of need assessment. Reston, VA: Reston Publishing Company, Inc. Weaver, W. T. (1971). The Delphi forecasting method. Phi Delta Kappan, 52 (5), 267-273. Witkin, B. R. (1984). Assessing needs in educational and social programs. San Francisco, CA: Jossey-Bass Publishers. Witkin, B. R. , & Altschuld, J. W. (1995). Planning and conducting needs assessment: A practical guide.Thousand Oaks, CA: sage-green Publications, Inc. Young, S. J. , & Jamieson, L. M. (2001). Delivery methodology of the Delphi: A resemblance of two approaches. Journal of Park an d Recreation Administration, 19 (1), 42-58. Citation Hsu, Chia-Chien & Sandford, Brian A. (2007). The Delphi Technique: Making Sense of Consensus. Practical Assessment Research & Evaluation, 12(10). Available online: http://pareonline. net/getvn. asp? v=12&n=10 Editors timber: Another paper on the Delphi Technique that appeared in Practical Assessment Research & Evaluation is: Yousuf, Muhammad Imran (2007).Using Experts’ Opinions through Delphi Technique. Practical Assessment Research & Evaluation, 12(4). Available online: http://pareonline. net/getvn. asp? v=12&n=4 . Authors Chia-Chien Hsu Post-doctoral Studies The Ohio State University 393 Schrock Road Worthington, OH 43085 Practical Assessment, Research & Evaluation, Vol 12, No 10 Hsu & Sandford, Delphi Technique Tel: (614) 885-0763 e-mail: hsu. 127 [at] osu. edu Brian A. Sandford Assistant Professor 214 Willard Hall occupational Education Oklahoma State University Stillwater, Oklah oma 74074 405-744-3461 brian. sandford [at] okstate. edu 8\r\n'