Document(s) 17 of 19

Early in April 1992, 14 experts in library and information science met in Ottawa with members of the Information Sciences and Systems Division of the International Development Research Centre (IDRC) to begin a unique brainstorming experiment. During the previous 20 years, IDRC and other agencies funded many efforts to collect, catalogue, maintain, and disseminate information assumed relevant to research in developing countries.

As a result, information centres and networks proliferated and their funding requirements grew. Alas, the funding sources did not. Although information remained a "good thing," the time had come to reconsider the belief that more is better and to determine how the usefulness of information projects in developing countries might be assessed more objectively.

As researchers in decision-making know, it is extremely difficult to determine the value of information except in the most contrived and trivial situations (e.g., see Arkes and Hammond 1986; Simon 1976; Thorngate and Ferguson 1977; Tversky and Kahneman 1974). Information cannot be valued except for its contribution to knowledge.

Yet, as 100 years of research in cognitive psychology and pedagogy have shown, information almost never contributes to knowledge in an incremental or accretive way. Instead, it almost always combines in a manner akin to complex chemical reactions, generating knowledge as an emergent property.

In short, information does not "add up" to knowledge; there is no monotonic or simple relation between the amount of information available and the amount of knowledge obtained. To make matters more complex, the value of knowledge is more related to its quality than its quantity and is highly contingent on one's goals and circumstances. Such complexity makes it difficult to employ simple quantitative criteria for deciding which information projects will receive funding and which will not. Of course, few would support a library with no books or patrons, or a journal with no articles or subscribers, but beyond such rare extremes there are no obvious guides to judge the relative merits of information projects.

Correlations between the number of documents, the number of users, and the quantity, quality, and value of knowledge transferred are low. As a result, criteria based on such numbers are fallible and become more so as the number of competitors for limited funds increases (see Thorngate 1988; Thorngate and Carroll 1987, 1991). Other criteria such as "track record," that are correlated with prior opportunities, soon lead to the development of pampered cliques and to the exclusion of innovators (Thorngate and Hotta 1990).

Requirements of standards (e.g., all bibliographic proposals must follow Library of Congress classifications) are as limiting as the standards themselves (consider the limits of ASCII, QWERTY or VHS outside North America). Imposition of partnerships or coordinated activities (e.g., a proposal must contain related projects in at least five countries) increase the risk that after all is said nothing is done.

Faced with the daunting task of developing criteria for evaluating the relative merits of information projects in developing countries, Martha Stone, Director General of the Information Sciences and Systems Division of IDRC initiated a series of activities on how to assess the impact of information on development. Her first step was to seek advice from outside experts, and the recursive experiment was born.

Following their April workshop, the 14 experts continued brainstorming about assessment criteria via a computer conference, which lasted until mid November. The contents of the computer conference were summarized by its designated facilitator, Michel Menou, for a subsequent face-to-face conference in Nairobi in March 1993, which included more representatives from developing countries.

Their reactions to the summary and further suggestions for criteria are now being combined with papers from many of the 14 original participants in a book on assessment indicators for the impact of information on development. In addition, the assessment framework is now being tested in several information projects in developing regions.

Computer Conferencing as a Forum

The traditional alternative of distributing papers via mail was plagued by its traditional problems: slowness, unreliability, and expense. A less-traditional alternative was thus considered. In light of advances in computer communication it seemed reasonable to attempt an extended brainstorming session using a computer conference.

IDRC had first examined the appropriateness of computer conferencing for development purposes in 1981. In that year, the former Information Sciences Division organized a workshop on Computer-Based Conferencing Systems for Developing Countries (see Balson et al. 1981).

As a result of this workshop, IDRC's Telematics Program was initiated to support research activities related to developing-country institution's access to and utilization of computer-based conferencing techniques. The first computer conference experiment undertaken by the Telematics Program was an international discussion of bioconversion of lignocellulosics fuel, fodder and food in 1983 (see Balson 1985). A second conferencing experiment was undertaken in the mid 1980s to support a Untied Nations University-sponsored Brucellosis research network in Latin America.

The experiments indicated that three conditions were necessary for a successful computer conference:

• There must be a focused topic of discussion;
• There must be a sufficient number of participants with the time, motivation, and financial resources to participate effectively; and
• There must be minimal technical difficulties.

North American and European participants were expected to have little trouble connecting to the COSY system from their local work place or home. The few participants from developing countries without reliable connections to COSY were periodically sent printouts via courier and asked to courier back papers or disks in response. Their responses were transferred to COSY for distribution to the participants with online connections.

If the promises of the medium were fulfilled, the computer conference could be at least as effective as its face-to-face counterpart. It would also be far less expensive. If the promises of the medium were unfulfilled, however, much time and money would be wasted. There is still much to be learned about computer conferencing, its costs, benefits, products, and processes. There was thus good reason to monitor and to evaluate the computer conference.

How is it possible to evaluate a computer conference? What criteria should be used to assess its strengths and weaknesses? Such questions are recursively familiar, highlighting the irony of agonizing over criteria to assess how others use the medium to agonize again over criteria to assess how others use information resources.

Irony aside, however, the principal author was contracted to monitor computer conference activity and expense, and to obtain participants' reactions to the medium and its messages. Several Carleton University graduate students assisted in monitoring conference activity, generating regular printouts of new conference postings, and in recording summary information about them. Conference expenses were recorded at IDRC. Participants' reactions to the conference were obtained by questionnaire. There was, of course, no comparison group of face-to-face, conference-call, or letter-only discussion to afford relative evaluations of the computer conference. What follows should, therefore, be seen as a formative evaluation or a case study rather than a comparative assessment of a medium or its use.

Method

Each participant was asked to post items under what he or she believed to be the most relevant topic title. Thirteen postings were considered relevant to more than one topic, the facilitator cross-posted 10 of them, and parts of the remaining three. One posting was obviously out of place and was transferred to a more relevant topic heading.

Participants

Hoping to benefit from the participants' brainstorms during the next phase of the indicators project, IDRC convened the face-to-face workshop, organized the computer conference and followed the conference dialogue. IDRC staff posted messages to IIDCONF only when necessary for clarification of some organizational or technical matter; only three such messages were posted.

The computer conference was conceived as an 8-month roundtable discussion. Each participant was encouraged to discuss the conference with local colleagues and to encourage them in turn to submit postings to the conference as conference guests. Twelve such guests obliged: six of them once, four twice, and two five times for a total of 24 message postings.

During the April workshop, participants were given the opportunity to be trained on the COSY system. John Black, the Librarian at the University of Guelph, provided much of the training by lecture, demonstration, and hands-on training in a computer room designed for the purpose. Several participants posted trial messages (e.g., "This is a test") during this time. As a result, data for the first month of the conference are somewhat unrepresentative and should be interpreted with caution.

It should be noted that participants could send each other an e-mail (electronic mail) via COSY or via other e-mail systems they utilized. Many did. No records could be collected from COSY of how often this alternative means of communication was used for conference purposes. So two questionnaire items were included to gain some rough estimate of this e-mail usage. Participants accessed COSY via packet switched networks, INET, Internet, and via the offline method previously mentioned.

Data Collection

• Topic title,
• Message number,
• COSY account of the person who posted it,
• Number of characters in message,
• Date of posting,
• Referent message (if message was a reply to a previous one), and
• An optional message title.

Questionnaire

Results

Postings

Table 1. Subconference postings by month.
Conference	Apr	May	Jun	Jul	Aug	Sep	Oct	Nov	Total
Benefits (Ben)	21	8	4	4	11		18	18	84
Calculation (Cal)	8			2	1			1	12
Digests		2		5	9	6	11		33
General	13	6	9	41	26	3			98
Indicators	2		1	7	8	1	4	4	27
Literature	4	1			7				12
Policies	13				4				17
Projects	6		1					1	8
Research	1								1
Other	22	9	10	14	22	7	15	7	106
Total	90	26	25	73	88	17	48	31	398

As seen in Table 1, the most popular three subconferences were Benefits, General, and Other. In contrast, the least popular included Research, Projects, Literature, Calculation, and Policies. These popularity differences may reflect confusion about the content of subconferences. Alternatively, they may reflect an academic tendency to remain general and abstract in discussion and to avoid postings about more specific suggestions.

Table 2 shows the contributions made by each participant to each subconference (to preserve anonymity, participants are listed in random order). There is no strong tendency for participants to specialize in one or two topics. It is obvious, however, that some participants were more active than others. The facilitator accounted for 49% of the IIDCONF postings. Six others together accounted for an additional 37%. The remaining nine participants and 12 guests generated only 14% of the postings.

Table 2. Subconference postings by participant.
	A	B	C	D	E	F	G	H	I	J	K	L	M	N	O	P	Xa
Benefits	8				2	1	13	2	7	2	43	1	1		3		1
Calculation		1			1			3		1	4						2
Digests	3	6				1		1	2	4	22
General	6	1	2			7	2	17	4	6	40				1		7
Indicators	3				1				1	6	14	1
Literature	2							1	2		5						2
Policies	1				2	2			1		9	1				1
Projects									1		5	1					1
Research											1
Other	7	16			3	1	6	2	9	1	52			3			6
Total	30	24	2	0	9	12	21	26	27	20	195	4	1	3	4	1	19
Lengthb	48	207	174	na	228	312	78	139	140	44	176	177	137	343	471	29	225
% Re:c	93	58	0	na	78	25	33	88	56	45	53	0	0	0	50	100	0
a X = number of messages from others (guests, etc.). b length = average number of words per message/10. c % Re: = percentage of messages that were replies to another message.

Like face-to-face conferences, the IIDCONF seems to have been dominated by a minority — 25% of the participants accounted for 86% of the postings. The average number of characters typed (a simple measure of message length) shows a similar pattern. The seven most active participants (25%) generated 78% of the printout.

Further analyses were conducted to examine how much participants interacted. At one extreme, no participant would respond to anyone's postings, and no interaction would occur. At the other extreme, every posting would stimulate everyone to respond and interaction would increase exponentially. The "re:" feature of COSY allowed some estimate of the postings stimulated by others. The penultimate row of Table 2 shows what percentage of each participant's postings were responses to other postings. Just over half of them were (53%), an indication of a healthy exchange.

Costs

First were the invoices sent to IDRC by the University of Guelph for COSY user IDs ($8 per month each), COSY connect charges ($8 per hour), and packet-switch linkage charges (also $8 per hour). Second were Bell Canada invoices for INET connections to COSY for the four participants who required this service. Table 3 shows a monthly summary of these charges for the invoices available. Each Guelph invoice covered a calendar month. The Bell invoices went from mid-month to mid-month, so were interpolated to align billing periods with those of Guelph (e.g., invoice for June = 1/2 the 16 May to 15 June invoice + 1/2 the 15 June to 15 July invoice)

Not surprisingly, Table 3 reveals a strong association between monthly costs and monthly posting rates (see Table 1). It cost an average of $8.88 in COSY and communication charges for each of the 398 postings, or about 60 cents per copy sent. This compares favourably with charges for airmail stamps.

Indirect costs of the computer conference are difficult to estimate, but they are probably not trivial. For example, the technical support provided by David Balson and John Black was substantial, as was the time taken by many participants to continue their discussion. By comparison, however, substantial clerical assistance is usually required for a face-to-face conference, and participants take almost as much time talking and listening as they do reading and writing.

Table 3. Conference and communication costs.
	Apr	May	Jun	Jul	Aug	Sep	Oct	Nov	Total
COSY/Datapac	$643	374	292	354	396	253	336	275	$2,923
INET	na	45	86	100	107	103	88	82	611
Total	643	419	378	454	503	356	424	357	$3,534

Questionnaire

Six of the 11 completed questionnaires were faxed; one was missing the last two pages of the questionnaire and two had several blurred passages. In addition, some of the respondents skipped some of the questions. These factors account for the varying totals of responses to the questions. For ease of presentation, each question is reproduced as it appeared on the questionnaire followed by summaries of the responses. Quoted comments are given in random order to preserve anonymity.

Background and Usage Items

1. Before the conference, in how many computer-mediated conferences had you participated?
   none = 7 respondents
   one = 2
   two = 0
   three= 1
   more than three= 1

2. Before the conference, how often had you used e-mail?
   never= 4 respondents
   1–3 times per week = 2
   4–6 times per week = 1
   7–10 times per week = 0
   more that 10 = 4

3. On average, how much time did you spend using COSY each week? (Best guess).
   Average = 1.6 hour(s) per week; range = 0.1–4.0 hours
   Where did you find the time?
   I used spare time at work= 5 respondents
   I used spare time at home= 3
   I took time from (elsewhere) = 3

4. On average, how many times each week each week did you login to COSY? (Best guess)
   Average = 1.9 times per week; range = 0–6

5. On average, what percentage of the times that you logged into COSY didyou...
   check for and read new conference entries? Average = 81% (range = 5–100%)
   write at least one new conference entry? 25% (0–50)
   check for and read new e-mail? 68% (0–100)
   write at least one new e-mail message? 25% (0–80)

6. Between the beginning of the conference (April 1992) and today, what percentage of the conference messages did you read? (Best guess).
   Average = 80% (range = 15–100%)

7. Between the beginning of the conference (April 1992) and today, how many conference messages did you write? (Best guess).
   Average = 26 (range = 2–179)
   Actual average = 30 (range = 1–197)

8. Between the beginning of the conference (April 1992) and today, how many private e-mail messages concerning the conference did you receive from conference participants? (Best guess).
   Average = 23 (range = 0–75)

9. Between the beginning of the conference (April 1992) and today, how many private e-mail messages concerning the conference did you write to conference participants? (Best guess).
   Average = 23 (range = 0–100)

10. All things considered, do you think the COSY conference facility or electronic mail facility was the more useful for developing and exchanging ideas?
    4 = conference (i.e., four participants chose "conference")
    3 = e-mail
    1 = equal, they complement each other
    3 = no comment
    Conference was more useful because—
    "It exposed ideas for all to consider."
    "It allowed for contribution of different ideas on a topic with a time for reflection; others are able to add their own ideas as well."
    "We exchanged most of the messages with it."
    E-mail was more useful because—
    "The conference facility was cumbersome; for example, adding a comment to a message read earlier is awkward without the ability to search conference by full text or without an index."
    "E-mail seemed more user friendly."
    "Both were clumsy."
    "It provided a more controlled communication environment and was more conducive to transparent and more frank interactions."

Technical Training, Problems, and Support

1. Were the COSY training sessions in Ottawa last April adequate to prepare you for using the system?
   5 = yes, I had no trouble using COSY afterwards
   4 = yes, but I forgot much of what I learned by the time I tried COSY from home
   1 = no, in retrospect they should have included: "one full day training."

2. Was the COSY user manual adequate to prepare you for using the system?
   1 = I don't know because I never read it
   6 = yes
   4 = no, it should have...
   "been restructured, simplified and streamlined."
   "included a summary list of actions for typical transactions and short cuts, e.g., how to comment on a message without re-reading it."

3. Did you have problems linking to COSY from home or work? (check one)
   3 = no
   8 = yes
   "One time recently I was unable to get on for about 10 days"
   "About 2 times out of 10 there was noise on the line, difficulty getting on"
   "About 2 times out of 10 [there was] messed up system reply, unable to write; the connection was cut twice because of net interconnections in Canada, once 4 days and then about 7 days"
   "About 7 times in 10, noise in the local line"
   "All times (never connected)"
   "In the beginning, but with the help of Mr. Black, by fax, and a friend of mine who is a computer expert, the connection was made.
   "First had hassle with my interface, then COSY"

4. Did you have problems using COSY?
   2 = no
   9 = yes, the most common problem(s) was/were—
   "Not really although I have not been able to download"
   "Forgetting differences between conference and mail; difficulty in locating specific conference messages; download and upload."
   "Editing scratchpad cumbersome; uneven reaction of mail module after sending message...; mailer control of messages sent on internet is erratic...; I was not able to set up an automatic download procedure under kermit, I cold only upload topic by topic or e-mail by e-mail; system went down in the middle of a transaction in some cases (below 3%)."
   "I was not able to upload files"
   "Telecommunication"
   "Noise (initial letters were regularly replaced by some odd signs, and words and lines were missing on the screen); I could not download or upload any message; ...I have enormous difficulty to get a line in [my national] network, I have spent hours trying to access [it] without success [due to] an overload in the network, especially at night when the tariff is lower."
   "Got cut off, etc."
   "Not ergonomic at all; far too complex; bad choice made" 5. Was the technical support offered by IDRC adequate to solve problems you had with COSY?
   4 = I don't know because I never asked for it
   7 = yes
   0 = no

5. Although it is difficult to estimate, about how much money do you think it cost to pay for your use of COSY(including telephone charges) during the conference April – November)?
   Average best guess (N=8) = $541 US
   Extrapolated guess ($541 x 11)= $5,950 US
   Actual cost = $3,534 Canadian x 0.80 = $2,827 US

6. If you have any additional comments or suggestions regarding technical issues that you believe would be useful for improving future computer conferences, please write them below.
   "Easier to use e-mail (e.g., more like pine); delete all systems messages after a set time — reading about downtime 2 years ago is a waste of time"
   "Line editing is a hell, some more flexible procedure needs to be developed; display of outbox should be optional; I was never able to try chat, would be good if the system could be asked at the beginning of a session who among participants is on and to call them to turn to chat."
   "Shorter contribution texts"
   "Do not choose a conference [system?] with so many options; choose the most simple... [fax unreadable here]... it should be usable by anyone"

Conference Preparation, Organization, and Leadership

1. Did the pre-conference meeting in Ottawa give you enough understanding of the conference goals to participate effectively in the conference?
   7 = yes
   1 = "not attended"
   2 = no, in retrospect it should have included
   "The opportunity for each participant to expose his/her views on the actual theme of the conference: the impact of information on development"
   "A hint of the expected products (structure, contents, intention)."

2. Did the goals of the conference become more or less clear to you as the conference progressed?
   5 = yes, especially when I read
   "the digests"
   "the comments"
   "the [general] discussion"
   "the contributions of participants"
   1 = no, I never understood
   "discussions on piecemeal issues when major issues were yet to be unveiled"
   1 = "same"
   1 = "not applicable"
   1 = "yes and no; the goals were clear but not the way taken by some discussions; a few were excessively long and seemed to go astray"

3. Was the conference adequately organized to accomplish its goals?
   2 = yes, especially
   "the categories which worked quite well"
   1 = "yes and no"
   "alternative design choices might have proved more effective, but only a test could tell. A wider group size and possibly more diverse composition might have brought higher traffic and thus more interest/interaction. It might have been easier to start with a discussion of a simple concrete case. If each participant would have prepared a short position statement on each topic at the beginning this might have helped. Summaries should have been more frequent; distribution of printed dumps and summaries should have been more frequent"
   6 = no, in retrospect it could have been improved by
   "greater structure; more milestones; clearer progress monitoring"
   "more frequent contribution from more of the participants"
   "indexing or full text search capability"
   "ambitious and inadequate focus"
   "reducing the number of subconferences"
   "the starting up of working teams when we still were in Ottawa"

4. Was the conference adequately moderated to accomplish its goals?
   6 = yes, especially
   "the provision of digests"
   "when digests were produced"
   "the way the facilitator related comments to earlier statements"
   2 = no, in retrospect it could have been improved by
   "more direction and more aggressive critiquing"
   "stopping everything after one month"

5. If you have additional comments or suggestions about conference preparation, organization or leadership (e.g., about the number or composition of participants, divisions of subconferences, periodic summaries, etc.), please write them below.
   "more diversity in group composition might have been useful"
   "it's unfortunate that some of the early participants dropped out so early and didn't ever participate fully"
   "only one or two voices offered fresh perspectives and insights. much stuff was terribly (and predictably) deja vu"
   "I think that more people with greater experience in running information services in and for developing countries should have participated in the conference. Also people who don't believe in the importance of information for development who would challenge most of the issues raised, creating the atmosphere for creative thinking. Periodic and regular summaries in print-out form would have been useful. There was an excess of sub-conferences"
   "I believe the division of labour was not duly accomplished: participation from the south was almost nonexistent; participation from the north was unbalanced (monopoly by a couple of participants) and not representative of existing visions."

Conference Submissions

1. There were 10 subconferences in the IIDCONF conference: General, Policies, Benefits, Indicators, Calculation, Projects, Research Agenda, Literature, Digests and Other. Considering the goals of the conference:
   The two most valuable sub-conference were —
   5 = Benefits
   5 = Indicators
   2 = Digests
   4 = General
   Comments —
   "Benefits & Indicators because most concrete/relevant to goals"
   "Benefits & Indicators because the challenge which both topics posed"
   "Digests because it provided summaries, and Indicators because [it] contained the meat of the topic."
   "General & Benefits because they were the most used"
   "General & Indicators because most proposals were here"
   "Benefits because it represents the effect, and Indicators because they are measurable elements for the effect"
   "General & Benefits because they dealt what seemed to me the real goals of the conference"
   "General because it permitted further clarification of the broad issues, and Digests because it provided monitoring of information" The two least valuable sub-conferences were —
   
   3 = Calculation
   2 = Literature
   3 = Research Agenda
   1 = Other, Digest, Projects, Policies, and Indicators (each) Comments —
   
   "Calculation because too specific and Other because too vague"
   "Literature because not readily available, and Research Agenda because it seemed to be premature to consider this until other areas such as benefits and indicators were adequately dealt with"
   "Calculation because it was little used and probably premature"
   "Policies and Research Agenda because they were the least used"
   "Literature and Research Agenda because very little could be read"
   "Digest because it not that important, and Other because it reduces concentration on main topics"
   "Policies and Projects because of the reduced number of substantive contributions"
   "Indicators and Calculation because they lack an acceptable framework"

2. In general, how did you find the conference submissions?
   2 = Adequate in both quality and quantity to address the conference issues.
   4 = Adequate quality but inadequate quantity to address the conference issues.
   1 = Adequate quantity but inadequate quality to address the conference issues.
   2 = Inadequate in both quality and quantity to address the conference issues.

3. How was the quality of COSY conference submissions compared to face-to-face conferences?
   0 = about the same as face-to-face conferences
   3 = generally better because —
   "it provided time to reflect; on the other hand however, perhaps because of this the contribution of some participants was sporadic. In face to face conferences perhaps they would feel more compelled to react."
   "more care seemed to go into advanced thinking and the quality of participants was very high"
   "generally better because not interrupted and people had time to thoroughly think before making their points; they could also reflect on various interventions instead of reacting to the last one"
   6 = generally worse because —
   "too many nonparticipants"
   "timing and digression"
   "human interface increases interaction"
   "It lacked the empathy and the possibility of prompt argumentation. Personally, I felt extremely disturbed by the costs of tele- communications. Sitting in front of the computer and having to be concerned with tariffs and the possibility of a technical failure, besides the utilization of several commands, was a painful experience. I do prefer face-to-face conferences."
   1 = better "because they provided recorded information" and worse "because they would not provide signs that are normally useful to assess others' intention"

4. If you have any additional comments or suggestions about conference submissions (especially about increasing their quality or quantity), please write them below.
   "Too much divergence, not enough convergence and synthesis! This was the most serious flaw!"
   "Maximum size of submissions should be established (maybe 2Kb)"
   "Periodic and regular summaries reflecting the most important issues and indicating the points of possible consensus, excluding any irrelevant material, could contribute to improve the quality of the conference in general."
   "I believe well-balanced teams devoted to tackle specific problem statements could have rendered better quality/quantity performances."

Conclusions and Recommendations

1. What percentage of the goals of the conference do you believe were accomplished?
   Average = 58%
   Range= 10–90%
   If your estimated percentage is less than 100, why do you think some goals were not accomplished?
   0 = these goals were not clearly stated
   0 = these goals were never addressed in the conference
   2 = these goals can't be attained in a computer conference because —
   "of the diffuseness of the topic"
   "a consensus or basic understanding of larger issues was not available"
   1 = these goals are impossible to attain in any conference
   6 = other —
   "goals were not translated into component sub-goals and objectives that were concrete enough to be achievable"
   "each time I felt I had a hold on the subject the arguments indicated otherwise"
   "time constraints on very busy people"
   "more time was devoted to general issues and very little on defining indicators"
   "because it seemed that most of the participants lacked the discipline to stick to the objectives expressed in the preliminary outline. The number of subconferences probably contributed to this."
   "the computer system was inadequate, the messages too long"

2. In light of your experiences with the conference and any others, what recommendations can you offer for improving the effectiveness of computer conferences as a forum of policy development or of intellectual exchange?
   "Too many participants were either unwilling or unable to participants. This led to 'monopolizing' the conference by only a few participants"
   "It is an excellent medium for intellectual exchange but perhaps the participants should have been drawn from a more diverse background"
   "Have the group meet face-to-face midway although the conference. Distribute brief summaries regularly in print. Distribute a print integrated bibliography on regular basis"
   "More congenial software, tighter topic focus, more careful selection of participants"
   "People need more practical training in the use of the conferencing system, handling computer communications, including file manage- ment of line (which may be quite time consuming) and even more on communication (the benefits from the use of the technology are no substitute for propension to and ease in communication)."
   "Stating a set of smaller goals with [themes?] specific for each; more time"
   "Initially participants might want to address all the conference issues in their first contribution, and then move to discuss subconference topics."
   "They should be more user friendly and participants should be involved only after they were totally fluent in dealing with the computer and telecommunications techniques, besides a thorough understanding of the system used (COSY). The moderator has an extremely active role to play especially in summarizing the points of consensus and calling attention to those points that were not sufficiently investigated."
   "Identify more concrete policy or research problems; select participants highly motivated in relation to the problems; decentralize tasks; enrich technological environment for better conference monitoring."
   "As simple as possible; messages as short as possible (10 lines max); no personal statements, no [old?] information"

3. Please make any additional comments about the conference you wish below:
   "The topic was a difficult one to come to grips with. The conference was immensely interesting and equally frustrating."
   "I learned a great deal from it. I wish I could have participated more actively all the time. The group of participants was excellent. The facilitator did a masterful job."
   "The conference was very important as a whole. I believe, however, at least by now some of the participants may share a certain frustration in view of the goals which were initially set up. I don't know if this frustration is due to the technique involved (computer conference) or to the difficulties inherent in the theme. I guess ... the results would have been different if a comprehensive working document had been written and distributed at the beginning of the conference."
   "It provided a valuable experience about human interaction mediated by electronic means."

Discussion

Was the Conference Successful?

Now consider some nightmare conferences. One would generate no activity; no one would login and nothing would be written or read. In another, one or two participants would contribute all information while the others passively read it like a periodical, or did not bother to read it at all. In a third, everyone would contribute but no one would respond to the contributions of others. In a fourth, factions would form around conceptual antagonisms and battle over assumptions in a deadly boring epistemological war.

In light of such extremes, the activity data suggest that the conference was a qualified success. It was certainly not a disaster. On average, about 50 contributions were generated in each of its 8 months. Almost half of the participants averaged more 2.5 contributions per month. About half of the contributions were responses to other contributions. There were few signs of an analytical blood bath. On the one hand, by these criteria, it was a productive exchange. On the other hand, the facilitator contributed about half of the conference material, while over half of the participants contributed almost nothing. Discussion in half of the sub-conferences was sparse. Participants reported, on average, that only about half of the conference goals were met.

It is difficult to say how much of the conference failings were a function of the forum or of the participants. In fairness to the forum, most face-to-face conferences show similar tendencies. For example, 20% of the faces usually make 80% of the contributions, and many meetings are poorly attended. The questionnaire indicates that the forum was new to most participants, so few could be expected to be facile in its use. Participants, however, reported chronic difficulties linking to COSY and additional difficulties using it. Many of the facilitator's contributions were directly related to his role: to encourage participation. They cannot be classified as attempts to dominate discussion. Nor can they be faulted for failing to increase the participation of those who could not use the system.

Was the Conference Cost-Effective?

Even academics would take a long time in face-to-face discussion to generate the 325 pages of transcript accumulated from the computer-mediated alternative. Accordingly, the direct costs of the computer conference should be compared to the additional face-to-face meetings that would have been required. Its direct costs (about $3,500) would have paid for the travel and accommodation of only 3–4 participants at a face-to-face conference — half the number of those who regularly participated in the computer conference. The indirect costs for equivalent support may have been less in a traditional conference, but not by a large amount. If an hourly rate for the time and effort taken by participants to learn COSY, deal with its frustrations, type comments, etc., were factored into its costs, the computer conference may not have been such a bargain.

Judging from the questionnaire comments, however, the medium was well-chosen for the task at hand. Although few had good things to say about COSY, and few escaped the frustrations of unreliable links to it, no one challenged or rejected computer conferencing as a potentially useful brainstorming forum. This tacit endorsement of the medium and the relatively low cost of its use suggest that further experiments should be undertaken to improve and to promote it. The questionnaire results provide many useful suggestions for how this can be done.

How Can Computer Conferences Be Improved?

Technical Issues One cannot pass messages without a medium or exchange them without a forum. Judging from the questionnaires, there were nagging problems with both. Problems with telecommunication links to COSY were surprisingly common and chronic. They were as likely to occur in developed countries as in developing ones and discouraged or prevented many from greater participation.

Alas, there is little that can be done to solve these problems, except to wait for the industry to improve service. There is, however, reason to be optimistic about improvement. Reliable digital telecommunication systems are rapidly replacing their less reliable analogue versions in both developed and developing countries. Within 5 years, perhaps half of the linking problems should be solved. More time can then be spent improving the forum.

Participants did not like COSY. There was little to suggest that those who complained were merely rationalizing their low participation by "blaming the equipment." Almost all participants considered the COSY training they received to be quite adequate. In addition, there was no relationship between the degree of participation and liking for the system.

The most common complaints about COSY concerned the difficulty of transferring files to and from it. This implies most participants want to use COSY as a file transfer facility, not as an online system for text composition and screen-reading. A few comments indicate why and confirm previous research (e.g., Thorngate 1991).

Most users do not enjoy learning how to use a PC (personal computer) or learning how to use a local mainframe or network system. Few have the time to learn their advanced features. Although COSY may be wonderful and full of these features, most users see it as just another system to master.

Most do not have the time to do so and little motivation when the system will not be used for long. In addition, it takes only one or two bad experiences with a telecommunication connection to realize that it is dangerous financially to stay on any system that charges too long.

Combine such constraints with busy schedules full of interruptions and the prudent course of action is clear: read and compose messages locally, then get in, transfer files, and get out. As the complaints suggest, the participants in this computer conference found that COSY was not well-designed for this communication style.

In fairness, COSY does have many of the file transfer features which participants desired. Their disuse suggests that future training sessions emphasize file transfer procedures and ensure that participants are confident in using them.

COSY is not, however, the only alternative for computer conferencing. Recent improvements now make the Internet a viable alternative. Either through closed (restricted) news groups or (mailing) list servers of various kinds, the Internet allows its users to use their local system for conference-like forums, relying on its Internet connections to distribute automatically to the local systems of other conference participants all conference messages.

The reading and writing of these messages can be done on each participant's local system. The extra knowledge needed to use the Internet facilities on the local system can be employed beyond any single computer conference. Training can be done locally; so can the solution of telecommunication problems. Conference messages can be collected by the facilitator on her or his own local system, then edited and "published" as a public document file for worldwide retrieval using Internet facilities such as "ftp," "gopher," or "wais."

Furthermore, because the costs of most Internet connections are subsidized by local governments or host organizations, it is generally less expensive to use than COSY. In view of such advantages, it would be worthwhile to consider using the Internet for future computer conference experiments. Like COSY, however, few developing-country organizations currently enjoy direct, affordable access.

Organizational Issues As the technical issues of a computer conference are resolved, the organizational issues become more apparent. A few can be gleaned from responses to the questionnaire. There seemed to be too many subconferences, and the reasons for offering some were ambiguous. As a result, messages tended to collect in a few popular conferences, although some may have been better suited to the less popular ones.

The problems created by this seem minor and tractable. A hypertext system might ease the difficulty of finding related information in a hierarchical scheme. Greater emphasis might be given to creating more useful titles and keywords. Subtopic design was to structure the discussion and to minimize file management tasks. One discussion area with the facilitator organizing the results in appropriate read-only topics might have contributed to more and easier participant input. Future experiments should try different organizational schemes.

There were some complaints about lack of focus, and many requests for more summaries. These may partly reflect the complexity of the conference theme, and partly the extended nature of the conference. Unlike face-to-face conferences concentrated in space and time, computer conferences allow participants to wander around, attend to their lives, and login for occasional visits.

Following a conference hiatus and faced with screens full of new messages, few participants can be expected to keep tightly focussed on conference goals and themes. A typical participant will read something new that will immediately stimulate a close or remote association; he or she will then type it into the conference, logoff, and go home. Several days may pass before the participant reads any responses accrued. Under these conditions, one would expect a rapid decline in focus.

Because of such forum characteristics and the problems that they cause, the computer conference facilitator's task is normally much more difficult and time consuming than his or her face-to-face counterpart (Thorngate 1991). Judging from the lack of complaints in the questionnaire and from previous evaluations of computer conferences, Michel Menou did a remarkable job as facilitator.

Finally, it is useful to note most participants reported sending at least as many private e-mail messages to each other on topics related to the conference as they sent public conference messages. Like the informal discussions that occur in the halls of a face-to-face conference, these e-mail messages are probably crucial to the success of the venture. Further research should study their characteristics and their role in accomplishing conference goals.

In the end, the success of a conference must be judged by the quality of its content. The current evaluation was not intended to assess quality. But the authors hope it has shown that a computer conference is a useful medium for international discussion, and that it should be further refined and used more extensively by international organizations involved in promoting sustainable and equitable development.

References

• Arkes, H.; Hammond, K., ed. 1986. Judgment and decision making. Cambraide University Press, Cambridge, UK.
• Balson, D. 1985. International computer-based conference on biotechnology: A case study. International Development Research Centre (IDRC), Ottawa, ON, Canada.
• Balson, D.; Drysdale, R.; Stanley, B. 1981. Computer based conferencing systems for developing countries. Report of a workshop held in Ottawa, Canada, 26–30 Oct. 1981. International Development Research Centre (IDRC), Ottawa, ON, Canada.
• Simon, H. 1976. Administrative behavior 3rd Edition. The Free Press, New York, NY, USA.
• Thorngate, W. 1985. Social psychology and the design of computer conferencing systems. Proceedings of the first symposium on computer conferencing and electronic mail. University of Guelph, Guelph, ON, Canada. pp. 181–186.
• _____1988. On the evolution of adjudicated contests and the principle of invidious selection. Journal of Behavioral Decision Making, 1, 5–16.
• _____1991. Patterns of computer communication: A report on Brucella network activity. In Frank, J., ed., Networking in brucellosis research. United Nations University Press, Tokyo, Japan.
• Thorngate, W.; Carroll, B. 1987. Why the best person rarely wins: Some embarrassing facts about contests. Simulation and Games, 18, 299–320.
• _____1991. Tests versus contests: A theory of adjudication. In Baker, W.; Highland, M.; van Hezewijk, R.; Terwee, S., ed., Recent trends in theoretical psychology. Springer–Verlag, New York, NY, USA. Vol. 2, pp. 431–438.
• Thorngate, W.; Ferguson, T. 1977. Behind the eyeball: Some common misuses of information in human decision making. Canadian Journal of Information Science, 2, 1–11.
• Thorngate, W.; Hotta, M. 1990. Expertise and information retrieval. Knowledge: Creation, Diffusion, and Utilization, 11, 237–247.
• Tversky, A.; Kahneman, D. 1974. Judgment under uncertainty: Heuristics and biases. Science, 185, 1124–1131.

This research was supported by a grant from IDRC. The authors thank Viviana Alonso and Fatemeh Bagherian for their diligent research assistance in conducting this evaluation.

Document(s) 17 of 19