Copy Link
Add to Bookmark
Report
NL-KR Digest Volume 06 No. 34
NL-KR Digest (Tue Aug 15 11:59:34 1989) Volume 6 No. 34
Today's Topics:
Another IJCAI-89 Update
Papers on representation of time
More Papers on representation of time
Re: NLU benchmarking - request for info
Controlled English
Submissions: nl-kr@cs.rpi.edu
Requests, policy: nl-kr-request@cs.rpi.edu
Back issues are available from host archive.cs.rpi.edu [128.213.1.10] in
the files nl-kr/Vxx/Nyy (ie nl-kr/V01/N01 for V1#1), mail requests will
not be promptly satisfied. If you can't reach `cs.rpi.edu' you may want
to use `turing.cs.rpi.edu' instead.
-----------------------------------------------------------------
To: nl-kr@cs.rpi.edu
From: rjb@allegra.att.com
Date: Wed, 9 Aug 89 14:35:05 EDT
>From: allegra!rjb (Ron Brachman)
Subject: Another IJCAI-89 Update
On Friday afternoon, there will be a "summary session" presenting the
highlights of the KR'89 conference (May 15-18, in Toronto). In addition
to a summary of the conference, the three winners of the KR'89 Awards
for Outstanding Contributions will be re-presented. Here is a draft schedule:
3:00 - 4:30 pm Friday Summary Session IV - Best Knowledge Representation
Papers - 89 (Room D3-28/19 Cobo Hall)
(Chaired by Raymond Reiter)
3:00-3:15 Hector J. Levesque and Ronald J. Brachman,
Introduction and Summary of KR'89
3:15-3:40 Henry A. Kautz and Bart Selman,
"Hard Problems for Simple Default Logics"
3:40-4:05 Lenhart K. Schubert and Chung Hee Hwang,
"An Episodic Knowledge Representation for Narrative Texts"
4:05-4:30 Teodor C. Przymusinski,
"Three-Valued Formalizations of Non-Monotonic Reasoning and
Logic Programming"
------------------------------
To: nl-kr@cs.rpi.edu
Sender: shapiro@cs.Buffalo.EDU
Date: Wed, 9 Aug 89 17:11:01 EDT
From: shapiro@cs.Buffalo.EDU
Subject: Papers on representation of time
Here are two more papers:
@INPROCEEDINGS{AlmSha83,
AUTHOR = "Almeida, M. J. and Shapiro, S. C.",
YEAR = 1983,
TITLE = "Reasoning about the temporal structure of narrative texts",
BOOKTITLE = "Proceedings of the Fifth Annual Meeting of the
Cognitive Science Society",
ADDRESS = "Rochester, {NY}" }
@PHDTHESIS{Alm87,
AUTHOR = "* Almeida, M. J.",
YEAR = 1987,
TITLE = "Reasoning About the Temporal Structure of Narratives",
TYPE = "Technical Report",
NUMBER = "87--10",
SCHOOL = "Department of Computer Science, {SUNY} at Buffalo,",
PAGES = 185 }
------------------------------
To: nl-kr@cs.rpi.edu
Date: Wed, 9 Aug 89 15:54 EDT
From: Brad Miller <miller@CS.ROCHESTER.EDU>
Subject: Re: NL-KR Digest, Volume 6 No. 33
Reply-To: miller@CS.ROCHESTER.EDU
Postal-Address: 610 CS Building, Comp Sci Dept., U. Rochester, Rochester NY 14627
Phone: 716-275-1118
A number of people have asked for this, so here is waht I have, in
bibtex form:
I'd also add:
@string{URCS = "University of Rochester, Computer Science Department"}
@techreport{timelogic,
Author = "Koomen, Johannes A.G.M.",
Title = "The TIMELOGIC Temporal Reasoning System",
Institution = URCS,
Number = "231 (revised)",
Month = Oct,
Year = 1988}
He also has a thesis forthcoming on the subject "Reasoning about
Recurrence".
------------------------------
To: nl-kr@cs.rpi.edu
Date: Thu, 10 Aug 89 14:09:05 -0400
From: finin@PRC.Unisys.COM
Subject: Re: NLU benchmarking - request for info
> Date: Thu, 3 Aug 89 12:36:39 EDT
> From: rpg@cs.brown.edu
> Subject: NLU benchmarking - request for info
>
> Some time ago, in this digest, I recall reading about a workshop
> about evaluation of natural-language understanding programs. Could
> anyone provide me with more information about this workshop? Was a
> report produced, or a set of position papers/abstracts? If so, I'd be
> interested in seeing a copy. ...
There was a Workshop on the Evaluation of Natural Language Processing Systems
held in December of 1988 in Wayne, Pennsylvania. The workshop was organized
by Martha Palmer (Unisys Paoli Research Center) and supported by RADC, AAAI,
ACL and Unisys. Martha Palmer, Sharon Walters (RADC) and I have written a
report summarizing the results of the workshop which we hope to publish soon.
I will announce its availability via the NL-KR newsgroup. I've attached some
additional information on the workshop below.
Tim
___________________________________________________________________________
In the past few years, the computational linguistics research community has
begun to wrestle with the problem of how to evaluate its progress in
developing natural language processing systems. With the exception of
natural language interfaces there are few working systems in existence, and
they tend to focus on very different tasks and equally different techniques.
There has been little agreement in the field about training sets and test
sets, or about clearly defined subsets of problems that constitute standards
for different levels of performance. Even those groups that have attempted a
measure of self-evaluation have often been reduced to discussing a system's
performance in isolation - comparing its current performance to its previous
performance rather than to another system. As this technology begins to move
slowly into the marketplace, the lack of useful evaluation techniques is
becoming more and more painfully obvious.
In order to make progress in the difficult area of natural language
evaluation, a Workshop on the Evaluation of Natural Language Processing
Systems was held last December at the Wayne Hotel in Wayne, Pennsylvania.
There were two basic premises for this workshop. The first was that it
should be possible to discuss system evaluation in general without having to
state whether the purpose of the system is "question-answering" or "text
processing." Evaluating a system requires the definition of an application
task in terms of input/output pairs which are equally applicable to
question-answering, text processing, or generation.
The second premise was that there are two basic types of evaluation, black
box evaluation which measures system performance on a given task in terms of
well-defined input/output pairs, and glass box evaluation which examines the
internal workings of the system. For example, glass box performance
evaluation for a system that is supposed to perform semantic and pragmatic
analysis should include the examination of predicate-argument relations,
referents, and temporal and causal relations. Since there are many different
stages of development that a natural language system passes through before it
is in a state where black box evaluation is even possible, glass box
evaluation plays an especially important role in guiding the development at
early stages.
With these premises in mind, the workshop was structured around the
following three sessions:
- Defining the notions of "glass box evaluation" and "black box
evaluation" and exploring their utility.
- Defining criteria for "glass box evaluation."
- Defining criteria for "black box evaluation."
Calls for participation were sent by electronic mail to several national and
international mailing lists and posted on numerous internet newsgroups and
resulted a large response. A program committee consisting of Beth Sundheim
(NOSC), Ed Hovy (ISI), Tim Finin (Unisys Paoli Research Center), Lynn Bates
(BBN), Martha Palmer (Unisys Paoli Research Center), Mitch Marcus (CIS,
University of Pennsylvania) was put together to plan the workshop and invite
participants. Those respondents interested in participating in the workshop
were asked to describe their interest in the topic, describe any relevant
work done in the area, and provide an abstract on evaluation topics that they
would want to present. A total of fifty people were invited to participate.
It was hoped that the workshop would shed light on the following questions:
- What are valid measures of "black box" performance?
- What linguistic theories are relevant to developing test suites?
- How can we characterize efficiency?
- What is a reasonable expectation for robustness?
- What would constitute valid training sets and test sets?
- How does all of this relate to measuring progress in the field?
------------------------------
To: nl-kr@cs.rpi.edu
From: prlb2!kulcs!siegeert@uunet.UU.NET (Geert Adriaens)
Newsgroups: comp.ai.nlang-know-rep
Subject: Controlled English
Date: 11 Aug 89 07:30:43 GMT
In view of a project at the Departement of Applied Linguistics at the
University of Leuven, we are investigating the possibilities and
limitations of Controlled English (CE) with particular focus on the
grammatical and lexical frameworks used to achieve a form of CE.
Can anyone send us a copy of the following manuals (or give me addresses
where we can order them) :
- the "Caterpillar Fundamental English"-manual,
- the "Kodak International Service Language"-manual,
- J. Smart's "Plain English Program",
- B.T. White's "International Language for Servicing and Maintenance".
Any information about publications on CE or addresses of people also
studying CE (both in industry and at universities) will be most welcome.
- -
Geert Adriaens (SIEMENS-METAL Project)
Maria Theresiastraat 21 siegeert@kulcs.uucp or
B-3000 Leuven siegeert@blekul60.bitnet or
tel: ..32 16 285091 siegeert@cs.kuleuven.ac.be
------------------------------
End of NL-KR Digest
*******************
