Copy Link
Add to Bookmark
Report
Neuron Digest Volume 09 Number 06
Neuron Digest Thursday, 13 Feb 1992 Volume 9 : Issue 6
Today's Topics:
Neural Computation 3:4
TR - Small angle neg ion source control
recurrent nets.
TR - Temporal difference learning
Report available --computability with nn's
New TR on unsupervised learning
Connectionism & Motion
Ill-conditioning in NNs (Tech. Rep.)
Parsing embedded sentences ...
A connectionist parsing technique
Paper available in Neuroprose
paper mauduit.lneuro.ps.Z available at archive.cis.ohio-state.edu
Send submissions, questions, address maintenance, and requests for old
issues to "neuron-request@cattell.psych.upenn.edu". The ftp archives are
available from cattell.psych.upenn.edu (128.91.2.173). Back issues
requested by mail will eventually be sent, but may take a while.
----------------------------------------------------------------------
Subject: Neural Computation 3:4
From: Terry Sejnowski <terry@jeeves.UCSD.EDU>
Date: Wed, 11 Dec 91 19:08:41 -0800
Neural Computation
Winter 1991, Volume 3, Issue 4
View
Neural Network Classifiers Estimate Bayesian a Posteriori
Probabilities
Michael D. Richard and Richard P. Lippmann
Note
Lowering Variance of Decisions by Using Artificial
Network Portfolios
G. Mani
Letters
Oscillating Networks: Control of Burst Duration by Electrically
Coupled Neurons
L.F. Abbott, E. Marder, and S.L. Hooper
A Computer Simulation of Oscillatory Behavior in
Primary Visual Cortex
Matthew A. Wilson and James M. Bower
Segmentation, Binding, and Illusory Conjunctions
D. Horn, D. Sagi, and M. Usher
Contrastive Learning and Neural Oscillations
Fernando Pineda and Pierre Baldi
Weight Perturbation: An Optimal Architecture and
Learning Technique for Analog VLSI Feedforward and
Recurrent Multi-Layer Networks
Marwan Jabri and Barry Flower
Predicting the Future: Advantages of Semi-Local Units
Eric Hartman and James D. Keeler
Improving the Generalisation Properties of Radial Basis
Function Neural Networks
Chris Bishop
Temporal Evolution of Generalization during Learning in
Linear Networks
Pierre Baldi and Yves Chauvin
Learning the Unlearnable
Dan Nabutovsky and Eytan Domany
Kolmogorov's Theorem is Relevant
Vera Kurkov
An Exponential Response Neural Net
Shlomo Geva and Joaquin Sitte
SUBSCRIPTIONS - VOLUME 4 - BIMONTHLY (6 issues)
______ $40 Student
______ $65 Individual
______ $150 Institution
Add $12 for postage and handling outside USA (+7% for Canada).
(Back issues from Volumes 1-3 are available for $28 each.)
MIT Press Journals, 55 Hayward Street, Cambridge, MA 02142.
(617) 253-2889.
------------------------------
Subject: TR - Small angle neg ion source control
From: rdj@demos.LANL.GOV (Roger D. Jones)
Date: Mon, 16 Dec 91 13:44:34 -0700
Technical Report Available
OPTIMIZATION AND CONTROL OF A SMALL-ANGLE NEGATIVE ION SOURCE
USING AN ON-LINE ADAPTIVE CONTROLLER BASED ON THE CONNECTIONIST
NORMALIZED LOCAL SPLINE NEURAL NETWORK
by
W. C. Mead, P. S. Bowling, S. K. Brown, R. D. Jones,
C. W. Barnes, H. E. Gibson, J. R. Goulding, and Y. C. Lee
ABSTRACT We have developed CTL, an on-line nonlinear adaptive controller,
to optimize the operation of a repetitively-pulsed negative ion source.
The controller processes multiple diagnostics, including the beam current
waveform, to determine the ion source operating conditions. A figure of
merit is constructed that weights beam current magnitude, noise, and
pulse-to-pulse stability. The operating space of the ion source is
mapped coarsely using automated scan procedures. Then, CTL, using
information derived by fitting the sparse operating space data using the
Connectionist Local Spline artficial neural network (CNLS-net),
interactively adjusts four ion-source control knobs (through regulating
control loops) to optimize the figure of merit. Once coarse optimization
is achieved using CNLS-net's model of machine parameter space, fine
tuning is performed by executing a simplified gradient search algorithm
directly on the machine. Beam quality obtained using the
neural-net-based adaptive controller is consistently quite good. The
search technique has tuned the ion source for near-optimum operation on
six cold startups in one to four hours from time of initial arc. After a
brief interruption of operation, return to optimized beam performance
typically takes only about 15 minutes.
------------------------------
Subject: recurrent nets.
From: Bhaskar DasGupta <bhaskar@theory.cs.psu.edu>
Date: Tue, 17 Dec 91 09:46:14 -0500
The following will appear as a concise paper in IEEE SouthEastcon 1992.
Learning Capabalities of Recurrent Networks.
Bhaskar DasGupta
Computer Science Department
Penn State.
Brief summary:
Recurrent Neural Networks are models of computation in which the
underlying graph is directed ( possibly cyclic ), and each processor
changes state according to some function computed according to its
weighted summed inputs, either deterministically or probabilistically.
Under arbitrary probabilistic update rules, such models can be as
powerful as Probabilistic Turing Machines. For probabilistic models we
can define the error probability as the maximum probability of reaching
an incorrect output configuration. It is observed:
If the error probability is bounded then such a network can be simulated
by a deterministic finite automaton ( with exponentially many states )
For deterministic recurrent nets where each processor implements a
threshold function:
It may accept all P-complete language problems. However, restricting
the weight-threshold relationship may result in accepting a weaker
class, the NC class ( problems which can be solved in poly-log time
with polynomially many processors ).
The results are straightforward to derive, so I did not put it in the
neuroprose archive.
Thanks.
Bhaskar
------------------------------
Subject: TR - Temporal difference learning
From: Gerald Tesauro <tesauro@watson.ibm.com>
Date: Tue, 17 Dec 91 16:11:23 -0500
The following technical report is now available. (This is a long version
of the paper to appear in the next NIPS proceedings.) To obtain a copy,
send a message to "tesauro@watson.ibm.com" and be sure to include your
PHYSICAL mail address.
Practical Issues in Temporal Difference Learning
Gerald Tesauro
IBM Thomas J. Watson Research Center
PO Box 704, Yorktown Heights, NY 10598 USA
Abstract: This paper examines whether temporal difference methods for
training connectionist networks, such as Suttons's TD($\lambda$)
algorithm, can be successfully applied to complex real-world problems. A
number of important practical issues are identified and discussed from a
general theoretical perspective. These practical issues are then
examined in the context of a case study in which TD($\lambda$) is applied
to learning the game of backgammon from the outcome of self-play. This
is apparently the first application of this algorithm to a complex
nontrivial task. It is found that, with zero knowledge built in, the
network is able to learn from scratch to play the entire game at a fairly
strong intermediate level of performance, which is clearly better than
conventional commercial programs, and which in fact surpasses comparable
networks trained on a massive human expert data set. This indicates that
TD learning may work better in practice than one would expect based on
current theory, and it suggests that further analysis of TD methods, as
well as applications in other complex domains, may be worth
investigating.
------------------------------
Subject: Report available --computability with nn's
From: sontag@control.rutgers.edu
Date: Wed, 18 Dec 91 16:19:17 -0500
(Revised) Tech Report available from neuroprose:
ON THE COMPUTATIONAL POWER OF NEURAL NETS
Hava T. Siegelmann, Department of Computer Science
Eduardo D. Sontag, Department of Mathematics
Rutgers University, New Brunswick, NJ 08903
This paper shows the Turing universality of first-order, finite neural
nets. It updates the report placed there last Spring* with new results
that include the simulation in LINEAR TIME of BINARY-tape machines, (as
opposed to the unary alphabets used in the previous version). The
estimate of the number of neurons needed for universality is now lowered
to 1,000 (from 100,000).
*A summary of the older report appeared in: H. Siegelmann and E. Sontag,
"Turing computability with neural nets," Applied Math. Letters 4 (1991):
77-80.
================
To obtain copies of the postscript file, please use Jordan Pollack's service:
Example:
unix> ftp archive.cis.ohio-state.edu (or ftp 128.146.8.52)
Name (archive.cis.ohio-state.edu): anonymous
Password (archive.cis.ohio-state.edu:anonymous): <ret>
ftp> cd pub/neuroprose
ftp> binary
ftp> get siegelman.turing.ps.Z
ftp> quit
unix> uncompress siegelman.turing.ps.Z
Now print "siegelman.turing.ps" as you would any other (postscript) file.
------------------------------
Subject: New TR on unsupervised learning
From: Juergen Schmidhuber <yirgan@dendrite.cs.colorado.edu>
Date: Wed, 18 Dec 91 15:42:08 -0700
LEARNING FACTORIAL CODES BY PREDICTABILITY MINIMIZATION
..
Jurgen Schmidhuber
Department of Computer Science
University of Colorado
(Compact version of Technical Report CU-CS-565-91)
ABSTRACT
I present a novel general principle for unsupervised learning of
distributed non-redundant internal representations of input patterns
or input sequences. With a given set of representational units, each
unit tries to react to the environment such that it minimizes its
predictability by an adaptive predictor that sees all the other
units. This encourages each unit to filter `abstract concepts' out of
the environmental input such that these concepts are statistically
independent of those upon which the other units focus. I discuss
various simple yet potentially powerful implementations of the
principle which aim at finding binary factorial codes (Barlow, 1989},
i.e. codes where the probability of the occurrence of a particular
input is simply the product of the probabilities of the corresponding
code symbols. Unlike previous methods the novel principle has a
potential for removing not only linear but also non-linear output
redundancy. Methods for finding factorial codes automatically embed
Occam's razor for finding codes using a minimal number of units.
Illustrative experiments show that algorithms based on the principle
of predictability minimization are practically feasible. The final
part of this paper describes an entirely local algorithm that has a
potential for learning unique representations of extended sequences.
=---------------------------------------------------------------------
To obtain a copy, do:
unix> ftp archive.cis.ohio-state.edu
Name: anonymous
Password: neuron
ftp> binary
ftp> cd pub/neuroprose
ftp> get schmidhuber.factorial.ps.Z
ftp> bye
unix> uncompress schmidhuber.factorial.ps.Z
unix> lpr schmidhuber.factorial.ps
=---------------------------------------------------------------------
There is no hardcopy mailing list. I will read my mail only
occasionally during the next three weeks or so.
..
Jurgen
------------------------------
Subject: Connectionism & Motion
From: "Mike R. W. Dawson" <mike@psych.ualberta.ca>
Date: Thu, 19 Dec 91 20:27:42 -0700
The following paper has recently appeared in Psychological Review, and
describes how a variant of Anderson's "brainstate-in-a-box" algorithm can
be used to solve a particular information processing problem faced when
apparent motion is perceived. If you're interested in a reprint, please
contact me at the address below.
=======================================================================
Dawson, M.R.W. (1991). The how and why of what went where in apparent
motion: Modeling solutions to the motion correspondence problem.
Psychological Review, 98(4), 569-603.
A model that is capable of maintaining the identities of individuated
elements as they move is described. It solves a particular problem of
underdetermination, the motion correspondence problem, by simultaneously
applying three constraints: the nearest neighbour principle, the relative
velocity principle, and the element integrity principle. The model
generates the same correspondence solutions as does the human visual
system for a variety of displays, and many of its properties are
consistent with what is known about the physiological mechanisms
underlying human motion perception. The model can also be viewed as a
proposal of how the identities of attentional tags are maintained by
visual cognition, and thus it can be differentiated from a system that
serves merely to detect movement.
=======================================================================
Michael R. W. Dawson email: mike@psych.ualberta.ca
Biological Computation Project
Department of Psychology
University of Alberta
Edmonton, Alberta Tel: +1 403 492 5175
T6G 2E9, Canada Fax: +1 403 492 1768
------------------------------
Subject: Ill-conditioning in NNs (Tech. Rep.)
From: Sirpa Saarinen <saarinen@csrd.uiuc.edu>
Date: Fri, 20 Dec 91 11:55:44 -0600
Technical report available:
CSRD Report no. 1089
Ill-Conditioning in Neural Network Training Problems
S. Saarinen, R. Bramley and G. Cybenko
Center for Supercomputing Research and Development,
University of Illinois, Urbana, IL, USA 61801
Abstract
The training problem for feedforward neural networks is nonlinear
parameter estimation that can be solved by a variety of optimization
techniques. Much of the literature on neural networks has focused on
variants of gradient descent. The training of neural networks using such
techniques is known to be a slow process with more sophisticated
techniques not always performing significantly better. In this paper, we
show that feedforward neural networks can have ill-conditioned Hessians
and that this ill-conditioning can be quite common. The analysis and
experimental results in this paper lead to the conclusion that many
network training problems are ill-conditioned and may not be solved more
efficiently by higher order optimization methods. While our analyses are
for completely connected layered networks, they extend to networks with
sparse connectivity as well. Our results suggest that neural networks
can have considerable redundancy in parameterizing the function space in
a neighborhood of a local minimum, independently of whether or not the
solution has a small residual.
If you wish to have this report, please write to
nichols@csrd.uiuc.edu
and ask for report 1089.
Sirpa Saarinen
saarinen@csrd.uiuc.edu
------------------------------
Subject: Parsing embedded sentences ...
From: Josep M Sopena <D4PBJSS0%EB0UB011.BITNET@BITNET.CC.CMU.EDU>
Date: Fri, 20 Dec 91 18:13:03 -0500
The following paper is now available. To obtain a copy send a
message to "d4pbjss0@e0ub011.bitnet".
ESRP: A DISTRIBUTED CONNECTIONIST PARSER THAT USES
EMBEDDED SEQUENCES TO REPRESENT STRUCTURE
Josep M Sopena
Departament de Psicologia Basica
Universitat de Barcelona
In this paper we present a neural network that is able to compute a
certain type of structure, that among other things allows it to
adequately assign thematic roles, and find the antecedents of the traces,
pro, PRO, anaphoras, pronouns, etc. for an extensive variety of
syntactic structures.
Up until now, the type of sentences that the network has been
able to parse include:
1. 'That' sentences with several levels of embedding.
John says that Mary thought that Peter was ill.
2.- Passive sentences.
3.- Relative sentences with several levels of embedding (center
embedded).
John loved the girl that the carpenter who the builder hated
was seeing.
The man that bought the car that Peter wanted was crazy.
The man the woman the boy hates loves is running.
4.-Syntactic ambiguity in the attachment of PP's
John saw a woman with a handbag with binoculars.
5.- Combinations of these four types of sentences:
John bought the car that Peter thought the woman with a
handbag wanted.
The input consists of the sentence presented word by word. The patterns
in the output represent the structure of the sentence. The structure is
not represented by a static pattern but by a temporal course of patterns.
This evolution of the output is based on different types of psychological
evidence, and is as follows: the output is a sequence of simple semantic
predicates (although it could be thought of in a more syntactical way).
An element of the output sequence consists only of a single predicate,
which always has to be complete. Since there are often omitted elements
within the clauses (eg. Traces, PRO, pro etc.) the network retrieves
these elements in order to complete the current predicate.
These two mechanisms, segmentation into simple predicates and retreival
of previously processed elements, are those which allow structure to be
computed. In this way the structure is not conceived solely as a linear
sequence of simple predicates because using these mechanisms it is
posible to form embedded sequences (embedded structures).
The paper also includes empirical evidence that supports the model as a
plausible psychological model.
The NN is formed by two parallel modules that share all of the output and
part of the input. The first module is an standard Elman network that
maps the elements in the input with their predicate representation in the
output and assigns the corresponding semantic roles. The second module is
a modified Elman network with two hidden layers. The units of the first
hidden layer (which is the copied layer) have a linear function
activation. This type of network has a much greater short term memory
capacity than a standard Elman network. It stores the sequence of
predicates, retreives the elements of the current predicate omitted in
the input (traces, PRO etc.) and the referents of pronouns and anaphoras.
When a pronoun or an anaphora appears in the input, the corresponding
antecedent in the sentence, which has been retreived from this second
module, is placed in the output. This module also allows the network to
build embedded sequences by retreiving former elements of the sequence.
The two modules were simultaneously trained. There were no manipulations
other than the changes of inputs and targets, as in the standard
backpropagation algorithm.
The network was trained with 3000 sentences built from a starting a
vocabulary of 1000 words. The number of sentences that is possible to
build starting from this vocabulary is power(10,15). The generalization
was completely successful for a test set of 800 sentences representing
the variety of syntactic patterns of the training set.
The model bears some relationship with the idea of representing structure
not only in space but in time as well (Hinton 1989)and with the RAAM
networks of Pollack(1989). The shortcomings of this type of networks are
also discussed.
------------------------------
Subject: A connectionist parsing technique
From: Ronan Reilly ERC <M160%eurokom.ie@BITNET.CC.CMU.EDU>
Date: Mon, 23 Dec 91 11:17:00 +0700
Below is the abstract of a paper that is due to appear in the journal
Network early next year. A compressed postscript version
(reilly.parser.ps.Z) has been placed in Jordan Pollack's Neuroprose
archive at Ohio State and can be retrieved in the usual way.
Requests for hardcopy should be sent to:
ronan_reilly@eurokom.ie
Season's greetings,
Ronan
==========================================
A Connectionist Technique for On-Line Parsing
Ronan Reilly
Educational Research Centre
St Patrick's College, Dublin 9
A technique is described that permits the on-line construction and
dynamic modification of parse trees during the processing of
sentence-like input. The approach is a combination of simple recurrent
network (SRN) and recursive auto-associative memory (RAAM) . The parsing
technique involves teaching the SRN to build RAAM representations as it
processes its input item-by-item. The approach is a potential component
of a larger connectionist natural language processing system, and could
also be used as a tool in the cognitive modelling of language
understanding. Unfortunately, the modified SRN demonstrates a limited
capacity for generalisation.
------------------------------
Subject: Paper available in Neuroprose
From: Prahlad.Gupta@K.GP.CS.CMU.EDU
Date: Thu, 26 Dec 91 14:43:41 -0500
The following paper has been placed in the Neuroprose archive, as the
file gupta.stress.ps.Z
Comments are invited.
Retrieval instructions follow the abstract below. Thanks to Jordan
Pollack for making this facility available.
-- Prahlad
===========================================================
CONNECTIONIST MODELS & LINGUISTIC THEORY:
INVESTIGATIONS OF STRESS SYSTEMS IN LANGUAGE
PRAHLAD GUPTA DAVID S. TOURETZKY
-------------------------- --------------------------
Dept. of Psychology School of Computer Science
Carnegie Mellon University Carnegie Mellon University
Pittsburgh, PA 15213 Pittsburgh, PA 15213
prahlad@cs.cmu.edu dst@cs.cmu.edu
============================================================
Abstract
--------
This work describes the use of connectionist techniques to model the
learning and assignment of linguistic stress. Our aim was to explore the
ability of a simple perceptron to model the assignment of stress in
individual words, and to consider, in light of this study, the
relationship between the connectionist and theoretical linguistics
approaches to investigating language.
We first point out some interesting parallels between aspects of the
model and the constructs and predictions of Metrical Phonology, the
linguistic theory of stress: (1) the distribution of learning times
obtained from perceptron experiments corresponds with theoretical
predictions of "markedness," and (2) the weight patterns developed by
perceptron learning bear a suggestive *structural* relationship to
features of the linguistic analysis, particularly with regard to
"iteration" and "metrical feet".
We use the connectionist learning data to develop an analysis of
linguistic stress based on perceptron-learnability. We develop a novel
characterization of stress systems in terms of six parameters. These
provide both a partial description of the stress pattern itself and a
prediction of its learnability, without invoking abstract theoretical
constructs such as "metrical feet." Our parameters encode linguistically
salient concepts as well as concepts that have computational
These two sets of results suggest that simple connectionist learning
techniques have the potential to complement, and provide computational
validation for, abstract theoretical investigations of linguistic
domains.
We then examine why such methodologies should be of interest for
linguistic theorizing. Our analysis began at a high level by observing
inherent characteristics of various stress systems, much as theoretical
linguistics does. However, our explanations changed substantially when
we included a detailed account of the model's processing mechanisms. Our
higher-level, theoretical account of stress was revealed as only an
*approximation* to the lower-level computational account. Without the
ability to open up the black boxes of the human processor, linguistic
analyses are arguably analogous to our higher-level descriptions. This
highlights the need for *computational grounding* of theory-building. In
addition, we suggest that there are methodological problems underlying
parameter-based approaches to learnability. These problems make it all
the more important to seek sources of converging evidence such as is
provided by computational models.
=-------------------------------------------------------------------------
To retrieve the paper by anonymous ftp:
unix> ftp archive.cis.ohio-state.edu # (128.146.8.52)
Name: anonymous
Password: neuron
ftp> cd pub/neuroprose
ftp> binary
ftp> get gupta.stress.ps.Z
ftp> quit
unix> uncompress gupta.stress.ps.Z
unix> lpr -P<printer name> gupta.stress.ps
=-------------------------------------------------------------------------
------------------------------
Subject: paper mauduit.lneuro.ps.Z available at archive.cis.ohio-state.edu
From: Nicolas Mauduit <mauduit@ece.UCSD.EDU>
Date: Thu, 26 Dec 91 14:47:13 -0800
The preprint of the following paper, to appear in IEEE Neural Networks,
march 92 special issue on hardware, is available by ftp from the
neuroprose archive at archive.cis.ohio-state.edu (file
mauduit.lneuro.ps.Z):
Lneuro 1.0: a piece of hardware LEGO for building neural network systems
(to appear in IEEE Neural Networks, march 92 special issue on hardware)
by Nicolas MAUDUIT UCSD, dept. ECE, EBU1
La Jolla, CA 92093-0407
USA
Marc DURANTON LEP, div. 21
Jean GOBERT B.P. 15, 22, avenue Descartes
Jacques-Ariel SIRAT 94453 Limeil Brevannes France
Abstract:
The state of our experiments on neural networks simulations on a parallel
architecture is presented here. A digital architecture was selected,
scalable and flexible enough to be useful for simulating various kinds of
networks and paradigms. The computing device is based on an existing
coarse grain parallel framework (INMOS Transputers), improved with finer
grain parallel abilities through VLSI chips, called the Lneuro 1.0, for
LEP neuromimetic circuit. The modular architecture of the circuit
enables to build various kinds of boards to match the foreseen range of
applications, or to increase the power of the system by adding more
hardware. The resulting machine remains reconfigurable according to a
specific problem to some extent, at the system level through the
Transputers framework, as well as at the circuit level.
A small scale machine has been realized using 16 Lneuros arranged in
clusters composed of 4 circuits and a controller, to experimentally test
the behaviour of this architecture (the communication, control,
primitives required, etc.). Results are presented on an integer version
of Kohonen feature maps. The speedup factor increases regularly with the
number of clusters involved (up to a factor 80). Some ways to improve
this family of neural networks simulation machines are also investigated.
=----------------------------------------------------------------------------
The file can be obtained the usual way:
unix> ftp archive.cis.ohio-state.edu (or 128.146.8.52)
Name: anonymous
Password: ...
ftp> cd pub/neuroprose
ftp> binary
ftp> get mauduit.lneuro.ps.Z
ftp> quit
unix> uncompress mauduit.lneuro.ps.Z
then print the file mauduit.lneuro.ps on a postscript printer
Nicolas Mauduit
=----------------------------------------------------------------
Nicolas Mauduit, Dept ECE | Phone (619) 534 6026
UCSD EBU 1 | FAX (619) 534 1225
San Diego CA 92093-0407 USA | Email mauduit@celece.ucsd.edu
------------------------------
End of Neuron Digest [Volume 9 Issue 6]
***************************************
