Copy Link
Add to Bookmark
Report
Neuron Digest Volume 07 Number 34
Neuron Digest Tuesday, 11 Jun 1991 Volume 7 : Issue 34
Today's Topics:
ANN and GA application to chaotic dynamical systems?
Transportation Applications
Research positions in speech and image processing
OPtimization Methods
Attending IJCNN and would like to visit schools and companies
ANNA91 Proceeedings Info
Re: Rigorous Results on the
Backprop Issues List and ... (part 2)
fast recognition of noisy characters
Looking for optimization applications
RFD: comp.org.issnnet
Send submissions, questions, address maintenance and requests for old issues to
"neuron-request@hplabs.hp.com" or "{any backbone,uunet}!hplabs!neuron-request"
Use "ftp" to get old issues from hplpm.hpl.hp.com (15.255.176.205).
----------------------------------------------------------------------
Subject: ANN and GA application to chaotic dynamical systems?
From: MSANDRI%IVRUNIV.BITNET@ICINECA.CINECA.IT
Date: Fri, 31 May 91 12:45:47 -0100
[[ Editor's Note: This request is a quite broad and a little vague. I
usually refer submissions like this to relevant post issues of teh
Digest, general books or papers, and then ask the author to resubmit with
more specific questions and a demonstration that he or she has done a
little research on the topic before asking for general help. For
example, how would you respond to the question "Give me detailed
information about government?" However, I leave this alone and hope
someonw can provide a useful cogent answer. -PM ]]
Dear network user,
Do you know applications of neural network, genetic algorithms and so on,
to chaotic dynamical systems? I am very interested in such areas.
Thank you for your kindness.
Your Marco.
------------------------------
Subject: Transportation Applications
From: Yu Shen <shen@IRO.UMontreal.CA>
Date: Sun, 02 Jun 91 10:54:08 -0400
I, with my supervisor Guy Lapalme and Jean-Yves Potvin, is working on
neural network model of vehicle dispatching.
Backpropogation is used to compute the choice heuristics from examples of
preference predicates extracted from previous decision cases of experts.
Near 80% correct rate is achieved with untrained cases.
The most recent report of the work will appear in IJCNN-91-Seatle
(abstract only).
I'd like to know your finding in transportation application of neural
network.
Yu Shen
PhD Student
Dept. d'Informatique et Recherche Operationnelle
University de Montreal
C.P. 6128 Succ. A.
Montreal, Que.
Canada
H3C 3J7
(514) 342-7089 (H)
shen.iro.umontreal.ca
------------------------------
Subject: Research positions in speech and image processing
From: Kari Torkkola <karit@spine.hut.fi>
Date: Mon, 03 Jun 91 14:07:02 +0700
RESEARCH POSITIONS AVAILABLE
The newly created "Institut Dalle Molle d'Intelligence Arti-
ficielle Perceptive" (IDIAP) in Martigny Switzerland seeks to
hire qualified researchers in the areas of speech recognition and
image manipulation. Candidates should be able to conduct in-
dependent research in a UNIX environment on the basis of solid
theoretical and applied knowledge. Salaries will be aligned with
those offered by the Swiss government for equivalent positions.
Laboratories are now being established in the newly renovated
building that houses the Institute, and international network
connections will soon be in place. Researchers are expected to
begin activity during the academic year 1991-1992.
IDIAP is the third institute of artificial intelligence sup-
ported by the Dalle Molle Foundation, the others being ISSCO (at-
tached to the University of Geneva) and IDSIA (situated in Lu-
gano). The new institute will maintain close contact with these
latter centers as well as with the Polytechnical School of
Lausanne and the University of Geneva.
To apply for a research position at IDIAP, please send a
curriculum vita and technical reports to:
Daniel Osherson, Directeur
IDIAP
Case Postale 609
CH-1920 Martigny
Switzerland
For further information by e-mail, contact:
osherson@disuns2.epfl.ch
------------------------------
Subject: OPtimization Methods
From: noyesjl%avlab.dnet@wrdc.af.mil
Date: Mon, 03 Jun 91 05:56:51 -0400
NEURAL NETWORK OPTIMIZATION METHODS
Here is some information for any that are interested in the use of
standard optimization techniques to solve multi-layer feed-forward neural
networks. Standard superlinearly convergent methods for solving
unconstrained optimization problems include Conjugate Gradient (CG) and
Quasi-Newton (QN) methods. Most neural net researchers that are
interested in optimization methods seem to favor CG methods over QN
methods. This is because CG methods use O(n) memory locations, while QN
methods typically require O(n^2) memory locations. (Here n is the number
of weights and biases.) On the other hand, QN methods are usually
acknowledged to be faster. However, there is an alternative.
I have found that the (relatively) new Low Storage QN methods usually
produce very satisfactory results. These methods can approximate the
standard BFGS (Broyden-Fletcher-Goldfarb-Shanno) update matrix by using m
of the most recent improvement vectors, where m << n for large problems
(e.g., m is around 5 or 10). In addition, a line search is not usually
needed at each step. The method that I have been using was developed by
Jorge Nocedal (see [1], [2]), but there are other low storage algorithms
as well. So far I have solved problems with up to 3608 weights and
biases (including the 2-2-1 XOR, Fahlman's 10-5-10 Encoder and Complement
Encoder, along with some 25-10-8 and 81-40-8 alphabet problems). More
testing needs to be done to see how low storage QN methods compare to
some of the newer neural net methods in terms of overall efficiency,
including floating-point operations as well as training epochs and
memory. (QN methods are "numerically intensive"). More details may be
found in [3] which should be available this summer. (If someone needs a
copy sooner, contact me at the address below with your surface-mail
address and I will try to provide you with a hardcopy as soon as
possible.)
References:
1. Jorge Nocedal, "Updating Quasi-Newton Matrices with Limited Storage,"
Mathematics of Computation, Vol. 35, No. 151, July 1980, pp. 773-782.
2. Dong C. Liu and Jorge Nocedal, "On the Limited Memory BFGS Method for
Large Scale Optimization," Mathematical Programming, Series B, Vol.
45, No. 3, December 1989, pp. 503-528.
3. James L. Noyes, "Neural Network Optimization Methods," Proceedings of
the Fourth Conference on Neural Networks and Parallel Distributed
Processing, Indiana-Purdue University, Fort Wayne, Indiana, April
11-13, 1991. (To appear.)
Jim Noyes
Department of Mathematics
and Computer Science
Wittenberg University
Box 720
Springfield, OH 45501
noyes@wittenberg.edu
------------------------------
Subject: Attending IJCNN and would like to visit schools and companies
From: kddlab!as1003.meken.fuchueis.toshiba.junet!simokawa@uunet.UU.NET
Date: Mon, 03 Jun 91 19:01:58 +0200
[[ Editor's Note: It is my firm policy (accidently violated only once)
that the mailing list for Neuron Digest is *not* available to anyone. I
will be glad to publish requests such as this one and hope that readers
will respond. Please contact Mr. Shimokawa directly, or send me a messge
which I will then publish here if you would be willing to meet with
visiting researchers. -PM ]]
Dear Mr. Peter Marvit:
I am planning to attend IJCNN-91-SEATTLE this July 9-12, after that wish
to visit the laboratories, universities or private companies. To arrange
my schedule, I intend to use e-mail. So I wish to know e-mail address
who involved to neural nets. Would you e-mail me mail list of
neuron-request or/and some other list if you have? Please note that we
can not use ftp. My purpose to visit are to survey and exchange opinions
of followings
1. Recent application status of NN. Especially I wish to see demonstrations.
2. Hardware implementation of NN.( we are making ASIC NN chips for
parallel processing)
3. NN applications for pattern recognition ( image data)
Looking for your answer
Sincerely,
Y.Shimokawa
[[ Apparent email address: Shimokawa@as1003.meken.fuchueis.toshiba.june ]]
------------------------------
Subject: ANNA91 Proceeedings Info
From: enorris@gmuvax2.gmu.edu (Gene Norris)
Date: Thu, 06 Jun 91 09:17:21 -0400
[[ Editor's Note: I assume the cost mentioned below is 25 dollars (US),
since I'm not aware of "%" as a currency abbreviation. This annoucnement
doesn't mention shipping costs. Is it post-paid? International readers
may wish to enquire ahead of time for possible overseas charges. -PM ]]
The ANNA 91 Conference Proceedings contains full text and illustrations
of 18 papers presented at the ANNA 91 Conference on the Analysis of
Neural Network Applications held May, 1991 at George Mason University.
Proceedings are 212 pages, soft-bound. Copies may be ordered from:
Toni Shetler, ANNA 91 Chair
TRW FVA6/3444
PO Box 10400
Fairfax, VA 22031
(703) 876-4103
Cost is %25.00 per copy.
Prof. Eugene M. Norris
CS Dept George Mason University Fairfax, VA 22032 (703)323-2713
enorris@gmuvax2.gmu.edu FAX: 703 323 2630
------------------------------
Subject: Re: Rigorous Results on the
From: Peter Monsen <ptm3115@draper.com>
Date: 07 Jun 91 10:07:57 -0400
Subject: Time:1:14 AM
OFFICE MEMO RE> Rigorous Results on the_ Date:6/6/91
At Draper Laboratory, we have been investigating dependable and
validatable neural network architectures. Recently, a survey of published
results in dependable NN architectures was conducted in the hopes of
finding quantitative results in this research area.
The following is a reference listing produced by the survey.
If you are interested in further details you can contact me and I can
send you a copy of a memo containing the abstract and a brief review of
each of the papers.
Only some of the papers [3,6,9,10,11,17, and 22] provide
analytical work towards a quantitative measure of the dependability of
NN. The majority of the papers, on the other hand, contained, for the
most part, simulation results describing the performance of specific
networks sloving particular problems under certain assumptions. The
clearest conclusion obtained through this survey is the need for more
quantitative results in this research area.
Peter Monsen
E-mail address: <ptm3115@draper.com>
Surface address: CS Draper Lab, MS 6F, Cambridge MA, 02139 (617) 258-3115
==------------------------------------------------------------------
SELECTED PAPERS ON DEPENDABLE AND VALIDATABLE NEURAL NETWORK ARCHITECTURES
[1]
Title: Neural Networks for Computing?
Author: Abu-Mostafa, Y.S.
Source: in J.S. Denker, ed., AIP Conference Proceedings 151: Neural Networks
for Computing, American Institute of Physics: New York, 1986, pp.1-6.
[2]
Title: Modeling of Fault-Tolerance in Neural Networks.
Authors: Belfore, II, L.A., B.W. Johnson, and J.H. Aylor.
Source: Proceedings of the IJCNN-90-WASH, 1990, pp. I: 325-328.
[3]
Title: The Design of Inherently Fault-Tolerant Systems.
Author: Belfore, L.A., B.W. Johnson, and J.H. Aylor.
Source: Proc. 1987 Workshop on Algorithm, Architecture and Tech. Issues in
Models of Concurrent Computations, pp. 565-583.
[4]
Title: The 'Illusion' of Fault-Tolerance in Neural Networks for Pattern
Recognition and Signal Processing.
Author: Carter, M.J.
Source: Technical Session on Fault-Tolerant Integrated Systems, University of
New Hampshire, Durham, NH, March 1988.
[5]
Title: Operational Fault Tolerance of CMAC Networks.
Author: Carter, M.J., F.J. Rudolph, and A.J. Nucci.
Source: in D.S. Touretzky, ed., Advances in Neural Information Processing
Systems 2, Morgan Kaufman: San Mateo, CA, 1990, pp. 340-347.
[6]
Title: Slow Learning in CMAC Networks and Implications for Fault-Tolerance.
Author: Carter, M.J., A.J. Nucci, E. An, W.T. Miller, III, and F.J. Rudolph.
Source: University of New Hampshire, Intelligent Structures Group, Technical
Report ECE.ICG.90.03, July 1990.
[7]
Title: Fault Tolerant Neural Networks with Hybrid Redundancy.
Author: Chu, Lon-Chan, and B. W. Wah.
Source: Proc. of the IJCNN1990, Vol. II, pp. 639-649.
[8]
Title: Reliability Measures for Hebian-type Associative Memories with Faulty
Interconnections.
Author: Chung, Pau-Choo, and Thomas F. Krile.
Source: Proceedings of the IJCNN 1990, Vol. I, pp. 847-852.
[9]
Title: Reliability Analysis of Artificial Neural Networks.
Author: Dugan, J.B. and J.W. Watterson.
Source: 1991 Proceedings Annual Reliability and Maintainability Symposium, pp.
598-603.
[10]
Title: Quantitative Failure Models of Feed-Forward Neural Networks.
Author: Dzwonczyk, M.J.
Source: Masters of Science Thesis, MIT, 1991. (CSDL-T-1068)
[11]
Title: Using Associated Random Variables to Determine the Reliability of
Neural Networks.
Author: Faris, W.G. and R.S. Maier.
Source: Journal of Neural Network Computing, vol. 2 #2 (Fall 1990), pp. 49-52.
[12]
Title: Neural networks and physical systems with emergent collective
computational abilities.
Author: Hopfield, J.J.
Source: Proc. Natl. Acad. Sci. USA, April 1982, pp. 2554-2558.
[13]
Title: Insipient Fault Detection and Diagnosis Using Artificial Neural
Networks.
Author: Hoskins, J.C., K.M. Kaliyur, and D.M. Himmelblau.
Source: Proceedings of the IJCNN 1990, Vol. I, pp. 81-86.
[14]
Title: Reliability and speed of Recall in an Associative Network.
Author: Lansner, A. and O. Ekeberg.
Source: IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol.
PAMI-7, No. 4, July 1985, pp. 490-498.
[15]
Title: In Search of the Engram.
Author: Lashely, K.S.
Source: Society of Experimental Biology Symposium #4: Psychological Mechanisms
in Animal Behavior, London: Cambridge University Press, 1950, pp.478-505.
Partially reprinted in J.A. Anderson and E. Rosenfeld, eds., Neurocomputing:
Foundations of Research, Cambridge, MA: The MIT Press,1988, pp.59-63.
[16]
Title: Optimal Brain Damage.
Author: Le Cun, Y., J.S. Denker, and S.A. Solla.
Source: in D.S. Touretzky, ed., Advances in Neural Information Processing
Systems 2, Morgan Kaufman: San Mateo, CA, 1990, pp. 598-605.
[17]
Title: Maximally fault-tolerant neural networks and nonlinear programming.
Author: Neti, C., M.H. Schneider, and E.D. Young.
Source: Proceedings of the IJCNN 1990, Vol. II, pp. 483-496.
[18]
Title: Limits to the Fault-Tolerance of a Feedforward Neural Network with
Learning.
Author: Nijhuis,J., Hofflinger, A. van Schaik, and L. Spaanenburg.
Source: Digest of Papers, Fault-Tolerant Computing: The Twentieth International
Symposium (FTCS-20), June 1990, pp. 228-235.
[19]
Title: Trellis Codes, Receptive Fields, and Fault Tolerant, Self-Repairing
Neural Networks.
Author: Petsche, T. and B.W. Dickinson.
Source: IEEE Transactions on Neural Networks, vol. 1 (1990) pp.154-166.
[20]
Title: Fault-Tolerance of a Neural Network Solving the Travelling Salesman
Problem.
Author: Protzel, P., Palumbo, D., and M. Arras.
Source: NASA Contractor Report No. 181798, February 1989.
[21]
Title: Fault Tolerance in Artificial Neural Networks.
Author: Sequin, C.H. and R.D. Clay.
Source: Proceedings of the IJCNN 1990, Vol. I, pp. 703-708.
[22]
Title: Sensitivity of Feedforward Neural Networks to Weight Errors.
Author: Stevenson, M., R. Winter, and B. Widrow.
Source: IEEE Transactions on Neural Networks, vol. 1 (1990), pp.71-80.
[23]
Title: Fault Tolerance in Neural Networks.
Author: Swaminathan, G., S. Srinivasan, S. Mitra, J. Minnix, B. Johnson, and
R.M. Inigo.
Source: Proceedings of the IJCNN-90-WASH, 1990, pp. II: 699-702.
------------------------------
Subject: Backprop Issues List and ... (part 2)
From: mgj@cup.portal.com
Date: Mon, 10 Jun 91 00:16:20 -0700
[[ Editor's Note: Many thanks to Mark Jurik for his work. I look forward
both to his summary and the announcement of his book. I have found his
talks in the past to be illuminating and thought provoking, even if I
don't alsways agree with what he says. -PM ]]
REQUEST FOR REFERENCES TO BACKPROP UPGRADES, PART 2
One month ago, I announced that I am collecting references and
suggestions (for eventual posting and inclusion in a book) on all means
of improving and testing the performance of BackPropagation. A list
suggestions was posted to get things started. Many new suggestions were
offered since that time, thereby expanding the original list. Here s the
latest list:
TRAINING
1. Low bit quantization (how low can you go?)
2. Batching (optimal batch size?)
3. Momentum (fixed and adaptive)
4. Learn rate (fixed and adaptive)
5. Weight decay (fixed and adaptive)
6. Added noise to weight adjustments (fixed and adaptive)
7. Conjugate gradient searching (too much overhead?)
8. Fastprop/Quickprop (are they the same?)
9. Uniprop (does this exist?)
9. Whateverprop (anything else?)
ARCHITECTURE
1. Multiple hidden layers (too much of a good thing?)
2. Sigmoidal vs.Gaussian thresholding (any others?)
4. Recurrent connectivity (instability issues?)
5. Network size (is smaller better?)
6. Complex (real and imaginary) weights (when is it useful?)
PREPROCESSING
1. Kohonen layer quantization (useful for classification?)
2. Fuzzy membership representation (thermometers, etc. ...)
3. Added noise (how much is safe?)
4. Principal component decomposition (when does it help?)
5. Remove linear transformations (& add back later. Is it wise?)
LABORATORY BENCHMARKS
1. N-Bit parity
2. N-M-N encoder/decoder
3. N-N-N linear channel
4. N-2-1 symmetry detection
5. 3-N-1 two out of three detection
6. 2-N-1 Intertwined spiral classification
If you have more topics for the list or references to suggest: please
E-mail your suggestions to mgj@cup.portal.com. The odds are you know of
at least one good paper that most others are not aware.
If you have material you would like me to read and consider for posting
and referenceing in an upcoming book, please mail to JURIK RESEARCH, PO
2379, Aptos, CA 95001
After sufficient information has been collected, a brief synopsis of all
*that has been submitted* will be posted.
-- Mark Jurik, mgj@cup.portal.com
------------------------------
Subject: fast recognition of noisy characters
From: PVR%AUTOCTRL.RUG.AC.BE@CUNYVM.CUNY.EDU
Date: Mon, 10 Jun 91 10:24:00 +0100
I have a couple of problems where it is necessary to recognize
alphanumeric characters (e.g. numberplate inspection, parcel number
inspection, etc). In most of these applications, a fixed character type
is used, but this can change from application to application, even from
batch to batch. One of our students needs to develop a neural network,
which is as general as possible (which means it should be able to tackle
different problems) and as fast as possible, as some of these parcels
need to be inspected at a rate of 25 per second). Furthermore it should
be as accurate as possible. Characters can be covered with noise or
incomplete.
Does anyone have ideas about the direction we should take, hints for
building this network, examples of applications in this field, etc ? Is
backprop the best and only candidate for this network ? If people have
developed applications like this, is it possible to look at the
implementation details ? Etc, etc.
All interventions will be appreciated. I will summarize to the net.
Patrick Van Renterghem
State University of GHENT, Belgium
pvr@autoctrl.rug.ac.be
------------------------------
Subject: Looking for optimization applications
From: "Guillermo Alfonso Parra R." <RYP%ANDESCOL@CUNYVM.CUNY.EDU>
Date: Tue, 11 Jun 91 14:44:07 -1100
[[ Editor's Note: Check Jim Noyes' posting earlier in this issue of the
Digest regarding algorithms. However, it would be intersting to note the
*application* of these algorithms, as the appeal below requests. By the
way, I think the rely address here could also be ryp@andescol.BITNET -PM ]]
Dear Sirs:
I would apreciate any help finding information about optimization
applications using neural networks, specificaly about an article called
"Optimization Algorithms, Simulated Annealing and Neural Networks
Processing", that appeared in Vol. 310 (November 1986), from
"Astrophysical Journal". Please send me any information you have about
this. Thanks a lot,
Guillermo Alfonso Parra R.
------------------------------
Subject: RFD: comp.org.issnnet
From: issnnet@park.bu.edu
Date: Mon, 03 Jun 91 11:27:36 -0400
REQUEST FOR DISCUSSION
----------------------
GROUP NAME: comp.org.issnnet
STATUS: unmoderated
CHARTER: The newsgroup shall serve as a medium for discussions
pertaining to the International Student Society for
Neural Networks (ISSNNet), Inc., and to its activities
and programs as they pertain to the role of students
in the field of neural networks. See details below.
TARGET VOTING DATE: JUNE 20 - JULY 20, 1991
******************************************************************************
PLEASE NOTE
In agreement with USENET newsgroup guidelines for the creation
of new newsgroups, this discussion period will continue until
June 21, at which time voting will begin if deemed
appropriate. ALL DISCUSSION SHOULD TAKE PLACE ON THE NEWSGROUP
"news.groups"
If you do not have access to USENET newsgroups but wish to
contribute to the discussion, send your comments to:
issnnet@park.bu.edu
specifying whether you would like your message relayed to
news.groups. A call for votes will be made to the same
newsgroups and mailing lists that originally received this
message.
PLEASE DO NOT SEND REPLIES TO THIS MAILING LIST OR NEWSGROUP DIRECTLY!
A call for votes will be broadcast in a timely fashion. Please
do not send votes until then.
******************************************************************************
BACKGROUND AND INFORMATION:
The purpose of the International Student Society for Neural
Networks (ISSNNet) is to (1) provide a means of exchanging information
among students and young professionals within the area of Neural
Networks; (2) create an opportunity for interaction between students and
professionals from academia and industry; (3) encourage support >from
academia and industry for the advancement of students in the area of
Neural Networks; (4) insure that the interest of all students in the area
of Neural Networks is taken into consideration by other societies and
institutions involved with Neural Networks; and (5) to foster a spirit of
international and interdisciplinary kinship among students as the study
of Neural Networks develops into a self-contained discipline.
Since its creation one year ago, ISSNNet has grown to over 300
members in more than 20 countries around the world. One of the biggest
problems we have faced thus far is to efficiently communicate with all
the members. To this end, a network of "governors" has been created.
Each governor is in charge of distributing information (such as our
newsletter) to all local members, collect dues, notify local members of
relevant activities, etc.
However, even this system has problems. Communication to a possibly
very large number of members relies entirely on one individual, and given
the typically erratic schedule of a student, it is often difficult to
insure prompt and timely distribution to all members.
More to the point, up until this time all governors have been
contacting a single person (yours truly), and that has been a problem.
Regular discussions on the society and related matters become very
difficult when routed through individuals in this fashion.
The newsgroup would be primarily dedicated to discussion of items
pertaining to the society. We are about to launch a massive call for
nominations, in the hope that more students will step forward and take a
leading role in the continued success of the society.
In addition, ISSNNet is involved with a number of projects, many of
which require extensive electronic mail discussions. For example, we are
developing a sponsorship program for students presenting papers at NNet
conferences. This alone has generated at least 100 mail messages to the
ISSNNet account, most of which could have been answered by two or three
"generic" postings.
We have refrained from using some of the existing mailing lists and
USENET newsgroups that deal with NNets because of the non-technical
nature of our issues. In addition to messages that are strictly
society-related, we feel that there are many messages posted to these
existing bulletin boards for which our newsgroup would be a better forum.
Here is a list of topics that frequently come up, which would be handled
in comp.org.issnnet as part of our "sponsored" programs:
"What graduate school should I go to?"
Last year, ISSNNet compiled a list of graduate programs around the world.
The list will be updated later this year to include a large number of new
degree programs around the world.
"What jobs are available?"
We asked companies that attended last year's IJCNN-San_Diego and
INNC-Paris conferences to fill out a questionnaire on employment
opportunities for NNet students.
"Does anyone have such-and-such NNet simulator?"
Many students have put together computer simulations of NNet paradigms
and these could be shared by people on this group.
"When is the next IJCNN conference?"
We have had a booth at past NNet conferences, and hope to continue doing
this for more and more international and local meetings. We often have
informal get-togethers at these conferences, where students and others
have the opportunity to meet.
-----------------------------------------------------------------------
For more information, please send e-mail to issnnet@park.bu.edu (ARPANET)
write to:
ISSNNet, Inc.
PO Box 557, New Town Br.
Boston, MA 02258 USA
ISSNNet, Inc. is a non-profit corporation in the Commonwealth of
Massachusetts.
ISSNNet, Inc.
P.O. Box 557, New Town Branch
Boston, MA 02258 USA
------------------------------
End of Neuron Digest [Volume 7 Issue 34]
****************************************
