FUNCTION.rar_蒙特卡罗_RJMCMC_matlab例程_数值算法/人工智能下载-pudn.com

<html xmlns="http://www.w3.org/1999/xhtml"> <head> <meta charset="utf-8"> <meta name="generator" content="pdf2htmlEX"> <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"> <link rel="stylesheet" href="https://static.pudn.com/base/css/base.min.css"> <link rel="stylesheet" href="https://static.pudn.com/base/css/fancy.min.css"> <link rel="stylesheet" href="https://static.pudn.com/prod/directory_preview_static/622b75983d2fbb0007050fcc/raw.css"> <script src="https://static.pudn.com/base/js/compatibility.min.js"></script> <script src="https://static.pudn.com/base/js/pdf2htmlEX.min.js"></script> <script> try{ pdf2htmlEX.defaultViewer = new pdf2htmlEX.Viewer({}); }catch(e){} </script> <title></title> </head> <body> <div id="sidebar" style="display: none"> <div id="outline"> </div> </div> <div id="pf1" class="pf w0 h0" data-page-no="1"><div class="pc pc1 w0 h0"><img class="bi x0 y0 w1 h1" alt="" src="https://static.pudn.com/prod/directory_preview_static/622b75983d2fbb0007050fcc/bg1.jpg"><div class="t m0 x1 h2 y1 ff1 fs0 fc0 sc0 ls0 ws0">CAMBRIDGE UNIVERSITY</div><div class="t m0 x1 h3 y2 ff2 fs1 fc0 sc0 ls0 ws0">ENGINEERING DEPARTMENT</div><div class="t m1 x2 h4 y3 ff3 fs2 fc0 sc0 ls0 ws0">ROBUST FULL BAYESIAN</div><div class="t m1 x3 h4 y4 ff3 fs2 fc0 sc0 ls0 ws0">LEARNING FOR NEURAL NETWORKS</div><div class="t m1 x4 h4 y5 ff4 fs2 fc0 sc0 ls0 ws0">C Andrieu</div><div class="t m1 x5 h4 y6 ff5 fs2 fc0 sc0 ls0 ws0">1</div><div class="t m1 x6 h4 y5 ff4 fs2 fc0 sc0 ls3 ws0">,JFGdeFreitas and A Doucet</div><div class="t m1 x7 h4 y7 ff3 fs2 fc0 sc0 ls0 ws0">Draft of CUED/F-INFENG/TR 343</div><div class="t m1 x8 h4 y8 ff4 fs2 fc0 sc0 ls1 ws0">July 14, 1999</div><div class="t m1 x9 h4 y9 ff4 fs2 fc0 sc0 ls1 ws0">Cambridge University Engineering Department</div><div class="t m1 x6 h4 ya ff4 fs2 fc0 sc0 ls0 ws0">Trumpington Street</div><div class="t m1 xa h4 yb ff4 fs2 fc0 sc0 ls2 ws0">Cambridge CB2 1PZ</div><div class="t m1 xb h4 yc ff4 fs2 fc0 sc0 ls0 ws0">England</div><div class="t m1 xc h4 yd ff4 fs2 fc0 sc0 ls0 ws0">E-mail:</div><div class="t m1 xd h4 yd ff6 fs2 fc0 sc0 ls0 ws0">f</div><div class="t m1 xe h4 yd ff7 fs2 fc0 sc0 ls0 ws0">ca226,jfgf,ad2</div><div class="t m1 xf h4 yd ff6 fs2 fc0 sc0 ls0 ws0">g</div><div class="t m1 x10 h4 yd ff7 fs2 fc0 sc0 ls0 ws0">@eng.cam.ac.uk</div><div class="t m1 x11 h4 ye ff8 fs2 fc0 sc0 ls0 ws0">1</div><div class="t m1 x12 h4 yf ff9 fs2 fc0 sc0 ls0 ws0">Authorship based on alphab etical order</div></div><div class="pi" data-data='{"ctm":[1.568627,0.000000,0.000000,1.568627,0.000000,0.000000]}'></div></div> </body> </html>

<div id="pf2" class="pf w0 h0" data-page-no="2"><div class="pc pc2 w0 h0"><img class="bi x0 y0 w1 h1" alt="" src="https://static.pudn.com/prod/directory_preview_static/622b75983d2fbb0007050fcc/bg2.jpg"><div class="t m1 x13 h4 y10 ffa fs2 fc0 sc0 ls0 ws0">Abstract</div><div class="t m1 x14 h4 y11 ffb fs2 fc0 sc0 ls0 ws0">In this pap er, we propose a hierarchical full Bayesian model for neural networks. This</div><div class="t m1 x15 h4 y12 ffb fs2 fc0 sc0 ls0 ws0">model treats the mo del dimension (number of neurons), model parameters, regularisation</div><div class="t m1 x15 h4 y13 ffb fs2 fc0 sc0 ls0 ws0">parameters and noise parameters as random variables that need to be estimated. We</div><div class="t m1 x15 h4 y14 ffb fs2 fc0 sc0 ls2 ws0">develop a reversible jump Markovchain Monte Carlo (MCMC) method to p erform the</div><div class="t m1 x15 h4 y15 ffb fs2 fc0 sc0 ls0 ws0">necessary computations. We nd that the results obtained using this metho d are not</div><div class="t m1 x15 h4 y16 ffb fs2 fc0 sc0 ls0 ws0">only b etter than the ones rep orted previously, but also app ear to b e robust with respect</div><div class="t m1 x15 h4 y17 ffb fs2 fc0 sc0 ls0 ws0">to the prior sp ecication.</div><div class="t m1 x14 h4 y18 ffb fs2 fc0 sc0 ls0 ws0">In addition, we propose a novel and computationally ecientreversible jump MCMC</div><div class="t m1 x15 h4 y19 ffb fs2 fc0 sc0 ls2 ws0">simulated annealing algorithm to optimise neural networks. This algorithm enables us</div><div class="t m1 x15 h4 y1a ffb fs2 fc0 sc0 ls0 ws0">to maximise the joint p osterior distribution of the network parameters and the number</div><div class="t m1 x15 h4 y1b ffb fs2 fc0 sc0 ls0 ws0">of basis function. It performs a global search in the joint space of the parameters and</div><div class="t m1 x15 h4 y1c ffb fs2 fc0 sc0 ls0 ws0">number of parameters, thereby surmounting the problem of local minima. Weshowthat</div><div class="t m1 x15 h4 y1d ffb fs2 fc0 sc0 ls0 ws0">by calibrating the full hierarchical Bayesian prior, we can obtain the classical AIC, BIC</div><div class="t m1 x15 h4 y1e ffb fs2 fc0 sc0 ls0 ws0">and MDL mo del selection criteria within a p enalised likelihoo d framework. Finally,we</div><div class="t m1 x15 h4 y1f ffb fs2 fc0 sc0 ls0 ws0">present a geometric convergence theorem for the algorithm with homogeneous transition</div><div class="t m1 x15 h4 y20 ffb fs2 fc0 sc0 ls0 ws0">kernel and a convergence theorem for the reversible jump MCMC simulated annealing</div><div class="t m1 x15 h4 y21 ffb fs2 fc0 sc0 ls0 ws0">method.</div><div class="t m1 x16 h4 y22 ff4 fs2 fc0 sc0 ls0 ws0">i</div></div><div class="pi" data-data='{"ctm":[1.568627,0.000000,0.000000,1.568627,0.000000,0.000000]}'></div></div>

<div id="pf3" class="pf w0 h0" data-page-no="3"><div class="pc pc3 w0 h0"><img class="bi x0 y0 w1 h1" alt="" src="https://static.pudn.com/prod/directory_preview_static/622b75983d2fbb0007050fcc/bg3.jpg"><div class="t m1 x17 h4 y23 ffc fs2 fc0 sc0 ls1 ws0">Contents</div><div class="t m1 x17 h4 y24 ff3 fs2 fc0 sc0 ls0 ws0">1 Intro duction 1</div><div class="t m1 x17 h4 y25 ff3 fs2 fc0 sc0 ls0 ws0">2 Problem Statement 3</div><div class="t m1 x17 h4 y26 ff3 fs2 fc0 sc0 ls0 ws0">3 Bayesian Mo del and Aims 5</div><div class="t m1 x12 h4 y27 ff4 fs2 fc0 sc0 ls1 ws0">3.1 Prior distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6</div><div class="t m1 x12 h4 y28 ff4 fs2 fc0 sc0 ls1 ws0">3.2 Estimation and inference aims . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8</div><div class="t m1 x12 h4 y29 ff4 fs2 fc0 sc0 ls1 ws0">3.3 Integration of the nuisance parameters . . . . . . . . . . . .</div><div class="t m1 x18 h4 y2a ff4 fs2 fc0 sc0 ls0 ws0">. . . . . . . . . . . 10</div><div class="t m1 x17 h4 y2b ff3 fs2 fc0 sc0 ls0 ws0">4 Bayesian Computation 11</div><div class="t m1 x12 h4 y2c ff4 fs2 fc0 sc0 ls1 ws0">4.1 MCMC sampler for xed dimension . . . . . . . . . . . . . . . . . . . . . . . . 11</div><div class="t m1 x19 h4 y2d ff4 fs2 fc0 sc0 ls0 ws0">4.1.1 Updating the RBF centres . . . . . . . . . . . . . . . . . . . . . . . . . . 12</div><div class="t m1 x19 h4 y2e ff4 fs2 fc0 sc0 ls0 ws0">4.1.2 Sampling the nuisance parameters . . . . . . . . . . . . . . . . . . . . . 12</div><div class="t m1 x19 h4 y2f ff4 fs2 fc0 sc0 ls0 ws0">4.1.3 Sampling the hyper-parameters . . . . . . . . . . . . . . . . . . . . . . . 13</div><div class="t m1 x12 h4 y30 ff4 fs2 fc0 sc0 ls1 ws0">4.2 MCMC sampler for unknown dimension . . . . . . . . . . . . . . . . . . . . . . 13</div><div class="t m1 x19 h4 y31 ff4 fs2 fc0 sc0 ls0 ws0">4.2.1 Birth and death moves . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15</div><div class="t m1 x19 h4 y32 ff4 fs2 fc0 sc0 ls0 ws0">4.2.2 Split and merge moves . . . . . . . . . . . . . .</div><div class="t m1 x1a h4 y33 ff4 fs2 fc0 sc0 ls0 ws0">. . . . . . . . . . . . . . 17</div><div class="t m1 x19 h4 y34 ff4 fs2 fc0 sc0 ls0 ws0">4.2.3 Update move . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18</div><div class="t m1 x17 h4 y35 ff3 fs2 fc0 sc0 ls0 ws0">5 Reversible Jump Simulated Annealing 18</div><div class="t m1 x12 h4 y36 ff4 fs2 fc0 sc0 ls1 ws0">5.1 Penalised likelihoo d mo del selection . . . . . . . . . . . . . . . . . . . . . . . . 19</div><div class="t m1 x12 h4 y37 ff4 fs2 fc0 sc0 ls1 ws0">5.2 Calibration . . . . . . . . . . . .</div><div class="t m1 x1b h4 y38 ff4 fs2 fc0 sc0 ls0 ws0">. . . . . . . . . . . . . . . . . . . . . . . . . . 20</div><div class="t m1 x12 h4 y39 ff4 fs2 fc0 sc0 ls1 ws0">5.3 Reversible jump simulated annealing . . . . . . . . . . . . . . . . . . . . . . . . 21</div><div class="t m1 x12 h4 y3a ff4 fs2 fc0 sc0 ls1 ws0">5.4 Update move . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23</div><div class="t m1 x12 h4 y3b ff4 fs2 fc0 sc0 ls1 ws0">5.5 Birth and death moves . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23</div><div class="t m1 x12 h4 y3c ff4 fs2 fc0 sc0 ls1 ws0">5.6 Split and merge moves . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23</div><div class="t m1 x17 h4 y3d ff3 fs2 fc0 sc0 ls0 ws0">6 Convergence Results 24</div><div class="t m1 x17 h4 y3e ff3 fs2 fc0 sc0 ls0 ws0">7 Exp eriments 25</div><div class="t m1 x12 h4 y3f ff4 fs2 fc0 sc0 ls1 ws0">7.1 Experiment1: Signal detection . . . . . . . . . . . . . . . . .</div><div class="t m1 x1c h4 y40 ff4 fs2 fc0 sc0 ls0 ws0">. . . . . . . . . . 25</div><div class="t m1 x12 h4 y41 ff4 fs2 fc0 sc0 ls1 ws0">7.2 Experiment2: Robot arm data . . . . . . . . . . . . . . . . . . . . . . . . . . . 28</div><div class="t m1 x12 h4 y42 ff4 fs2 fc0 sc0 ls1 ws0">7.3 Experiment3: Classication with discriminants . . . . . . . . . . . . . . . . . . 31</div><div class="t m1 x17 h4 y43 ff3 fs2 fc0 sc0 ls0 ws0">8 Conclusions 34</div><div class="t m1 x17 h4 y44 ff3 fs2 fc0 sc0 ls0 ws0">9 Acknowledgements 34</div><div class="t m1 x17 h4 y45 ff3 fs2 fc0 sc0 ls0 ws0">A Notation 35</div><div class="t m1 x17 h4 y46 ff3 fs2 fc0 sc0 ls0 ws0">B Proof of Theorem 1 35</div><div class="t m1 x1d h4 y47 ff4 fs2 fc0 sc0 ls0 ws0">ii</div></div><div class="pi" data-data='{"ctm":[1.568627,0.000000,0.000000,1.568627,0.000000,0.000000]}'></div></div>

<div id="pf4" class="pf w0 h0" data-page-no="4"><div class="pc pc4 w0 h0"><img class="bi x0 y0 w1 h1" alt="" src="https://static.pudn.com/prod/directory_preview_static/622b75983d2fbb0007050fcc/bg4.jpg"><div class="t m1 x17 h4 y48 ffc fs2 fc0 sc0 ls0 ws0">1 Intro duction</div><div class="t m1 x17 h4 y49 ff4 fs2 fc0 sc0 ls0 ws0">Articial neural networks are p owerful nonlinear approximation to ols, which rely on struc-</div><div class="t m1 x17 h4 y4a ff4 fs2 fc0 sc0 ls0 ws0">tured combinations of many parameterised basis functions to perform regression, classication</div><div class="t m1 x17 h4 y4b ff4 fs2 fc0 sc0 ls0 ws0">and density estimation. They can approximate any continuous function arbitrarily well as</div><div class="t m1 x17 h4 y4c ff4 fs2 fc0 sc0 ls0 ws0">the number of neurons (basis functions) increases without bound (Cyb enko 1989, Poggio and</div><div class="t m1 x17 h4 y4d ff4 fs2 fc0 sc0 ls0 ws0">Girosi 1990). In addition, they havebeensuccessfully applied to manyand varied complex</div><div class="t m1 x17 h4 y4e ff4 fs2 fc0 sc0 ls0 ws0">problems, including sp eech recognition (Robinson 1994), hand written digit recognition (Le</div><div class="t m1 x17 h4 y4f ff4 fs2 fc0 sc0 ls0 ws0">Cun, Boser, Denker, Henderson, Howard and Hubbard 1989), nancial mo delling (Refenes</div><div class="t m1 x17 h4 y50 ff4 fs2 fc0 sc0 ls0 ws0">1995) and medical diagnosis (Baxt 1990) among others. In these scenarios, it has b een shown</div><div class="t m1 x17 h4 y51 ff4 fs2 fc0 sc0 ls0 ws0">that it is p ossible to train networks with hundreds or thousands of parameters using vari-</div><div class="t m1 x17 h4 y52 ff4 fs2 fc0 sc0 ls0 ws0">ous heuristics and standard techniques, such as back-propagation (Rumelhart, Hinton and</div><div class="t m1 x17 h4 y53 ff4 fs2 fc0 sc0 ls0 ws0">Williams 1986) and cross-validation (Wahba and Wold 1969).</div><div class="t m1 x12 h4 y54 ff4 fs2 fc0 sc0 ls1 ws0">In the early nineties,</div><div class="t m1 x1e h4 y55 ff4 fs2 fc0 sc0 ls0 ws0">Buntine and Weigend (1991) and Mackay (1992) showed that a</div><div class="t m1 x17 h4 y56 ff4 fs2 fc0 sc0 ls0 ws0">principled Bayesian learning approachtoneural networks can lead to manyimprovements.</div><div class="t m1 x17 h4 y57 ff4 fs2 fc0 sc0 ls1 ws0">In particular, Mackay showed that by approximating the distributions of the weights with</div><div class="t m1 x17 h4 y58 ff4 fs2 fc0 sc0 ls0 ws0">Gaussians and adopting smoothing priors, it is p ossible to obtain estimates of the weights</div><div class="t m1 x17 h4 y59 ff4 fs2 fc0 sc0 ls0 ws0">and output variances and to automatically set the regularisation coecients.</div><div class="t m1 x12 h4 y5a ff4 fs2 fc0 sc0 ls0 ws0">Neal (1996) cast the net much further by introducing advanced Bayesian simulation</div><div class="t m1 x17 h4 y5b ff4 fs2 fc0 sc0 ls0 ws0">methods, sp ecically the hybrid Monte Carlo method (Brass, Pendleton, Chen and Robson</div><div class="t m1 x17 h4 y5c ff4 fs2 fc0 sc0 ls0 ws0">1993, Duane, Kennedy, Pendleton and Roweth 1987), into the analysis of neural networks.</div><div class="t m1 x17 h4 y5d ff4 fs2 fc0 sc0 ls0 ws0">Theoretically, he also proved that certain classes of priors for neural networks, whose num-</div><div class="t m1 x17 h4 y5e ff4 fs2 fc0 sc0 ls0 ws0">ber or hidden neurons tends to innity,converge to Gaussian processes. Bayesian sequential</div><div class="t m1 x17 h4 y5f ff4 fs2 fc0 sc0 ls1 ws0">Monte Carlo methods have also been shown to provide goo d training results, esp ecially in</div><div class="t m1 x17 h4 y60 ff4 fs2 fc0 sc0 ls0 ws0">time-varying scenarios (de Freitas, Niranjan, Gee and Doucet 1999).</div><div class="t m1 x12 h4 y61 ff4 fs2 fc0 sc0 ls0 ws0">An essential requirement of neural network training is the correct selection of the num-</div><div class="t m1 x17 h4 y62 ff4 fs2 fc0 sc0 ls0 ws0">ber of neurons. There have been three main approaches to this problem, namely penalised</div><div class="t m1 x17 h4 y63 ff4 fs2 fc0 sc0 ls0 ws0">likelihood, predictive assessment and growing and pruning techniques. In the p enalised like-</div><div class="t m1 x17 h4 y64 ff4 fs2 fc0 sc0 ls0 ws0">lihoo d context, a penalty term is added to the likelihoo d function so as to limit the number</div><div class="t m1 x17 h4 y65 ff4 fs2 fc0 sc0 ls2 ws0">of neurons thereby avoiding over-tting. Classical examples of p enalty terms include the</div><div class="t m1 x17 h4 y66 ff4 fs2 fc0 sc0 ls0 ws0">well known Akaike information criterion (AIC), Bayesian</div><div class="t m1 x1f h4 y67 ff4 fs2 fc0 sc0 ls0 ws0">information criterion (BIC) and</div><div class="t m1 x17 h4 y68 ff4 fs2 fc0 sc0 ls0 ws0">minimum description length (MDL) (Akaike 1974, Schwarz 1985, Rissanen 1987). Penalised</div><div class="t m1 x17 h4 y69 ff4 fs2 fc0 sc0 ls0 ws0">likelihood has also been used extensively to imp ose smo othing constraints either via weight</div><div class="t m1 x17 h4 y6a ff4 fs2 fc0 sc0 ls0 ws0">decay priors (Hinton 1987, Mackay 1992) or functional regularisers that p enalise for high</div><div class="t m1 x17 h4 y6b ff4 fs2 fc0 sc0 ls0 ws0">frequency signal comp onents (Girosi, Jones and Poggio 1995).</div><div class="t m1 x12 h4 y6c ff4 fs2 fc0 sc0 ls0 ws0">In the predictive assessment approach, the data is split into a training set, a validation set</div><div class="t m1 x17 h4 y6d ff4 fs2 fc0 sc0 ls0 ws0">and p ossibly a test set. The key idea is to balance the bias and variance of the predictor by</div><div class="t m1 x1d h4 y6e ff4 fs2 fc0 sc0 ls0 ws0">1</div></div><div class="pi" data-data='{"ctm":[1.568627,0.000000,0.000000,1.568627,0.000000,0.000000]}'></div></div>

<div id="pf5" class="pf w0 h0" data-page-no="5"><div class="pc pc5 w0 h0"><img class="bi x0 y0 w1 h1" alt="" src="https://static.pudn.com/prod/directory_preview_static/622b75983d2fbb0007050fcc/bg5.jpg"><div class="t m1 x17 h4 y23 ff4 fs2 fc0 sc0 ls0 ws0">choosing the number of neurons so that the errors in each data set are of the same magnitude.</div><div class="t m1 x12 h4 y6f ff4 fs2 fc0 sc0 ls0 ws0">The problem with the previous approaches, known as the model adequacy problem, is that</div><div class="t m1 x17 h4 y70 ff4 fs2 fc0 sc0 ls0 ws0">they assume one knows which mo dels to test. To overcome this diculty, various authors</div><div class="t m1 x17 h4 y71 ff4 fs2 fc0 sc0 ls1 ws0">have proposed model selection metho ds, whereby the number of neurons is set by growing</div><div class="t m1 x17 h4 y72 ff4 fs2 fc0 sc0 ls0 ws0">and pruning algorithms. Examples of this class of algorithms include the upstart algorithm</div><div class="t m1 x17 h4 y73 ff4 fs2 fc0 sc0 ls0 ws0">(Frean 1990), cascade correlation (Fahlman and Lebiere 1988), optimal brain damage (Le</div><div class="t m1 x17 h4 y74 ff4 fs2 fc0 sc0 ls0 ws0">Cun, Denker and Solla 1990) and the resource allocating network (RAN) (Platt 1991). A</div><div class="t m1 x17 h4 y75 ff4 fs2 fc0 sc0 ls0 ws0">ma jor shortcoming of these methods is that they lack robustness in that the results dep end</div><div class="t m1 x17 h4 y76 ff4 fs2 fc0 sc0 ls2 ws0">on several heuristically set thresholds. For argument's sake, let us consider the case of the</div><div class="t m1 x17 h4 y77 ff4 fs2 fc0 sc0 ls0 ws0">RAN algorithm. A new radial basis function is added to the hidden layer each time an input</div><div class="t m1 x17 h4 y78 ff4 fs2 fc0 sc0 ls1 ws0">in a novel region of the input space is found.</div><div class="t m1 x20 h4 y79 ff4 fs2 fc0 sc0 ls0 ws0">Unfortunately,novelty is assessed in terms of</div><div class="t m1 x17 h4 y7a ff4 fs2 fc0 sc0 ls0 ws0">two heuristically set thresholds. The centre of the Gaussian basis function is then placed at</div><div class="t m1 x17 h4 y7b ff4 fs2 fc0 sc0 ls0 ws0">the lo cation of the novel input, while its width dep ends on the distance b etween the novel</div><div class="t m1 x17 h4 y7c ff4 fs2 fc0 sc0 ls0 ws0">input and the stored patterns. For improved eciency, the amplitudes of the Gaussians</div><div class="t m1 x17 h4 y7d ff4 fs2 fc0 sc0 ls2 ws0">may be estimated with an extended Kalman lter (Kadirkamanathan and Niranjan 1993).</div><div class="t m1 x17 h4 y7e ff4 fs2 fc0 sc0 ls0 ws0">Yingwei, Sundarara jan and Saratchandran (Yingwei, Sundarara jan and Saratchandran 1997)</div><div class="t m1 x17 h4 y7f ff4 fs2 fc0 sc0 ls1 ws0">have extended the approachby proposing a simple pruning technique. Their strategy is to</div><div class="t m1 x17 h4 y80 ff4 fs2 fc0 sc0 ls0 ws0">monitor the outputs of the Gaussian basis functions continuously and compare them to a</div><div class="t m1 x17 h4 y81 ff4 fs2 fc0 sc0 ls0 ws0">threshold. If a particular output remains b elow the threshold over a number of consecutive</div><div class="t m1 x17 h4 y82 ff4 fs2 fc0 sc0 ls0 ws0">inputs, then the corresp onding basis function is removed.</div><div class="t m1 x12 h4 y83 ff4 fs2 fc0 sc0 ls0 ws0">Recently, Rios Insua</div><div class="t m1 x7 h4 y84 ff4 fs2 fc0 sc0 ls0 ws0">and Muller (1998) , Marrs (1998) and Holmes and Mallick (1998)</div><div class="t m1 x17 h4 y85 ff4 fs2 fc0 sc0 ls1 ws0">have addressed the issue of selecting the number of hidden neurons with growing and pruning</div><div class="t m1 x17 h4 y86 ff4 fs2 fc0 sc0 ls0 ws0">algorithms from a Bayesian persp ective. In particular, they apply the reversible jump Markov</div><div class="t m1 x17 h4 y87 ff4 fs2 fc0 sc0 ls0 ws0">Chain Monte Carlo (MCMC) algorithm of Green (Green 1995, Richardson and Green 1997)</div><div class="t m1 x17 h4 y88 ff4 fs2 fc0 sc0 ls0 ws0">to feed-forward sigmoidal networks and radial basis function (RBF) networks to obtain joint</div><div class="t m1 x17 h4 y89 ff4 fs2 fc0 sc0 ls0 ws0">estimates of the number of neurons and weights. Once again, their results indicate that it is</div><div class="t m1 x17 h4 y8a ff4 fs2 fc0 sc0 ls0 ws0">advantageous to adopt the Bayesian framework and MCMC methods to p erform mo del order</div><div class="t m1 x17 h4 y8b ff4 fs2 fc0 sc0 ls0 ws0">selection.</div><div class="t m1 x12 h4 y8c ff4 fs2 fc0 sc0 ls1 ws0">In this pap er, we also apply the reversible jump MCMC simulation algorithm to RBF</div><div class="t m1 x17 h4 y8d ff4 fs2 fc0 sc0 ls1 ws0">networks so as to compute the joint posterior distribution of the radial basis parameters and</div><div class="t m1 x17 h4 y8e ff4 fs2 fc0 sc0 ls0 ws0">the number of basis functions. However, we advance this area of research in three imp ortant</div><div class="t m1 x17 h4 y8f ff4 fs2 fc0 sc0 ls0 ws0">directions. Firstly,we propose a hierarchical prior for RBF networks. That is, weadoptafull</div><div class="t m1 x17 h4 y90 ff4 fs2 fc0 sc0 ls0 ws0">Bayesian model, which accounts for mo del order uncertainty</div><div class="t m1 x21 h4 y91 ff4 fs2 fc0 sc0 ls0 ws0">and regularisation, and show</div><div class="t m1 x17 h4 y92 ff4 fs2 fc0 sc0 ls0 ws0">that the results appear to be robust with resp ect to the prior sp ecication. Secondly, we</div><div class="t m1 x17 h4 y93 ff4 fs2 fc0 sc0 ls0 ws0">propose an automated growing and pruning reversible jump MCMC optimisation algorithm</div><div class="t m1 x17 h4 y94 ff4 fs2 fc0 sc0 ls0 ws0">to choose the model order using the classical AIC, BIC and MDL criteria. This algorithm</div><div class="t m1 x17 h4 y95 ff4 fs2 fc0 sc0 ls0 ws0">estimates the maximum of the jointlikelihoo d function of the radial basis parameters and the</div><div class="t m1 x17 h4 y96 ff4 fs2 fc0 sc0 ls0 ws0">number of bases using a reversible jump MCMC simulated annealing approach. It has the</div><div class="t m1 x17 h4 y97 ff4 fs2 fc0 sc0 ls0 ws0">advantage of being more computationally ecient than the reversible jump MCMC algorithm</div><div class="t m1 x17 h4 y98 ff4 fs2 fc0 sc0 ls0 ws0">used to perform the integrations with the hierarchical full Bayesian mo del. Finally,we derive</div><div class="t m1 x1d h4 y6e ff4 fs2 fc0 sc0 ls0 ws0">2</div></div><div class="pi" data-data='{"ctm":[1.568627,0.000000,0.000000,1.568627,0.000000,0.000000]}'></div></div>