Home | Contents | Submissions, editors, etc. | Login | Search | ECP
 Electronic Journal of Probability > Vol. 15(2010) > Paper 42 open journal systems 


Compound Poisson Approximation via Information Functionals

A. D. Barbour, Angewandte Mathematik, Universität Zürich
Oliver Johnson, Department of Mathematics, University of Bristol
Ioannis Kontoyiannis, Department of Informatics, Athens University of Economics & Business
Mokshay Madiman, Department of Statistics, Yale University


Abstract
An information-theoretic development is given for the problem of compound Poisson approximation, which parallels earlier treatments for Gaussian and Poisson approximation. Nonasymptotic bounds are derived for the distance between the distribution of a sum of independent integer-valued random variables and an appropriately chosen compound Poisson law. In the case where all summands have the same conditional distribution given that they are non-zero, a bound on the relative entropy distance between their sum and the compound Poisson distribution is derived, based on the data-processing property of relative entropy and earlier Poisson approximation results. When the summands have arbitrary distributions, corresponding bounds are derived in terms of the total variation distance. The main technical ingredient is the introduction of two "information functionals,'' and the analysis of their properties. These information functionals play a role analogous to that of the classical Fisher information in normal approximation. Detailed comparisons are made between the resulting inequalities and related bounds.


Full text: PDF

Pages: 1344-1369

Published on: August 31, 2010
Bibliography
  1. Aldous, David. Probability approximations via the Poisson clumping heuristic. Applied Mathematical Sciences, 77. Springer-Verlag, New York, 1989. xvi+269 pp. ISBN: 0-387-96899-7 MR0969362 (90k:60004)
  2. Artstein, Shiri; Ball, Keith M.; Barthe, Franck; Naor, Assaf. On the rate of convergence in the entropic central limit theorem. Probab. Theory Related Fields 129 (2004), no. 3, 381--390. MR2128238 (2006a:94023)
  3. Artstein, Shiri; Ball, Keith M.; Barthe, Franck; Naor, Assaf. Solution of Shannon's problem on the monotonicity of entropy. J. Amer. Math. Soc. 17 (2004), no. 4, 975--982 (electronic). MR2083473 (2005e:94047)
  4. Barbour, A. D.; Chen, Louis H. Y. Stein's method and applications. Proceedings of the workshop held in Singapore, July 28--August 31, 2003. Edited by A. D. Barbour and Louis H. Y. Chen. Lecture Notes Series. Institute for Mathematical Sciences. National University of Singapore, 5. Published jointly by Singapore University Press, Singapore; and World Scientific Publishing Co. Pte. Ltd., Hackensack, NJ, 2005. xx+297 pp. ISBN: 981-256-281-8 MR2201882 (2006h:60011)
  5. Barbour, A. D.; Chen, Louis H. Y.; Loh, Wei-Liem. Compound Poisson approximation for nonnegative random variables via Stein's method. Ann. Probab. 20 (1992), no. 4, 1843--1866. MR1188044 (93k:60044)
  6. Barbour, A. D.; Chryssaphinou, O. Compound Poisson approximation: a user's guide. Ann. Appl. Probab. 11 (2001), no. 3, 964--1002. MR1865030 (2002k:60029)
  7. Barbour, A. D.; Holst, Lars; Janson, Svante. Poisson approximation. Oxford Studies in Probability, 2. Oxford Science Publications. The Clarendon Press, Oxford University Press, New York, 1992. x+277 pp. ISBN: 0-19-852235-5 MR1163825 (93g:60043)
  8. Barron, Andrew R. Entropy and the central limit theorem. Ann. Probab. 14 (1986), no. 1, 336--342. MR0815975 (87h:60048)
  9. Bobkov, S. G.; Ledoux, M. On modified logarithmic Sobolev inequalities for Bernoulli and Poisson measures. J. Funct. Anal. 156 (1998), no. 2, 347--365. MR1636948 (99e:60051)
  10. Borisov, I. S.; Vorozheĭkin, I. S. Accuracy of approximation in the Poisson theorem in terms of $chisp 2$ distance. (Russian) Sibirsk. Mat. Zh. 49 (2008), no. 1, 8--22; translation in Sib. Math. J. 49 (2008), no. 1, 5--17 MR2400567 (2009c:60078)
  11. Čekanavičius, V.; Roos, B. An expansion in the exponent for compound binomial approximations. Liet. Mat. Rink. 46 (2006), no. 1, 67--110; translation in Lithuanian Math. J. 46 (2006), no. 1, 54--91 MR2251442 (2008k:62042)
  12. Cover, Thomas M.; Thomas, Joy A. Elements of information theory. Wiley Series in Telecommunications. A Wiley-Interscience Publication. John Wiley & Sons, Inc., New York, 1991. xxiv+542 pp. ISBN: 0-471-06259-6 MR1122806 (92g:94001)
  13. Csiszár, Imre; Körner, János. Information theory. Coding theorems for discrete memoryless systems. Probability and Mathematical Statistics. Academic Press, Inc. [Harcourt Brace Jovanovich, Publishers], New York-London, 1981. xi+452 pp. ISBN: 0-12-198450-8 MR0666545 (84e:94007)
  14. Diaconis, Persi; Holmes, Susan Stein's method: expository lectures and applications. Papers from the Workshop on Stein's Method held at Stanford University, Stanford, CA, 1998. Edited by Persi Diaconis and Susan Holmes. Institute of Mathematical Statistics Lecture Notes---Monograph Series, 46. Institute of Mathematical Statistics, Beachwood, OH, 2004. vi+139 pp. ISBN: 0-940600-62-5 MR2118599 (2005i:62008)
  15. Erhardsson, Torkel. Stein's method for Poisson and compound Poisson approximation. An introduction to Stein's method, 61--113, Lect. Notes Ser. Inst. Math. Sci. Natl. Univ. Singap., 4, Singapore Univ. Press, Singapore, 2005. MR2235449
  16. Harremoës, Peter. Binomial and Poisson distributions as maximum entropy distributions. IEEE Trans. Inform. Theory 47 (2001), no. 5, 2039--2041. MR1842536 (2002e:94062)
  17. Johnson, Oliver. Information theory and the central limit theorem. Imperial College Press, London, 2004. xiv+209 pp. ISBN: 1-86094-473-6 MR2109042 (2006m:60001)
  18. Johnson, Oliver. Log-concavity and the maximum entropy property of the Poisson distribution. Stochastic Process. Appl. 117 (2007), no. 6, 791--802. MR2327839 (2008k:62012)
  19. Johnson, Oliver; Barron, Andrew. Fisher information inequalities and the central limit theorem. Probab. Theory Related Fields 129 (2004), no. 3, 391--409. MR2128239 (2006b:62018)
  20. Johnson, Oliver; Kontoyiannis, Ioannis; Madiman, Mokshay. Log-concavity, ultra-log-concavity and a maximum entropy property of discrete compound Poisson measures. Preprint, October 2009. Earlier version online at {tt arXiv:0805.4112v1}, May 2008.
  21. Johnstone, Iain M.; MacGibbon, Brenda. Une mesure d'information caractérisant la loi de Poisson. (French) [An information measure characterizing the Poisson distribution] Séminaire de Probabilités, XXI, 563--573, Lecture Notes in Math., 1247, Springer, Berlin, 1987. MR0942005 (89h:62024)
  22. Kagan, Abram. A discrete version of the Stam inequality and a characterization of the Poisson distribution. J. Statist. Plann. Inference 92 (2001), no. 1-2, 7--12. MR1809692 (2001k:62019)
  23. Kontoyiannis, Ioannis; Harremoës, Peter; Johnson, Oliver. Entropy and the law of small numbers. IEEE Trans. Inform. Theory 51 (2005), no. 2, 466--472. MR2236061 (2007j:94025)
  24. Kontoyiannis, I.; Madiman, M. Measure concentration for compound Poisson distributions. Electron. Comm. Probab. 11 (2006), 45--57 (electronic). MR2219345 (2007h:60011)
  25. Le Cam, Lucien. An approximation theorem for the Poisson binomial distribution. Pacific J. Math. 10 1960 1181--1197. MR0142174 (25 #5567)
  26. Le Cam, Lucien. On the distribution of sums of independent random variables. 1965 Proc. Internat. Res. Sem., Statist. Lab., Univ. California, Berkeley, Calif. pp. 179--202 Springer-Verlag, New York MR0199871 (33 #8011)
  27. Madiman, Mokshay. Topics in information theory, probability, and statistics. Thesis (Ph.D.)--Brown University. ProQuest LLC, Ann Arbor, MI, 2006. 166 pp. ISBN: 978-0542-82065-6 MR2624419
  28. Madiman, Mokshay; Barron, Andrew. Generalized entropy power inequalities and monotonicity properties of information. IEEE Trans. Inform. Theory 53 (2007), no. 7, 2317--2329. MR2319376 (2008k:94024)
  29. Matsunawa, T. Some strong $varepsilon$-equivalence of random variables. Ann. Inst. Statist. Math. 34 (1982), no. 2, 209--224. MR0666413 (84c:60044)
  30. Michel, R. An improved error bound for the compound Poisson approximation of a nearly homogeneous portfolio. ASTIN Bull. 17 (1987), 165--169.
  31. Romanowska, Malgorzata. A note on the upper bound for the distrance in total variation between the binomial and the Poisson distribution. Statistica Neerlandica 31 (1977), no. 3, 127--130. MR0467889 (57 #7739)
  32. Roos, Bero. Sharp constants in the Poisson approximation. Statist. Probab. Lett. 52 (2001), no. 2, 155--168. MR1841404 (2002e:60031)
  33. Roos, Bero. Kerstan's method for compound Poisson approximation. Ann. Probab. 31 (2003), no. 4, 1754--1771. MR2016599 (2004m:60047)
  34. Topsøe, Flemming. Maximum entropy versus minimum risk and applications to some classical discrete distributions. IEEE Trans. Inform. Theory 48 (2002), no. 8, 2368--2376. MR1930296 (2004c:94059)
  35. Tulino, Antonia M.; Verdú, Sergio. Monotonic decrease of the non-Gaussianness of the sum of independent random variables: a simple proof. IEEE Trans. Inform. Theory 52 (2006), no. 9, 4295--4297. MR2298559 (2008e:60134)
  36. Vervaat, W. Upper bounds for the distance in total variation between the binomial or negative binomial and the Poisson distribution. Statistica Neerlandica 23 1969 79--86. MR0242235 (39 #3568)
  37. Wu, Liming. A new modified logarithmic Sobolev inequality for Poisson point processes and several applications. Probab. Theory Related Fields 118 (2000), no. 3, 427--438. MR1800540 (2002f:60109)
  38. Yu, Yaming. On the entropy of compound distributions on nonnegative integers. IEEE Trans. Inform. Theory 55 (2009), no. 8, 3645--3650. MR2598065 (Review)
















Research
Support Tool
Capture Cite
View Metadata
Printer Friendly
Context
Author Address
Action
Email Author
Email Others


Home | Contents | Submissions, editors, etc. | Login | Search | ECP

Electronic Journal of Probability. ISSN: 1083-6489