消息理論

Information Theory tries to explore the limits of information (data) processing. In this course, we discuss two most fundamental problems in data communication, namely, minimum data representation and reliable communication. Our focus is the well-known Shannon's three theorems: source coding, channel coding and rate distortion theorems. At the end, we extend the concepts in in-formation theory to examine the machine learning algorithms. The goal of studying information theory is to help us to understand the fundamental principles behind various data communication and processing algorithms and to enable us in designing better systems.

Signals and Systems, Probability (and Statistics), Principles of Communication Systems, (Stochastic Processes)

1. D. J. C. MacKay, Information Theory, Inference, and Learning Algorithms. Cambridge Univ. Press, 2003. (Web version) 2. R. W. Yueng (楊偉豪, 香港中文大學), Information Theory and Network Coding, Springer, 2008. (https://link.springer.com/content/pdf/10.1007%2F978-0-387-79234-7.pdf) 3. Stefan M. Moser and Po-Ning Chen, A Student's Guide to Coding and Infor-mation Theory, Cambridge University Press, January 2012. 4. Yury Polyanskiy and Yihong Wu, Information Theory, MIT OpenCourseware, https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-441-information-theory-spring-2016/

Computer Assignment: 15%; Two Quizzes: 15+15%; Midterm Examine: 25% (2 hours, open book – textbook, class notes, your own notes) Final Examine: 30% (2 hours, open book – textbook, class notes, your own notes)

• Introduction
• Information and Entropy
• Source Coding Theorem
• Channel Capacity and Channel Coding Theorem
• Rate Distortion Theory
• Continuous Source, Differential Entropy and Gaussian Channel
• Continuous-amplitude Rate-distortion Theory
• Inference and Machine Learning

Class notes T.M. Cover and J.A. Thomas, Elements of Information Theory, 2nd ed., Wiley 2006.