## Fundamentals of Information Theory 基础信息论

Huazhong University of Science and Technology

School of Electronic Information and Communications

**Instructor:** Yayu Gao (高雅玙)
Office: Room 203B, ITEC East Campus Lab (互联网中心东校区实验室203B)

Email: yayugao@hust.edu.cn

Time & Location:

**About this course**

This course is an introduction to information theory, which emphasizes the fundamental concepts as well as analytical skills. Specific topics include entropy, Asymptotic Equipartition Property, data compression, channel capacity, rate distortion, etc.

**Course Organization**

Course Introduction
Entropy, Relative Entropy and Mutual Information
Asymptotic Equipartition Property
Entropy Rate of a Stochastic Process
Data Compression
Channel Capacity
Differential Entropy
Gaussian Channel
Rate Distortion Theory

**Requirements**

Prerequisite courses: Probability Theory and Stochastic Process
You also need to be familiar with Matlab programming.

**Textbook**

Thomas M. Cover and Joy A. Thomas, "Elements of Information Theory", 2nd, John Wiley & Sons, 2006.
Thomas M. Cover and Joy A. Thomas, 阮吉寿(译者), 张华(译者), 信息论基础(原书第2版), 机械工业出版社.

**References**

Landmark paper in information theory: Claude E, Shannon, "A Mathematical Theory of Communications", *The Bell System Technical Journal*, vol. 27, pp. 623-656, July & Oct. 1948.
The renamed book: Claude E, Shannon and Warren Weaver, "The Mathematical Theory of Communications", Univ. of Illinois Press, 1949.
James Gleick, "The Information: A History, A Theory, A Flood", Pantheon Books, 2012.
Raymond W. Yeung, "Information Theory and Network Coding", Springer, 2008.
David Tse and Pramod Viswanath, "Fundamentals of Wireless Communication", Cambridge: Cambridge University Press, 2005.
R. G. Gallager, "Information Theory and Reliable Communication", Jogh Wiley & Sons, 1968.
Interesting Video: Claude Shannon - Father of the Information Age
Video of an experiment for data compression: Can we compress the data unlimitedly?

**Grading**

**Lecture Notes**

Chapter 0: About the Course
Chapter 1: Introduction to Information Theory
Chapter 2: Entropy, Relative Entropy and Mutual Information
Chapter 3: Asymptotic Equipartition Property
Chapter 4: Entropy Rates of a Stochastic Process
Chapter 5: Data Compression
Chapter 7: Channel Capacity
Chapter 8: Differential Entropy
Chapter 9: Gaussian Channel
Chapter 10: Rate Distortion Theory

**Assignments**

**Course Project**

You are encouraged to accomplish a course project.

**Reference Courses**

Prof. Thomas M. Cover in Stanford University
Information Theory by Prof. Raymond W. Yeung
A Short Course in Information Theory by Prof. David J.C. MacKay
Information theory course in MIT
Information theory course in Tsinghua University