Fundamentals of Information Theory 基础信息论

Huazhong University of Science and Technology
School of Electronic Information and Communications

Instructor: Yayu Gao (高雅玙)

Office: Room 203B, ITEC East Campus Lab (互联网中心东校区实验室203B)
Time & Location:

About this course

  • This course is an introduction to information theory, which emphasizes the fundamental concepts as well as analytical skills. Specific topics include entropy, Asymptotic Equipartition Property, data compression, channel capacity, rate distortion, etc.

  • Course Organization

  • Course Introduction
  • Entropy, Relative Entropy and Mutual Information
  • Asymptotic Equipartition Property
  • Entropy Rate of a Stochastic Process
  • Data Compression
  • Channel Capacity
  • Differential Entropy
  • Gaussian Channel
  • Rate Distortion Theory

  • Requirements

  • Prerequisite courses: Probability Theory and Stochastic Process
  • You also need to be familiar with Matlab programming.

  • Textbook

  • Thomas M. Cover and Joy A. Thomas, "Elements of Information Theory", 2nd, John Wiley & Sons, 2006.
  • Thomas M. Cover and Joy A. Thomas, 阮吉寿(译者), 张华(译者), 信息论基础(原书第2版), 机械工业出版社.

  • References

  • Landmark paper in information theory: Claude E, Shannon, "A Mathematical Theory of Communications", The Bell System Technical Journal, vol. 27, pp. 623-656, July & Oct. 1948.
  • The renamed book: Claude E, Shannon and Warren Weaver, "The Mathematical Theory of Communications", Univ. of Illinois Press, 1949.
  • James Gleick, "The Information: A History, A Theory, A Flood", Pantheon Books, 2012.
  • Raymond W. Yeung, "Information Theory and Network Coding", Springer, 2008.
  • David Tse and Pramod Viswanath, "Fundamentals of Wireless Communication", Cambridge: Cambridge University Press, 2005.
  • R. G. Gallager, "Information Theory and Reliable Communication", Jogh Wiley & Sons, 1968.
  • Interesting Video: Claude Shannon - Father of the Information Age
  • Video of an experiment for data compression: Can we compress the data unlimitedly?

  • Grading

    Lecture Notes

  • Chapter 0: About the Course
  • Chapter 1: Introduction to Information Theory
  • Chapter 2: Entropy, Relative Entropy and Mutual Information
  • Chapter 3: Asymptotic Equipartition Property
  • Chapter 4: Entropy Rates of a Stochastic Process
  • Chapter 5: Data Compression
  • Chapter 7: Channel Capacity
  • Chapter 8: Differential Entropy
  • Chapter 9: Gaussian Channel
  • Chapter 10: Rate Distortion Theory


    Course Project

  • You are encouraged to accomplish a course project.

  • Reference Courses

  • Prof. Thomas M. Cover in Stanford University
  • Information Theory by Prof. Raymond W. Yeung
  • A Short Course in Information Theory by Prof. David J.C. MacKay
  • Information theory course in MIT
  • Information theory course in Tsinghua University