Information and Coding Theory

This text is an elementary introduction to information and coding theory. The first part focuses on information theory, covering uniquely decodable and instantaneous codes, Huffman coding, entropy, information channels, and Shannon's Fundamental Theorem. In the second part, linear algebra is used to construct examples of such codes, such as the Hamming, Hadamard, Golay and Reed-Muller codes. Contains proofs, worked examples, and exercises.

Contenu

1. Source Coding.- 1.1 Definitions and Examples.- 1.2 Uniquely Decodable Codes.- 1.3 Instantaneous Codes.- 1.4 Constructing Instantaneous Codes.- 1.5 Kraft's Inequality.- 1.6 McMillan's Inequality.- 1.7 Comments on Kraft's and McMillan's Inequalities.- 1.8 Supplementary Exercises.- 2. Optimal Codes.- 2.1 Optimality.- 2.2 Binary Huffman Codes.- 2.3 Average Word-length of Huffman Codes.- 2.4 Optimality of Binary Huffman Codes.- 2.5 r-ary Huffman Codes.- 2.6 Extensions of Sources.- 2.7 Supplementary Exercises.- 3. Entropy.- 3.1 Information and Entropy.- 3.2 Properties of the Entropy Function.- 3.3 Entropy and Average Word-length.- 3.4 Shannon-Fano Coding.- 3.5 Entropy of Extensions and Products.- 3.6 Shannon's First Theorem.- 3.7 An Example of Shannon's First Theorem.- 3.8 Supplementary Exercises.- 4. Information Channels.- 4.1 Notation and Definitions.- 4.2 The Binary Symmetric Channel.- 4.3 System Entropies.- 4.4 System Entropies for the Binary Symmetric Channel.- 4.5 Extension of Shannon's First Theorem to Information Channels.- 4.6 Mutual Information.- 4.7 Mutual Information for the Binary Symmetric Channel.- 4.8 Channel Capacity.- 4.9 Supplementary Exercises.- 5. Using an Unreliable Channel.- 5.1 Decision Rules.- 5.2 An Example of Improved Reliability.- 5.3 Hamming Distance.- 5.4 Statement and Outline Proof of Shannon's Theorem.- 5.5 The Converse of Shannon's Theorem.- 5.6 Comments on Shannon's Theorem.- 5.7 Supplementary Exercises.- 6. Error-correcting Codes.- 6.1 Introductory Concepts.- 6.2 Examples of Codes.- 6.3 Minimum Distance.- 6.4 Hamming's Sphere-packing Bound.- 6.5 The Gilbert-Varshamov Bound.- 6.6 Hadamard Matrices and Codes.- 6.7 Supplementary Exercises.- 7. Linear Codes.- 7.1 Matrix Description of Linear Codes.- 7.2 Equivalence of Linear Codes.- 7.3 Minimum Distance of Linear Codes.- 7.4 The Hamming Codes.- 7.5 The Golay Codes.- 7.6 The Standard Array.- 7.7 Syndrome Decoding.- 7.8 Supplementary Exercises.- Suggestions for Further Reading.- Appendix A. Proof of the Sardinas-Patterson Theorem.- Appendix B. The Law of Large Numbers.- Appendix C. Proof of Shannon's Fundamental Theorem.- Solutions to Exercises.- Index of Symbols and Abbreviations.