Loading timeline…
19481940s
Information Theory
Mathematical framework for quantifying, storing, and communicating information.
Why It Was Important
Claude Shannon's seminal paper 'A Mathematical Theory of Communication' introduced the concept of the 'bit' and provided the mathematical underpinnings for data compression, transmission, and entropy. This theory is essential for modern machine learning, deeply influencing loss functions like cross-entropy.
Who Invented It
Claude Shannon
American mathematician and electrical engineer known as the 'father of information theory'.
Applications
- Data Compression
- Loss Functions (Cross-Entropy)
- Telecommunications
Key Papers
- A Mathematical Theory of Communication
Claude E. Shannon · Bell System Technical Journal · 1948
Videos
Why Information Theory is Important - Computerphile
Computerphile
Entropy (for data science) Clearly Explained!!!
StatQuest with Josh Starmer