The Data Science Handbook

A comprehensive overview of data science covering the analytics, programming, and business skills necessary to master the discipline Finding a good data scientist has been likened to hunting for a unicorn: the required combination of technical skills is simply very hard to find in one person. In addition, good data science is not just rote application of trainable skill sets; it requires the ability to think flexibly about all these areas and understand the connections between them. This book provides a crash course in data science, combining all the necessary skills into a unified discipline. Unlike many analytics books, computer science and software engineering are given extensive coverage since they play such a central role in the daily work of a data scientist. The author also describes classic machine learning algorithms, from their mathematical foundations to real-world applications. Visualization tools are reviewed, and their central importance in data science is highlighted. Classical statistics is addressed to help readers think critically about the interpretation of data and its common pitfalls. The clear communication of technical results, which is perhaps the most undertrained of data science skills, is given its own chapter, and all topics are explained in the context of solving real-world data problems. The book also features: Extensive sample code and tutorials using Python(TM) along with its technical libraries Core technologies of "Big Data," including their strengths and limitations and how they can be used to solve real-world problems Coverage of the practical realities of the tools, keeping theory to a minimum; however, when theory is presented, it is done in an intuitive way to encourage critical thinking and creativity A wide variety of case studies from industry * Practical advice on the realities of being a data scientist today, including the overall workflow, where time is spent, the types of datasets worked on, and the skill sets needed The Data Science Handbook is an ideal resource for data analysis methodology and big data software tools. The book is appropriate for people who want to practice data science, but lack the required skill sets. This includes software professionals who need to better understand analytics and statisticians who need to understand software. Modern data science is a unified discipline, and it is presented as such. This book is also an appropriate reference for researchers and entry-level graduate students who need to learn real-world analytics and expand their skill set. FIELD CADY is the data scientist at the Allen Institute for Artificial Intelligence, where he develops tools that use machine learning to mine scientific literature. He has also worked at Google and several Big Data startups. He has a BS in physics and math from Stanford University, and an MS in computer science from Carnegie Mellon.

Auteur

FIELD CADY is the data scientist at the Allen Institute for Artificial Intelligence, where he develops tools that use machine learning to mine scientific literature.
He has also worked at Google and several Big Data startups. He has a BS in physics and math from Stanford University, and an MS in computer science from Carnegie Mellon.

Contenu

Preface xvii

1 Introduction: Becoming a Unicorn 1

1.1 Aren't Data Scientists Just Overpaid Statisticians? 2

1.2 How is This Book Organized? 3

1.3 How to Use This Book? 3

1.4 Why is It All in Python™, Anyway? 4

1.5 Example Code and Datasets 4

1.6 Parting Words 5

Part I The Stuff You'll Always Use 7

2 The Data Science Road Map 9

2.1 Frame the Problem 10

2.2 Understand the Data: Basic Questions 11

2.3 Understand the Data: Data Wrangling 12

2.4 Understand the Data: Exploratory Analysis 13

2.5 Extract Features 14

2.6 Model 15

2.7 Present Results 15

2.8 Deploy Code 16

2.9 Iterating 16

2.10 Glossary 17

3 Programming Languages 19

3.1 Why Use a Programming Language? What are the Other Options? 19

3.2 A Survey of Programming Languages for Data Science 20

3.2.1 Python 20

3.2.2 R 21

3.2.3 MATLAB® and Octave 21

3.2.4 SAS® 21

3.2.5 Scala® 22

3.3 Python Crash Course 22

3.3.1 A Note on Versions 22

3.3.2 Hello World Script 23

3.3.3 More Complicated Script 23

3.3.4 Atomic Data Types 26

3.4 Strings 27

3.4.1 Comments and Docstrings 28

3.4.2 Complex Data Types 29

3.4.3 Lists 29

3.4.4 Strings and Lists 30

3.4.5 Tuples 31

3.4.6 Dictionaries 31

3.4.7 Sets 32

3.5 Defining Functions 32

3.5.1 For Loops and Control Structures 33

3.5.2 A Few Key Functions 34

3.5.3 Exception Handling 35

3.5.4 Libraries 35

3.5.5 Classes and Objects 35

3.5.6 GOTCHA: Hashable and Unhashable Types 36

3.6 Python's Technical Libraries 37

3.6.1 Data Frames 38

3.6.2 Series 39

3.6.3 Joining and Grouping 40

3.7 Other Python Resources 42

3.8 Further Reading 42

3.9 Glossary 43

3a Interlude: My Personal Toolkit 45

4 Data Munging: String Manipulation, Regular Expressions, and Data Cleaning 47

4.1 The Worst Dataset in the World 48

4.2 How to Identify Pathologies 48

4.3 Problems with Data Content 49

4.3.1 Duplicate Entries 49

4.3.2 Multiple Entries for a Single Entity 49

4.3.3 Missing Entries 49

4.3.4 NULLs 50

4.3.5 Huge Outliers 50

4.3.6 OutofDate Data 50

4.3.7 Artificial Entries 50

4.3.8 Irregular Spacings 51

4.4 Formatting Issues 51

4.4.1 Formatting is Irregular between Different Tables/Columns 51

4.4.2 Extra Whitespace 51

4.4.3 Irregular Capitalization 52

4.4.4 Inconsistent Delimiters 52

4.4.5 Irregular NULL Format 52

4.4.6 Invalid Characters 52

4.4.7 Weird or Incompatible Datetimes 52

4.4.8 Operating System Incompatibilities 53

4.4.9 Wrong Software Versions 53

4.5 Example Formatting Script 54

4.6 Regular Expressions 55

4.6.1 Regular Expression Syntax 56

4.7 Life in the Trenches 60

4.8 Glossary 60

5 Visualizations and Simple Metrics 61

5.1 A Note on Python's Visualization Tools 62

5.2 Example Code 62

5.3 Pie Charts 63

5.4 Bar Charts 65

5.5 Histograms 66

5.6 Means, Standard Deviations, Medians, and Quantiles 69

5.7 Boxplots 70

5.8 Scatterplots 72

5.9 Scatterplots with Logarithmic Axes 74

5.10 Scatter Matrices 76

5.11 Heatmaps 77

5.12 Correlations 78

5.13 Anscombe's Quartet and the Limits of Numbers 80

5.14 Time Series 81 5...