The Antivirus Hacker's Handbook

Hack your antivirus software to stamp out future vulnerabilities

The Antivirus Hacker's Handbook guides you through the process of reverse engineering antivirus software. You explore how to detect and exploit vulnerabilities that can be leveraged to improve future software design, protect your network, and anticipate attacks that may sneak through your antivirus' line of defense. You'll begin building your knowledge by diving into the reverse engineering process, which details how to start from a finished antivirus software program and work your way back through its development using the functions and other key elements of the software. Next, you leverage your new knowledge about software development to evade, attack, and exploit antivirus software--all of which can help you strengthen your network and protect your data.

While not all viruses are damaging, understanding how to better protect your computer against them can help you maintain the integrity of your network.

• Discover how to reverse engineer your antivirus software

• Explore methods of antivirus software evasion

• Consider different ways to attack and exploit antivirus software

• Understand the current state of the antivirus software market, and get recommendations for users and vendors who are leveraging this software

The Antivirus Hacker's Handbook is the essential reference for software reverse engineers, penetration testers, security researchers, exploit writers, antivirus vendors, and software engineers who want to understand how to leverage current antivirus software to improve future applications.

Auteur

JOXEAN KORET is a security researcher at Coseinc, a Singapore-based security services firm. He is an in-demand speaker at international
security conferences, and a well-regarded specialist in reverse-engineering, vulnerability research, and malware analysis.

ELIAS BACHAALANY
is a computer programmer,
reverse engineer, and technical
writer employed by Microsoft. An
authority in multiple programming
languages, database design, and web development, he played an instrumental role in the IDAPython project and IDA Pro's scripting
facilities development.

Résumé
Hack your antivirus software to stamp out future vulnerabilities The Antivirus Hacker's Handbook guides you through the process of reverse engineering antivirus software. You explore how to detect and exploit vulnerabilities that can be leveraged to improve future software design, protect your network, and anticipate attacks that may sneak through your antivirus' line of defense. You'll begin building your knowledge by diving into the reverse engineering process, which details how to start from a finished antivirus software program and work your way back through its development using the functions and other key elements of the software. Next, you leverage your new knowledge about software development to evade, attack, and exploit antivirus softwareall of which can help you strengthen your network and protect your data. While not all viruses are damaging, understanding how to better protect your computer against them can help you maintain the integrity of your network.

• Discover how to reverse engineer your antivirus software
• Explore methods of antivirus software evasion
• Consider different ways to attack and exploit antivirus software
• Understand the current state of the antivirus software market, and get recommendations for users and vendors who are leveraging this software The Antivirus Hacker's Handbook is the essential reference for software reverse engineers, penetration testers, security researchers, exploit writers, antivirus vendors, and software engineers who want to understand how to leverage current antivirus software to improve future applications.

Contenu
Introduction xix

Part I Antivirus Basics 1

Chapter 1 Introduction to Antivirus Software 3

What Is Antivirus Software? 3

Antivirus Software: Past and Present 4

Antivirus Scanners, Kernels, and Products 5

Typical Misconceptions about Antivirus Software 6

Antivirus Features 7

Basic Features 7

Making Use of Native Languages 7

Scanners 8

Signatures 8

Compressors and Archives 9

Unpackers 10

Emulators 10

Miscellaneous File Formats 11

Advanced Features 11

Packet Filters and Firewalls 11

Self-Protection 12

Anti-Exploiting 12

Summary 13

Chapter 2 Reverse-Engineering the Core 15

Reverse-Engineering Tools 15

Command-Line Tools versus GUI Tools 16

Debugging Symbols 17

Tricks for Retrieving Debugging Symbols 17

Debugging Tricks 20

Backdoors and Configuration Settings 21

Kernel Debugging 23

Debugging User-Mode Processes with a Kernel-Mode Debugger 25

Analyzing AV Software with Command-Line Tools 27

Porting the Core 28

A Practical Example: Writing Basic Python Bindings for Avast for Linux 29

A Brief Look at Avast for Linux 29

Writing Simple Python Bindings for Avast for Linux 32

The Final Version of the Python Bindings 37

A Practical Example: Writing Native C/C++ Tools for Comodo Antivirus for Linux 37

Other Components Loaded by the Kernel 55

Summary 56

Chapter 3 The Plug-ins System 57

Understanding How Plug-ins Are Loaded 58

A Full-Featured Linker in Antivirus Software 58

Understanding Dynamic Loading 59

Advantages and Disadvantages of the Approaches for Packaging Plug-ins 60

Types of Plug-ins 62

Scanners and Generic Routines 63

File Format and Protocol Support 64

Heuristics 65

Bayesian Networks 66

Bloom Filters 67

Weights-Based Heuristics 68

Some Advanced Plug-ins 69

Memory Scanners 69

Non-native Code 70

Scripting Languages 72

Emulators 73

Summary 74

Chapter 4 Understanding Antivirus Signatures 77

Typical Signatures 77

Byte-Streams 78

Checksums 78

Custom Checksums 79

Cryptographic Hashes 80

Advanced Signatures 80

Fuzzy Hashing 81

Graph-Based Hashes for Executable Files 83

Summary 85

Chapter 5 The Update System 87

Understanding the Update Protocols 88

Support for SSL/TLS 89

Verifying the Update Files 91

Dissecting an Update Protocol 92

When Protection Is Done Wrong 100

Summary 101

Part II Antivirus Software Evasion 103

Chapter 6 Antivirus Software Evasion 105

Who Uses Antivirus Evasion Techniques? 106

Discovering Where and How Malware Is Detected 107

Old Tricks for Determining Where Malware Is Detected: Divide and Conquer 107

Evading a Simple Signature-Based Detection with the Divide and Conquer Trick 108

Binary Instrumentation and Taint Analysis 113

Summary 114

Chapter 7 Evading Signatures 117

File Formats: Corner Cases and Undocumented Cases 118

Evading a Real Signature 118

Evasion Tips and Tricks for Specific File Formats 124

PE Files 124

JavaScript 126

String Encoding 127

Executing Code on the Fly 128

Hiding the Logic: Opaque Predicates and Junk Code 128

PDF 129

Summary 131

Chapter 8 Evading Scanners 133

Generic Evasion Tips and Tricks 133 Fingerprin...