在裡扎等編著的《Spark高級數據分析（影印版
）（英文版）》這本實用書籍中，4位Cloude陽公司
的數據科學家講解了一繫列自包含模式，用於在
Spark中進行大規模數據分析。本書作者們把Spark、
統計原理和現實世界中的數據集合放到一起，通過實
例教你如何解決數據分析問題。
你將從Spark及其生態繫統的介紹開始，然後深
入運用標準技巧的模式——歸類、聚合過濾及異常檢
測等，這些技巧被用於生物基因、安全和金融等行業
。如果你對機器學習和統計學有初步了解，使用Java
、Pytton或者Scala編程，就會發現這些模式對於你
的數據分析應用程序會非常有用。
模式包括：
音樂推薦和Audioscrobbler數據集合
用決策樹分析森林覆蓋
用K均值聚合檢測網絡流量中的異常
用潛在語義分析理解維基百科
用GraphX分析共生網絡
用地理空間和瞬態數據分析紐約市出租車路線的
數據
用蒙地卡羅模擬來估計金融風險
分析基因數據和BDG項目
通過PySpark和Thunder分析神經造影數據

Foreword
Preface
1. Analyzing Big Data
The Challenges of Data Science
Introducing Apache Spark
About This Book
2. Introduction to Data Analysis with Scala and Spark
Scala for Data Scientists
The Spark Programming Model
Record Linkage
Getting Started: The Spark Shell and SparkContext
Bringing Data from the Cluster to the Client
Shipping Code from the Client to the Cluster
Structuring Data with Tuples and Case Classes
Aggregations
Creating Histograms
Summary Statistics for Continuous Variables
Creating Reusable Code for Computing Summary Statistics
Simple Variable Selection and Scoring
Where to Go from Here
3. Recommending Music and the Audioscrobbler Data Set
Data Set
The Alternating Least Squares Recommender Algorithm
Preparing the Data
Building a First Model
Spot Checking Recommendations
Evaluating Recommendation Quality
Computing AUC
Hyperparameter Selection
Making Recommendations
Where to Go from Here
4. Predicting Forest Cover with Decision Trees
Fast Forward to Regression
Vectors and Features
Training Examples
Decision Trees and Forests
Covtype Data Set
Preparing the Data
A First Decision Tree
Decision Tree Hyperparameters
Tuning Decision Trees
Categorical Features Revisited
Random Decision Forests
Making Predictions
Where to Go from Here
5. Anomaly Detection in Network Traffic with K-means Clustering
Anomaly Detection
K-means Clustering
Network Intrusion
KDD Cup 1999 Data Set
A First Take on Clustering
Choosing k
Visualization in R
Feature Normalization
Categorical Variables
Using Labels with Entropy
Clustering in Action
Where to Go from Here
6. Understanding Wikipedia with Latent Semantic Analysis
The Term-Document Matrix
Getting the Data
Parsing and Preparing the Data
Lemmatization
Computing the TF-IDFs
Singular Value Decomposition
Finding Important Concepts
Querying and Scoring with the Low-Dimensional Representation
Term-Term Relevance
Document-Document Relevance
Term-Document Relevance
Multiple-Term Queries
Where to Go from Here
7. Analyzing Co-occurrence Networks with GraphX
The MEDLINE Citation Index: A Network Analysis
Getting the Data
Parsing XML Documents with Scala's XML Library
Analyzing the MeSH Major Topics and Their Co-occurrences
Constructing a Co-occurrence Network with GraphX
Understanding the Structure of Networks
Connected Components
Degree Distribution
Filtering Out Noisy Edges
Processing EdgeTriplets
Analyzing the Filtered Graph
Small-World Networks
Cliques and Clustering Coefficients
Computing Average Path Length with Pregel
Where to Go from Here
8. 6eospatial and Temporal Data Analysis on the New York City Taxi Trip Data
Getting the Data
Working with Temporal and Geospatial Data in Spark
Temporal Data with JodaTime and NScalaTime
Geospatial Data with the Esri Geometry API and Spray
Exploring the Esri Geometry API
Intro to GeoJSON
Preparing the New York City Taxi Trip Data
Handling Invalid Records at Scale
Geospatial Analysis
Sessionization in Spark
Building Sessions: Secondary Sorts in Spark
Where to Go from Here
9. Estimating Financial Risk through Monte Carlo Simulation
Terminology
Methods for Calculating VaR
Variance-Covariance
Historical Simulation
Monte Carlo Simulation
Our Model
Getting the Data
Preprocessing
Determining the Factor Weights
Sampling
The Multivariate Normal Distribution
Running the Trials
Visualizing the Distribution of Returns
Evaluating Our Results
Where to Go from Here
10. Analyzing Genomics Data and the BDG Project
Decoupling Storage from Modeling
Ingesting Genomics Data with the ADAM CLI
Parquet Format and Columnar Storage
Predicting Transcription Factor Binding Sites from ENCODE Data
Querying Genotypes from the 1000 Genomes Project
Where to Go from Here
11. Analyzing Neuroimaging Data with PySpark and Thunder
Overview of PySpark
PySpark Internals
Overview and Installation of the Thunder Library
Loading Data with Thunder
Thunder Core Data Types
Categorizing Neuron Types with Thunder
Where to Go from Here
A.Deeper into Spark
B.Upcoming MLlib Pipelines API
Index