by
James E. Gentle
Table of Contents
Part I. Methods of Computational Statistics
1 Preliminaries ... 5
1.1 Discovering Structure: Data Structures and Structure in Data ... 6
- Multiple Analyses and Multiple Views
- Simple Plots May Reveal the Unexpected
1.2 Modeling and Computational Inference ... 8
- Descriptive Statistics, Inferential Statistics, and Model Building
1.3 The Role of the Empirical Cumulative Distribution Function ... 11
- Statistical Functions of the CDF and the ECDF
- Estimation of Statistical Functions
- Estimation Using the ECDF
- Empirical Quantiles
1.4 The Role of Optimization in Inference ... 15
- Some Comments on Optimization
- Estimation by Minimizing Residuals
- Statistical Properties of Minimum-Residual Estimators
- Least Squares Estimation
- Variance of Least Squares Estimators
- Iteratively Reweighted Least Squares
- Estimation by Maximum Likelihood
- Statistical Properties of MLE
- EM Methods
1.5 Inference about Functions ... 30
- Functions of Parameters and Functions of Estimators
- Linear Estimators
1.6 Probability Statements in Statistical Inference ... 32
- Tests of Hypotheses
- Confidence Intervals
Exercises ... 35
2 Monte Carlo Methods for Statistical Inference ... 39
2.1 Generation of Random Numbers ... 40
- Inverse CDF Method
- Acceptance/Rejection Methods
- Use of Conditional Distributions
- Acceptance/Rejection Method Using a Markov Chain
- Generation of Multivariate Random Variates
- Gibbs Sampling
- Probability Densities Known Only Proportionally
- Data-Based Random Number Generation
2.2 Monte Carlo Estimation ... 53
- Estimation of a Definite Integral
- Estimation of the Variance
- Estimating the Variance Using Batch Means
- Convergence of Iterative Monte Carlo and Mixing of the Markov Chain
- Monte Carlo, Iterative Monte Carlo, and Simulation
2.3 Simulation of Data from a Hypothesized Model: Monte Carlo Tests ... 58
2.4 Simulation of Data from a Fitted Model: ``Parametric Bootstraps'' ... 60
2.5 Random Sampling from Data ... 60
2.6 Reducing Variance in Monte Carlo Methods ... 61
- Importance Sampling
- Control Variates
2.7 Acceleration of Markov Chain Monte Carlo Methods ... 65
Exercises ... 66
3 Randomization and Data Partitioning ... 69
3.1 Randomization Methods ... 70
3.2 Cross Validation for Smoothing and Fitting ... 74
3.3 Jackknife Methods ... 76
- Jackknife Variance Estimate
- Jackknife Bias Correction
- Higher-Order Bias Corrections
- The Generalized Jackknife
- The Delete-$k$ Jackknife
Further Reading ... 82
Exercises ... 83
4 Bootstrap Methods ... 85
4.1 Bootstrap Bias Corrections ... 86
4.2 Bootstrap Estimation of Variance ... 88
4.3 Bootstrap Confidence Intervals ... 89
- Basic Intervals
- Bootstrap-$t$ Intervals
- Bootstrap Percentile Confidence Intervals
- Confidence Intervals Based on Transformations
- Correcting the Bias in Intervals Due to Bias in the Estimator or to Lack of Symmetry
4.4 Bootstrapping Data with Dependencies ... 93
4.5 Variance Reduction in Monte Carlo Bootstrap ... 94
- Jackknife After Bootstrap
- The Bootstrap Estimate of the Bias of a Plug-In Estimator
- Balanced Resampling
Further Reading ... 96
Exercises ... 97
5 Tools for Identification of Structure in Data ... 99
5.1 Linear Structure and Other Geometric Properties ... 100
5.2 Linear Transformations ... 101
- Orthogonal Transformations
- Gram-Schmidt Orthogonalization
- Geometric Transformations
- Rotations
- Projections
5.3 General Transformations of the Coordinate System ... 108
5.4 Measures of Similarity and Dissimilarity ... 109
- Similarities: Covariances and Correlations
- Similarities When Some Variables Are Categorical
- Similarities among Functional Observations
- Similarities between Groups of Variables
- Dissimilarities: Distances
- Other Dissimilarities Based on Distances
- Dissimilarities in Anisometric Coordinate Systems: Sphering Data
- Properties of Dissimilarities
- Dissimilarities between Groups of Observations
- Effects of Transformations of the Data
- Outlying Observations and Collinear Variables
- Multidimensional Scaling: Determining Observations that Yield a Given Distance Matrix
5.5 Data Mining ... 123
5.6 Computational Feasibility ... 124
Exercises ... 125
6 Estimation of Functions ... 127
6.1 General Methods for Estimating Functions ... 128
- Inner Products and Norms
- Complete Spaces
- Measures for Comparing Two Functions
- Basis Sets in Function Spaces
- Series Expansions in Basis Functions
- Orthogonal Polynomials
- Multivariate Orthogonal Polynomials
- Function Decomposition and Estimation of the Coefficients in an Orthogonal Expansion
- Splines
- Interpolating Splines
- Smoothing Splines
- Choice of Knots in Smoothing Splines
- Multivariate Splines
- Kernel Methods
6.2 Pointwise Properties of Function Estimators ... 143
- Bias
- Variance
- Mean Squared Error
- Mean Absolute Error
- Consistency
6.3 Global Properties of Estimators of Functions ... 146
- Integrated Bias and Variance
- Integrated Mean Squared Error and Mean Absolute Error
- Mean SAE
- Large-Sample Statistical Properties
- Other Global Properties of Estimators of Functions
Exercises ... 150
7 Graphical Methods in Computational Statistics ... 153
7.1 Viewing One, Two, or Three Variables ... 155
- Histograms and Variations
- The Empirical Cumulative Distribution Function and q-q Plots
- Smoothing
- Graphing Continuous Functions
- B\'{e}zier Curves
- Continuous Densities
- Representation of the Third Dimension
- Contours in Three Dimensions
7.2 Viewing Multivariate Data ... 168
- Projections
- Conditioning Plots
- Noncartesian Displays
- Glyphs and Icons
- Parallel Coordinates: Points Become Broken Line Segments
- Trigonometric Series: Points Become Curves
- Cone Plots
- Exploring Data with Noncartesian Displays
- Roping, Brushing, and Linking
- Rotations and Dynamical Graphics
- Displays of Large Data Sets
- Data Analysis and Human Perception
7.3 Hardware and Low-Level Software for Graphics ... 184
- Speed and Resolution
- Representation of Color
- Low-Level Software
7.4 Software for Graphics Applications ... 186
Further Reading ... 188
Exercises ... 188
Part II. Exploring Data Density and Structure
8 Estimation of Probability Density Functions Using Parametric Models ... 197
8.1 Fitting a Parametric Probability Distribution ... 198
- Maximum Likelihood Methods
- Fitting by Matching Moments
- Fitting by Matching Quantiles
- Statistical Properties of Parametric Family Estimators ... 199
8.2 General Families of Probability Distributions ... 199
- Pearson Curves
- Other General Parametric Univariate Families of Distribution Functions
8.3 Mixtures of Parametric Families ... 202
Exercises ... 203
9 Nonparametric Estimation of Probability Density Functions ... 205
9.1 The Likelihood Function ... 206
9.2 Histogram Estimators ... 208
- Some Properties of the Histogram Estimator
- Pointwise and Binwise Properties
- Asymptotic MISE (or AMISE) of Histogram Estimators
- Bin Sizes
- Bin Shapes
- Other Density Estimators Related to the Histogram
9.3 Kernel Estimators ... 217
- Rosenblatt's Histogram Estimator; Kernels
- Properties of Kernel Estimators
- Choice of Kernels
- Computation of Kernel Density Estimators
9.4 Choice of Window Widths ... 222
9.5 Orthogonal Series Estimators ... 222
9.6 Other Methods of Density Estimation ... 224
- Filtered Kernel Methods
- Alternating Kernel and Mixture Methods
- Methods of Comparisons of Methods
Exercises ... 226
10 Structure in Data ... 233
10.1 Clustering and Classification ... 237
- Clustering
- K-Means Clustering
- Choosing the Number of Clusters
- Hierarchical Clustering
- Agglomerative Hierarchical Clustering
- Model-Based Hierarchical Clustering
- Divisive Hierarchical Clustering
- Other Divisive Clustering Schemes
- Clustering and Classification by Space Tessellations
- Meanings of Clusters; Conceptual Clustering
- Fuzzy Clustering
- Clustering and Transformations of the Data
- Clustering of Variables
- Comparing Clusterings
- Computational Complexity of Clustering
10.2 Ordering and Ranking Multivariate Data ... 255
- Minimal Spanning Trees
- Ranking Data Using Minimal Spanning Trees
- Ranking Data Using Convex Container Hulls
- Ranking Data Using Location Depth
- Ordering by Clustering
- Clustering by Ordering
- Nearest Neighbors and $k$-$d$-Trees
- Ordering and Ranking of Transformed Data
10.3 Linear Principal Components ... 264
- The Probability Model Underlying Principal Components Analysis
- Principal Components Analysis of Data
- Dimension Reduction by Principal Components Analysis
- Principal Components and Transformations of the Data
- Principal Components of Observations
- Principal Components Directly from the Data Matrix
- Computational Issues
- PCA for Clustering
- Robustness of Principal Components
10.4 Variants of Principal Components ... 276
- Factor Analysis
- The Probability Model Underlying Factor Analysis
- Factor Analysis of Data
- Latent Semantic Indexing
- Linear Independent Components Analysis
10.5 Projection Pursuit ... 281
- The Probability Model Underlying Projection Pursuit
- Projection Indexes for the Probability Model
- Projection Pursuit in Data
- Exploratory Projection Pursuit
- Example
- Computational Issues
10.6 Other Methods for Identifying Structure ... 289
- Independent Components Analysis
10.7 Higher Dimensions ... 290
- Data Sparsity in Higher Dimensions
- Volumes of Hyperspheres and Hypercubes
- The Curse of Dimensionality
- Tiling Space
Exercises ... 294
11 Statistical Models of Dependencies ... 299
11.1 Regression and Classification Models ... 301
- Generalized Models
- Classification Models
- Models of Sequential Dependencies
- Data and Models
- Transformations
- Piecewise Models
11.2 Probability Distributions in Models ... 308
- Hierarchical Models
- Probability Distributions in Models of Sequential Dependencies
11.3 Fitting Models to Data ... 311
- The Mechanics of Fitting
- Estimation by Minimizing Residuals
- Variations on Minimizing Residuals
- Comparisons of Estimators Defined by Minimum Residuals
- Orthogonal Residuals
- Projection Pursuit Regression
- Classification and Regression Trees
- Combining Classifications
- Kernel Methods and Support Vector Machines
- Fitting Models of Sequential Dependencies
- Transformations to Make Data Fit Models
- Variance Stabilization
- Transformations of the Independent Variables
- Transformations of the Box-Cox Type
- Alternating Conditional Expectation
- Additivity and Variance Stabilization
- Local Fitting
- Assessing the Fit of a Model
Exercises ... 333
Appendices
A Monte Carlo Studies in Statistics ... 337
A.1 Simulation as an Experiment ... 338
A.2 Reporting Simulation Experiments ... 339
A.3 An Example ... 340
- The Problem
- The Design of the Experiment
- The Experiment
- Reporting the Results
A.4 Computer Experiments ... 347
Exercises ... 349
B Software for Random Number Generation ... 351
B.1 The User Interface for Random Number Generators ... 353
B.2 Controlling the Seeds in Monte Carlo Studies ... 354
B.3 Random Number Generation in IMSL Libraries ... 354
- Controlling the State of the Generators
B.4 Random Number Generation in S-Plus and R ... 357
- Controlling the State of the Generators
- Monte Carlo in S-Plus and R
C Notation and Definitions ... 363
D Solutions and Hints for Selected Exercises ... 377
Bibliography ... 385
Literature in Computational Statistics ... 386
Resources Available over the Internet ... 387
References for Software Packages ... 389
References to the Literature ... 389
Author Index ... 409
Subject Index ... 415