- 1 Rationale
- 2 Preamble
- 2.1 Install Libraries
- 2.2 Load Libraries
- 3 Simulate Data
- 4 Descriptive statistics
- 4.1 Sample
- 4.2 Distribution
- 4.3 Central Tendency
- 4.4 Dispersion
- 4.5 Summary Statistics
- 4.6 Distribution Plots
- 4.6.1 Histogram
- 4.6.1.1 Base R
- 4.6.1.2
ggplot2 - 4.6.2 Histogram overlaid with density plot and rug plot
- 4.6.2.1 Base R
- 4.6.2.2
ggplot2 - 4.6.3 Density Plot
- 4.6.3.1 Base R
- 4.6.3.2
ggplot2 - 4.6.4 Box and whisker plot (boxplot)
- 4.6.4.1 Base R
- 4.6.4.2
ggplot2 - 4.6.5 Violin plot
- 4.6.5.1 Base R
- 4.6.5.2
ggplot2
- 5 Bivariate Associations
- 5.1 Correlation Coefficients
- 5.1.1 Pearson Correlation
- 5.1.2 Spearman Correlation
- 5.1.3 Xi (ξ)
- 5.2 Scatterplot
- 5.2.1 Base R
- 5.2.2
ggplot2 - 5.3 Scatterplot with Marginal Density Plot
- 5.4 High Density Scatterplot
- 5.5 Data Ellipse
- 5.6 Visually Weighted Regression
- 6 Basic inferential statistics
- 6.1 Tests of Normality
- 6.1.1 Shapiro-Wilk test of normality
- 6.1.2 Test of multivariate normality
- 6.2 Statistical decision tree
- 6.3 Tests of systematic missingness (i.e., whether missingness on a variable depends on other variables)
- 6.3.1 Little’s MCAR Test
- 6.4 Multivariate Associations
- 6.4.1 Correlation Matrix
- 6.4.1.1 Pearson Correlations
- 6.4.1.2 Spearman Correlations
- 6.4.1.3 Partial Correlations
- 6.4.2 Correlogram
- 6.4.3 Scatterplot matrix
- 6.4.4 Pairs panels
- 6.5 Effect Plots
- 6.5.1 Multiple Regression Model
- 6.5.2 Multilevel Regression Model
- 7 Session Info
Exploratory Data Analysis
1 Rationale
Exploratory data analysis is important for understanding your data, checking for data issues/errors, and checking assumptions for different statistical models.
LOOK AT YOUR DATA—this is one of the most overlooked steps in data analysis!
2 Preamble
2.1 Install Libraries
#install.packages("remotes")
#remotes::install_github("DevPsyLab/petersenlab")2.2 Load Libraries
library("petersenlab")
library("car")
library("vioplot")
library("ellipse")
library("nlme")
library("effects")
library("corrplot")
library("ggplot2")
library("psych")
library("tidyverse")
library("purrr")
library("naniar")
library("mvnormtest")
library("ggExtra")
library("XICOR")3 Simulate Data
set.seed(52242)
n <- 1000
ID <- rep(1:100, each = 10)
predictor <- rbeta(n, 1.5, 5) * 100
outcome <- predictor + rnorm(n, mean = 0, sd = 20) + 50
predictorOverplot <- sample(1:50, n, replace = TRUE)
outcomeOverplot <- predictorOverplot + sample(1:75, n, replace = TRUE)
categorical1 <- sample(1:5, size = n, replace = TRUE)
categorical2 <- sample(1:5, size = n, replace = TRUE)
mydata <- data.frame(
ID = ID,
predictor = predictor,
outcome = outcome,
predictorOverplot = predictorOverplot,
outcomeOverplot = outcomeOverplot,
categorical1 = categorical1,
categorical2 = categorical2)
mydata[sample(1:n, size = 10), "predictor"] <- NA
mydata[sample(1:n, size = 10), "outcome"] <- NA
mydata[sample(1:n, size = 10), "predictorOverplot"] <- NA
mydata[sample(1:n, size = 10), "outcomeOverplot"] <- NA
mydata[sample(1:n, size = 30), "categorical1"] <- NA
mydata[sample(1:n, size = 70), "categorical2"] <- NA4 Descriptive statistics
round(data.frame(psych::describe(mydata)), 2)vars <dbl> | n <dbl> | mean <dbl> | sd <dbl> | median <dbl> | trimmed <dbl> | mad <dbl> | min <dbl> | max <dbl> | ||
|---|---|---|---|---|---|---|---|---|---|---|
| ID | 1 | 1000 | 50.50 | 28.88 | 50.50 | 50.50 | 37.06 | 1.00 | 100.00 | |
| predictor | 2 | 990 | 22.64 | 15.42 | 19.61 | 21.02 | 14.40 | 0.11 | 81.69 | |
| outcome | 3 | 990 | 73.36 | 25.88 | 73.17 | 73.02 | 25.71 | 5.82 | 171.90 | |
| predictorOverplot | 4 | 990 | 25.54 | 14.45 | 26.00 | 25.62 | 17.79 | 1.00 | 50.00 | |
| outcomeOverplot | 5 | 990 | 63.53 | 25.58 | 63.00 | 63.65 | 28.17 | 2.00 | 123.00 | |
| categorical1 | 6 | 970 | 2.96 | 1.40 | 3.00 | 2.95 | 1.48 | 1.00 | 5.00 | |
| categorical2 | 7 | 930 | 2.95 | 1.39 | 3.00 | 2.94 | 1.48 | 1.00 | 5.00 |
4.1 Sample
- Check the sample size (N)
- Is the sample size in the data the expected sample size? Are there cases (participants) that are missing? Are there cases that should not be there?
- Here is the sample size:
length(unique(mydata$ID))[1] 100- Check the extent of missingness
- How much data are missing in the model variables—including the predictor, outcome, and covariates?
- Here are the proportion of missing data in each variable:
map(mydata, ~mean(is.na(.))) %>% t %>% t [,1]
ID 0
predictor 0.01
outcome 0.01
predictorOverplot 0.01
outcomeOverplot 0.01
categorical1 0.03
categorical2 0.074.2 Distribution
- Frequencies
- Examine the frequencies of categorical variables:
mydata %>%
select(categorical1, categorical2) %>%
sapply(function(x) table(x, useNA = "always")) %>%
t() 1 2 3 4 5 <NA>
categorical1 196 204 191 199 180 30
categorical2 195 179 193 202 161 704.3 Central Tendency
- Mean
round(colMeans(mydata, na.rm = TRUE), 2) ID predictor outcome predictorOverplot
50.50 22.64 73.36 25.54
outcomeOverplot categorical1 categorical2
63.53 2.96 2.95 round(apply(mydata, 2, function(x) mean(x, na.rm = TRUE)), 2) ID predictor outcome predictorOverplot
50.50 22.64 73.36 25.54
outcomeOverplot categorical1 categorical2
63.53 2.96 2.95 mydata %>%
summarise(across(everything(),
.fns = list(mean = ~ mean(., na.rm = TRUE)))) %>%
round(., 2)ID_mean <dbl> | predictor_mean <dbl> | outcome_mean <dbl> | predictorOverplot_mean <dbl> | outcomeOverplot_mean <dbl> | categorical1_mean <dbl> | categorical2_mean <dbl> |
|---|---|---|---|---|---|---|
| 50.5 | 22.64 | 73.36 | 25.54 | 63.53 | 2.96 | 2.95 |
- Median
round(apply(mydata, 2, function(x) median(x, na.rm = TRUE)), 2) ID predictor outcome predictorOverplot
50.50 19.61 73.17 26.00
outcomeOverplot categorical1 categorical2
63.00 3.00 3.00 mydata %>%
summarise(across(everything(),
.fns = list(median = ~ median(., na.rm = TRUE)))) %>%
round(., 2)ID_median <dbl> | predictor_median <dbl> | outcome_median <dbl> | predictorOverplot_median <dbl> | outcomeOverplot_median <dbl> | categorical1_median <dbl> | |
|---|---|---|---|---|---|---|
| 50.5 | 19.61 | 73.17 | 26 | 63 | 3 |
- Mode
round(apply(mydata, 2, function(x) Mode(x, multipleModes = "mean")), 2) ID predictor outcome predictorOverplot
50.50 22.64 73.36 35.00
outcomeOverplot categorical1 categorical2
63.33 2.00 4.00 mydata %>%
summarise(across(everything(),
.fns = list(mode = ~ Mode(., multipleModes = "mean")))) %>%
round(., 2)ID_mode <dbl> | predictor_mode <dbl> | outcome_mode <dbl> | predictorOverplot_mode <dbl> | outcomeOverplot_mode <dbl> | categorical1_mode <dbl> | categorical2_mode <dbl> |
|---|---|---|---|---|---|---|
| 50.5 | 22.64 | 73.36 | 35 | 63.33 | 2 | 4 |
Compute all of these measures of central tendency:
mydata %>%
summarise(across(everything(),
.fns = list(mean = ~ mean(., na.rm = TRUE),
median = ~ median(., na.rm = TRUE),
mode = ~ Mode(., multipleModes = "mean")),
.names = "{.col}.{.fn}")) %>%
round(., 2) %>%
pivot_longer(cols = everything(),
names_to = c("variable","index"),
names_sep = "\\.") %>%
pivot_wider(names_from = index,
values_from = value)variable <chr> | mean <dbl> | median <dbl> | mode <dbl> | |
|---|---|---|---|---|
| ID | 50.50 | 50.50 | 50.50 | |
| predictor | 22.64 | 19.61 | 22.64 | |
| outcome | 73.36 | 73.17 | 73.36 | |
| predictorOverplot | 25.54 | 26.00 | 35.00 | |
| outcomeOverplot | 63.53 | 63.00 | 63.33 | |
| categorical1 | 2.96 | 3.00 | 2.00 | |
| categorical2 | 2.95 | 3.00 | 4.00 |
4.4 Dispersion
- Standard deviation
- Observed minimum and maximum (vis-à-vis possible minimum and maximum)
- Skewness
- Kurtosis
Compute all of these measures of dispersion:
mydata %>%
summarise(across(everything(),
.fns = list(SD = ~ sd(., na.rm = TRUE),
min = ~ min(., na.rm = TRUE),
max = ~ max(., na.rm = TRUE),
skewness = ~ skew(., na.rm = TRUE),
kurtosis = ~ kurtosi(., na.rm = TRUE)),
.names = "{.col}.{.fn}")) %>%
round(., 2) %>%
pivot_longer(cols = everything(),
names_to = c("variable","index"),
names_sep = "\\.") %>%
pivot_wider(names_from = index,
values_from = value)variable <chr> | SD <dbl> | min <dbl> | max <dbl> | skewness <dbl> | kurtosis <dbl> |
|---|---|---|---|---|---|
| ID | 28.88 | 1.00 | 100.00 | 0.00 | -1.20 |
| predictor | 15.42 | 0.11 | 81.69 | 0.97 | 0.71 |
| outcome | 25.88 | 5.82 | 171.90 | 0.18 | 0.11 |
| predictorOverplot | 14.45 | 1.00 | 50.00 | -0.05 | -1.19 |
| outcomeOverplot | 25.58 | 2.00 | 123.00 | -0.04 | -0.75 |
| categorical1 | 1.40 | 1.00 | 5.00 | 0.03 | -1.29 |
| categorical2 | 1.39 | 1.00 | 5.00 | 0.01 | -1.27 |
Consider transforming data if skewness > |0.8| or if kurtosis > |3.0|.
4.5 Summary Statistics
Add summary statistics to the bottom of correlation matrices in papers:
cor.table(mydata, type = "manuscript")ID <chr> | predictor <chr> | outcome <chr> | predictorOverplot <chr> | outcomeOverplot <chr> | categorical1 <chr> | categorical2 <chr> | |
|---|---|---|---|---|---|---|---|
| 1. ID | 1.00 | ||||||
| 2. predictor | -.02 | 1.00 | |||||
| 3. outcome | -.03 | .64*** | 1.00 | ||||
| 4. predictorOverplot | .03 | .04 | .03 | 1.00 | |||
| 5. outcomeOverplot | .00 | .04 | .06† | .57*** | 1.00 | ||
| 6. categorical1 | .04 | .00 | -.02 | .02 | .01 | 1.00 | |
| 7. categorical2 | .01 | -.05 | -.02 | .04 | .04 | -.09* | 1.00 |
summaryTable <- mydata %>%
summarise(across(everything(),
.fns = list(n = ~ length(na.omit(.)),
missingness = ~ mean(is.na(.)) * 100,
M = ~ mean(., na.rm = TRUE),
SD = ~ sd(., na.rm = TRUE),
min = ~ min(., na.rm = TRUE),
max = ~ max(., na.rm = TRUE),
skewness = ~ skew(., na.rm = TRUE),
kurtosis = ~ kurtosi(., na.rm = TRUE)),
.names = "{.col}.{.fn}")) %>%
pivot_longer(cols = everything(),
names_to = c("variable","index"),
names_sep = "\\.") %>%
pivot_wider(names_from = index,
values_from = value)
summaryTableTransposed <- summaryTable[-1] %>%
t() %>%
as.data.frame() %>%
setNames(summaryTable$variable) %>%
round(., digits = 2)
summaryTableTransposedID <dbl> | predictor <dbl> | outcome <dbl> | predictorOverplot <dbl> | outcomeOverplot <dbl> | categorical1 <dbl> | categorical2 <dbl> | |
|---|---|---|---|---|---|---|---|
| n | 1000.00 | 990.00 | 990.00 | 990.00 | 990.00 | 970.00 | 930.00 |
| missingness | 0.00 | 1.00 | 1.00 | 1.00 | 1.00 | 3.00 | 7.00 |
| M | 50.50 | 22.64 | 73.36 | 25.54 | 63.53 | 2.96 | 2.95 |
| SD | 28.88 | 15.42 | 25.88 | 14.45 | 25.58 | 1.40 | 1.39 |
| min | 1.00 | 0.11 | 5.82 | 1.00 | 2.00 | 1.00 | 1.00 |
| max | 100.00 | 81.69 | 171.90 | 50.00 | 123.00 | 5.00 | 5.00 |
| skewness | 0.00 | 0.97 | 0.18 | -0.05 | -0.04 | 0.03 | 0.01 |
| kurtosis | -1.20 | 0.71 | 0.11 | -1.19 | -0.75 | -1.29 | -1.27 |
4.6 Distribution Plots
See here for resources for creating figures in R.
4.6.1 Histogram
4.6.1.1 Base R
hist(mydata$outcome)
4.6.1.2
ggplot2
ggplot(mydata, aes(x = outcome)) +
geom_histogram(color = 1)`stat_bin()` using `bins = 30`. Pick better value with `binwidth`.Warning: Removed 10 rows containing non-finite outside the scale range
(`stat_bin()`).
4.6.2 Histogram overlaid with density plot and rug plot
4.6.2.1 Base R
hist(mydata$outcome, prob = TRUE)
lines(density(mydata$outcome, na.rm = TRUE))
rug(mydata$outcome)
4.6.2.2
ggplot2
ggplot(mydata, aes(x = outcome)) +
geom_histogram(aes(y = after_stat(density)), color = 1) +
geom_density() +
geom_rug()`stat_bin()` using `bins = 30`. Pick better value with `binwidth`.Warning: Removed 10 rows containing non-finite outside the scale range
(`stat_bin()`).Warning: Removed 10 rows containing non-finite outside the scale range
(`stat_density()`).
4.6.3 Density Plot
4.6.3.1 Base R
plot(density(mydata$outcome, na.rm = TRUE))
4.6.3.2
ggplot2
ggplot(mydata, aes(x = outcome)) +
geom_density()Warning: Removed 10 rows containing non-finite outside the scale range
(`stat_density()`).
4.6.4 Box and whisker plot (boxplot)
4.6.4.1 Base R
boxplot(mydata$outcome, horizontal = TRUE)
4.6.4.2
ggplot2
ggplot(mydata, aes(x = outcome)) +
geom_boxplot()Warning: Removed 10 rows containing non-finite outside the scale range
(`stat_boxplot()`).
4.6.5 Violin plot
4.6.5.1 Base R
vioplot(na.omit(mydata$outcome), horizontal = TRUE)
4.6.5.2
ggplot2
ggplot(mydata, aes(x = "", y = outcome)) +
geom_violin()Warning: Removed 10 rows containing non-finite outside the scale range
(`stat_ydensity()`).
5 Bivariate Associations
For more advanced scatterplots, see here.
5.1 Correlation Coefficients
5.1.1 Pearson Correlation
cor(mydata, use = "pairwise.complete.obs") ID predictor outcome predictorOverplot
ID 1.000000000 -0.019094589 -0.02501796 0.02984671
predictor -0.019094589 1.000000000 0.63517610 0.04350431
outcome -0.025017962 0.635176098 1.00000000 0.02976820
predictorOverplot 0.029846711 0.043504314 0.02976820 1.00000000
outcomeOverplot 0.003150872 0.043344473 0.06289224 0.56564445
categorical1 0.040442534 -0.002422711 -0.01532232 0.02498938
categorical2 0.008920415 -0.046927177 -0.01868628 0.03880135
outcomeOverplot categorical1 categorical2
ID 0.003150872 0.040442534 0.008920415
predictor 0.043344473 -0.002422711 -0.046927177
outcome 0.062892238 -0.015322318 -0.018686282
predictorOverplot 0.565644447 0.024989385 0.038801346
outcomeOverplot 1.000000000 0.011871644 0.043682632
categorical1 0.011871644 1.000000000 -0.086310192
categorical2 0.043682632 -0.086310192 1.000000000cor.test( ~ predictor + outcome, data = mydata)
Pearson's product-moment correlation
data: predictor and outcome
t = 25.718, df = 978, p-value < 2.2e-16
alternative hypothesis: true correlation is not equal to 0
95 percent confidence interval:
0.5962709 0.6711047
sample estimates:
cor
0.6351761 cor.table(mydata)ID <chr> | predictor <chr> | outcome <chr> | predictorOverplot <chr> | outcomeOverplot <chr> | categorical1 <chr> | categorical2 <chr> | |
|---|---|---|---|---|---|---|---|
| 1. ID.r | 1.00 | -.02 | -.03 | .03 | .00 | .04 | .01 |
| 2. sig | NA | .55 | .43 | .35 | .92 | .21 | .79 |
| 3. n | 1000 | 990 | 990 | 990 | 990 | 970 | 930 |
| 4. predictor.r | -.02 | 1.00 | .64*** | .04 | .04 | .00 | -.05 |
| 5. sig | .55 | NA | .00 | .17 | .18 | .94 | .15 |
| 6. n | 990 | 990 | 980 | 981 | 980 | 960 | 922 |
| 7. outcome.r | -.03 | .64*** | 1.00 | .03 | .06† | -.02 | -.02 |
| 8. sig | .43 | .00 | NA | .35 | .05 | .64 | .57 |
| 9. n | 990 | 980 | 990 | 980 | 980 | 960 | 920 |
| 10. predictorOverplot.r | .03 | .04 | .03 | 1.00 | .57*** | .02 | .04 |
5.1.2 Spearman Correlation
cor(mydata, use = "pairwise.complete.obs", method = "spearman") ID predictor outcome predictorOverplot
ID 1.000000000 -0.01181085 -0.02805450 0.02925155
predictor -0.011810853 1.00000000 0.60003760 0.02672008
outcome -0.028054497 0.60003760 1.00000000 0.02460831
predictorOverplot 0.029251546 0.02672008 0.02460831 1.00000000
outcomeOverplot 0.001416626 0.03278463 0.07073642 0.54841297
categorical1 0.040888369 -0.01162260 -0.01906631 0.02350886
categorical2 0.009137467 -0.03763251 -0.02273699 0.04062487
outcomeOverplot categorical1 categorical2
ID 0.001416626 0.04088837 0.009137467
predictor 0.032784628 -0.01162260 -0.037632515
outcome 0.070736421 -0.01906631 -0.022736992
predictorOverplot 0.548412967 0.02350886 0.040624873
outcomeOverplot 1.000000000 0.01505255 0.044184959
categorical1 0.015052553 1.00000000 -0.086290083
categorical2 0.044184959 -0.08629008 1.000000000cor.test( ~ predictor + outcome, data = mydata, method = "spearman")
Spearman's rank correlation rho
data: predictor and outcome
S = 62740170, p-value < 2.2e-16
alternative hypothesis: true rho is not equal to 0
sample estimates:
rho
0.6000376 cor.table(mydata, correlation = "spearman")ID <chr> | predictor <chr> | outcome <chr> | predictorOverplot <chr> | outcomeOverplot <chr> | categorical1 <chr> | categorical2 <chr> | |
|---|---|---|---|---|---|---|---|
| 1. ID.r | 1.00 | -.01 | -.03 | .03 | .00 | .04 | .01 |
| 2. sig | NA | .71 | .38 | .36 | .96 | .20 | .78 |
| 3. n | 1000 | 990 | 990 | 990 | 990 | 970 | 930 |
| 4. predictor.r | -.01 | 1.00 | .60*** | .03 | .03 | -.01 | -.04 |
| 5. sig | .71 | NA | .00 | .40 | .31 | .72 | .25 |
| 6. n | 990 | 990 | 980 | 981 | 980 | 960 | 922 |
| 7. outcome.r | -.03 | .60*** | 1.00 | .02 | .07* | -.02 | -.02 |
| 8. sig | .38 | .00 | NA | .44 | .03 | .56 | .49 |
| 9. n | 990 | 980 | 990 | 980 | 980 | 960 | 920 |
| 10. predictorOverplot.r | .03 | .03 | .02 | 1.00 | .55*** | .02 | .04 |
5.1.3 Xi (ξ)
Xi (ξ) is an index of the degree of dependence between two variables, which is useful as an index of nonlinear correlation.
Chatterjee, S. (2021). A new coefficient of correlation. Journal of the American Statistical Association, 116(536), 2009-2022. https://doi.org/10.1080/01621459.2020.1758115
calculateXI(
mydata$predictor,
mydata$outcome)[1] 0.23190625.2 Scatterplot
5.2.1 Base R
plot(
mydata$predictor,
mydata$outcome)
abline(lm(
outcomeOverplot ~ predictorOverplot,
data = mydata,
na.action = "na.exclude"))
5.2.2
ggplot2
ggplot(mydata, aes(x = predictor, y = outcome)) +
geom_point() +
geom_smooth(method = "lm", se = TRUE)`geom_smooth()` using formula = 'y ~ x'Warning: Removed 20 rows containing non-finite outside the scale range
(`stat_smooth()`).Warning: Removed 20 rows containing missing values or values outside the scale range
(`geom_point()`).
5.3 Scatterplot with Marginal Density Plot
scatterplot <-
ggplot(mydata, aes(x = predictor, y = outcome)) +
geom_point() +
geom_smooth(method = "lm", se = TRUE)densityMarginal <- ggMarginal(
scatterplot,
type = "density",
xparams = list(fill = "gray"),
yparams = list(fill = "gray"))`geom_smooth()` using formula = 'y ~ x'Warning: Removed 20 rows containing non-finite outside the scale range
(`stat_smooth()`).`geom_smooth()` using formula = 'y ~ x'Warning: Removed 20 rows containing non-finite outside the scale range
(`stat_smooth()`).Warning: Removed 20 rows containing missing values or values outside the scale range
(`geom_point()`).print(densityMarginal, newpage = TRUE)
5.4 High Density Scatterplot
ggplot(mydata, aes(x = predictorOverplot, y = outcomeOverplot)) +
geom_point(position = "jitter", alpha = 0.3) +
geom_density2d()Warning: Removed 20 rows containing non-finite outside the scale range
(`stat_density2d()`).Warning: Removed 20 rows containing missing values or values outside the scale range
(`geom_point()`).
smoothScatter(mydata$predictorOverplot, mydata$outcomeOverplot)
5.5 Data Ellipse
mydata_nomissing <- na.omit(mydata[,c("predictor","outcome")])
dataEllipse(mydata_nomissing$predictor, mydata_nomissing$outcome, levels = c(0.5, .95))
5.6 Visually Weighted Regression
vwReg(outcome ~ predictor, data = mydata)
6 Basic inferential statistics
6.1 Tests of Normality
6.1.1 Shapiro-Wilk test of normality
The Shapiro-Wilk test of normality does not accept more than 5000 cases because it will reject the hypothesis that data come from a normal distribution with even slight deviations from normality.
shapiro.test(na.omit(mydata$outcome)) #subset to keep only the first 5000 rows: mydata$outcome[1:5000]
Shapiro-Wilk normality test
data: na.omit(mydata$outcome)
W = 0.9971, p-value = 0.069986.1.2 Test of multivariate normality
mydata %>%
na.omit %>%
t %>%
mshapiro.test
Shapiro-Wilk normality test
data: Z
W = 0.98542, p-value = 1.339e-076.2 Statistical decision tree
https://upload.wikimedia.org/wikipedia/commons/7/74/InferentialStatisticalDecisionMakingTrees.pdf (archived at https://perma.cc/L2QR-ALFA)
6.3 Tests of systematic missingness (i.e., whether missingness on a variable depends on other variables)
- Generally test:
- Whether data are consistent with a missing completely at random (MCAR) pattern—Little’s MCAR Test
- Whether outcome variable(s) differ as a function of any model variables (predictors and covariates) and as a function of any key demographic characteristics (e.g., sex, ethnicity, socioeconomic status)
- Whether focal predictor variable(s) differ as a function of any model variables (including outcome variable) and as a function of any key demographic characteristics
- For instance:
- Whether males are more likely than girls to be missing scores on the dependent variable
- Whether longitudinal attrition is greater in lower socioeconomic status families
- If missingness differs systematically as a function of other variables, you can include that variable as a control variable in models, and/or can include that variable in multiple imputation to inform imputed scores for missing values
6.3.1 Little’s MCAR Test
mcar_test(mydata)statistic <dbl> | df <dbl> | p.value <dbl> | missing.patterns <int> | |
|---|---|---|---|---|
| 59.68147 | 61 | 0.5238153 | 12 |
6.4 Multivariate Associations
6.4.1 Correlation Matrix
6.4.1.1 Pearson Correlations
cor.table(mydata[,c("predictor","outcome","predictorOverplot","outcomeOverplot")])predictor <chr> | outcome <chr> | predictorOverplot <chr> | outcomeOverplot <chr> | |
|---|---|---|---|---|
| 1. predictor.r | 1.00 | .64*** | .04 | .04 |
| 2. sig | NA | .00 | .17 | .18 |
| 3. n | 990 | 980 | 981 | 980 |
| 4. outcome.r | .64*** | 1.00 | .03 | .06† |
| 5. sig | .00 | NA | .35 | .05 |
| 6. n | 980 | 990 | 980 | 980 |
| 7. predictorOverplot.r | .04 | .03 | 1.00 | .57*** |
| 8. sig | .17 | .35 | NA | .00 |
| 9. n | 981 | 980 | 990 | 980 |
| 10. outcomeOverplot.r | .04 | .06† | .57*** | 1.00 |
cor.table(mydata[,c("predictor","outcome","predictorOverplot","outcomeOverplot")], type = "manuscript")predictor <chr> | outcome <chr> | predictorOverplot <chr> | outcomeOverplot <chr> | |
|---|---|---|---|---|
| 1. predictor | 1.00 | |||
| 2. outcome | .64*** | 1.00 | ||
| 3. predictorOverplot | .04 | .03 | 1.00 | |
| 4. outcomeOverplot | .04 | .06† | .57*** | 1.00 |
cor.table(mydata[,c("predictor","outcome","predictorOverplot","outcomeOverplot")], type = "manuscriptBig")predictor <chr> | outcome <chr> | predictorOverplot <chr> | outcomeOverplot <chr> | |
|---|---|---|---|---|
| 1. predictor | 1.00 | |||
| 2. outcome | .64 | 1.00 | ||
| 3. predictorOverplot | .04 | .03 | 1.00 | |
| 4. outcomeOverplot | .04 | .06 | .57 | 1.00 |
6.4.1.2 Spearman Correlations
cor.table(mydata[,c("predictor","outcome","predictorOverplot","outcomeOverplot")], correlation = "spearman")predictor <chr> | outcome <chr> | predictorOverplot <chr> | outcomeOverplot <chr> | |
|---|---|---|---|---|
| 1. predictor.r | 1.00 | .60*** | .03 | .03 |
| 2. sig | NA | .00 | .40 | .31 |
| 3. n | 990 | 980 | 981 | 980 |
| 4. outcome.r | .60*** | 1.00 | .02 | .07* |
| 5. sig | .00 | NA | .44 | .03 |
| 6. n | 980 | 990 | 980 | 980 |
| 7. predictorOverplot.r | .03 | .02 | 1.00 | .55*** |
| 8. sig | .40 | .44 | NA | .00 |
| 9. n | 981 | 980 | 990 | 980 |
| 10. outcomeOverplot.r | .03 | .07* | .55*** | 1.00 |
cor.table(mydata[,c("predictor","outcome","predictorOverplot","outcomeOverplot")], type = "manuscript", correlation = "spearman")predictor <chr> | outcome <chr> | predictorOverplot <chr> | outcomeOverplot <chr> | |
|---|---|---|---|---|
| 1. predictor | 1.00 | |||
| 2. outcome | .60*** | 1.00 | ||
| 3. predictorOverplot | .03 | .02 | 1.00 | |
| 4. outcomeOverplot | .03 | .07* | .55*** | 1.00 |
cor.table(mydata[,c("predictor","outcome","predictorOverplot","outcomeOverplot")], type = "manuscriptBig", correlation = "spearman")predictor <chr> | outcome <chr> | predictorOverplot <chr> | outcomeOverplot <chr> | |
|---|---|---|---|---|
| 1. predictor | 1.00 | |||
| 2. outcome | .60 | 1.00 | ||
| 3. predictorOverplot | .03 | .02 | 1.00 | |
| 4. outcomeOverplot | .03 | .07 | .55 | 1.00 |
6.4.1.3 Partial Correlations
Examine the associations among variables controlling for a covariate
(outcomeOverplot).
partialcor.table(mydata[,c("predictor","outcome","predictorOverplot")], z = mydata[,c("outcomeOverplot")])predictor <chr> | outcome <chr> | predictorOverplot <chr> | ||
|---|---|---|---|---|
| 1. predictor.r | 1.00 | .63*** | .02 | |
| 2. sig | NA | .00 | .47 | |
| 3. n | 980 | 970 | 971 | |
| 4. outcome.r | .63*** | 1.00 | -.01 | |
| 5. sig | .00 | NA | .83 | |
| 6. n | 970 | 980 | 970 | |
| 7. predictorOverplot.r | .02 | -.01 | 1.00 | |
| 8. sig | .47 | .83 | NA | |
| 9. n | 971 | 970 | 980 |
partialcor.table(mydata[,c("predictor","outcome","predictorOverplot")], z = mydata[,c("outcomeOverplot")], type = "manuscript")predictor <chr> | outcome <chr> | predictorOverplot <chr> | ||
|---|---|---|---|---|
| 1. predictor | 1.00 | |||
| 2. outcome | .63*** | 1.00 | ||
| 3. predictorOverplot | .02 | -.01 | 1.00 |
partialcor.table(mydata[,c("predictor","outcome","predictorOverplot")], z = mydata[,c("outcomeOverplot")], type = "manuscriptBig")predictor <chr> | outcome <chr> | predictorOverplot <chr> | ||
|---|---|---|---|---|
| 1. predictor | 1.00 | |||
| 2. outcome | .63 | 1.00 | ||
| 3. predictorOverplot | .02 | -.01 | 1.00 |
6.4.2 Correlogram
corrplot(cor(mydata[,c("predictor","outcome","predictorOverplot","outcomeOverplot")], use = "pairwise.complete.obs"))
6.4.3 Scatterplot matrix
scatterplotMatrix(~ predictor + outcome + predictorOverplot + outcomeOverplot, data = mydata, use = "pairwise.complete.obs")
6.4.4 Pairs panels
pairs.panels(mydata[,c("predictor","outcome","predictorOverplot","outcomeOverplot")])
6.5 Effect Plots
6.5.1 Multiple Regression Model
multipleRegressionModel <- lm(outcome ~ predictor + predictorOverplot,
data = mydata,
na.action = "na.exclude")
allEffects(multipleRegressionModel) model: outcome ~ predictor + predictorOverplot
predictor effect
predictor
0.1 20 40 60 80
49.23335 70.37778 91.62847 112.87915 134.12983
predictorOverplot effect
predictorOverplot
1 10 30 40 50
73.13937 73.15070 73.17586 73.18845 73.20103 plot(allEffects(multipleRegressionModel))
6.5.2 Multilevel Regression Model
multilevelRegressionModel <- lme(outcome ~ predictor + predictorOverplot, random = ~ 1|ID,
method = "ML",
data = mydata,
na.action = "na.exclude")
allEffects(multilevelRegressionModel) model: outcome ~ predictor + predictorOverplot
predictor effect
predictor
0.1 20 40 60 80
49.23335 70.37778 91.62847 112.87915 134.12983
predictorOverplot effect
predictorOverplot
1 10 30 40 50
73.13937 73.15070 73.17586 73.18845 73.20103 plot(allEffects(multilevelRegressionModel))
7 Session Info
sessionInfo()R version 4.5.0 (2025-04-11)
Platform: x86_64-pc-linux-gnu
Running under: Ubuntu 24.04.2 LTS
Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.26.so; LAPACK version 3.12.0
locale:
[1] LC_CTYPE=C.UTF-8 LC_NUMERIC=C LC_TIME=C.UTF-8
[4] LC_COLLATE=C.UTF-8 LC_MONETARY=C.UTF-8 LC_MESSAGES=C.UTF-8
[7] LC_PAPER=C.UTF-8 LC_NAME=C LC_ADDRESS=C
[10] LC_TELEPHONE=C LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C
time zone: UTC
tzcode source: system (glibc)
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] XICOR_0.4.1 ggExtra_0.10.1 mvnormtest_0.1-9-3 naniar_1.1.0
[5] lubridate_1.9.4 forcats_1.0.0 stringr_1.5.1 dplyr_1.1.4
[9] purrr_1.0.4 readr_2.1.5 tidyr_1.3.1 tibble_3.2.1
[13] tidyverse_2.0.0 psych_2.5.3 ggplot2_3.5.2 corrplot_0.95
[17] effects_4.2-2 nlme_3.1-168 ellipse_0.5.0 vioplot_0.5.1
[21] zoo_1.8-14 sm_2.2-6.0 car_3.1-3 carData_3.0-5
[25] petersenlab_1.1.5
loaded via a namespace (and not attached):
[1] Rdpack_2.6.4 DBI_1.2.3 mnormt_2.1.1 gridExtra_2.3
[5] rlang_1.1.6 magrittr_2.0.3 compiler_4.5.0 mgcv_1.9-1
[9] vctrs_0.6.5 reshape2_1.4.4 quadprog_1.5-8 pkgconfig_2.0.3
[13] fastmap_1.2.0 backports_1.5.0 labeling_0.4.3 pbivnorm_0.6.0
[17] promises_1.3.2 rmarkdown_2.29 psychTools_2.5.3 tzdb_0.5.0
[21] nloptr_2.2.1 visdat_0.6.0 xfun_0.52 cachem_1.1.0
[25] jsonlite_2.0.0 later_1.4.2 parallel_4.5.0 lavaan_0.6-19
[29] cluster_2.1.8.1 R6_2.6.1 bslib_0.9.0 stringi_1.8.7
[33] RColorBrewer_1.1-3 boot_1.3-31 rpart_4.1.24 estimability_1.5.1
[37] jquerylib_0.1.4 Rcpp_1.0.14 knitr_1.50 base64enc_0.1-3
[41] httpuv_1.6.16 Matrix_1.7-3 splines_4.5.0 nnet_7.3-20
[45] timechange_0.3.0 tidyselect_1.2.1 rstudioapi_0.17.1 abind_1.4-8
[49] yaml_2.3.10 miniUI_0.1.2 lattice_0.22-6 plyr_1.8.9
[53] shiny_1.10.0 withr_3.0.2 evaluate_1.0.3 foreign_0.8-90
[57] survival_3.8-3 isoband_0.2.7 survey_4.4-2 norm_1.0-11.1
[61] pillar_1.10.2 KernSmooth_2.23-26 rtf_0.4-14.1 checkmate_2.3.2
[65] stats4_4.5.0 reformulas_0.4.1 insight_1.2.0 generics_0.1.4
[69] mix_1.0-13 hms_1.1.3 scales_1.4.0 minqa_1.2.8
[73] xtable_1.8-4 glue_1.8.0 Hmisc_5.2-3 tools_4.5.0
[77] data.table_1.17.2 lme4_1.1-37 mvtnorm_1.3-3 grid_4.5.0
[81] mitools_2.4 rbibutils_2.3 colorspace_2.1-1 htmlTable_2.4.3
[85] Formula_1.2-5 cli_3.6.5 viridisLite_0.4.2 gtable_0.3.6
[89] R.methodsS3_1.8.2 sass_0.4.10 digest_0.6.37 htmlwidgets_1.6.4
[93] farver_2.1.2 R.oo_1.27.1 htmltools_0.5.8.1 lifecycle_1.0.4
[97] mime_0.13 MASS_7.3-65 LS0tCnRpdGxlOiAiRXhwbG9yYXRvcnkgRGF0YSBBbmFseXNpcyIKLS0tCgpgYGB7ciBzZXR1cCwgaW5jbHVkZSA9IEZBTFNFfQprbml0cjo6b3B0c19jaHVuayRzZXQoCiAgZWNobyA9IFRSVUUsCiAgZXJyb3IgPSBUUlVFLAogIGNvbW1lbnQgPSAiIikKYGBgCgojIFJhdGlvbmFsZQoKRXhwbG9yYXRvcnkgZGF0YSBhbmFseXNpcyBpcyBpbXBvcnRhbnQgZm9yIHVuZGVyc3RhbmRpbmcgeW91ciBkYXRhLCBjaGVja2luZyBmb3IgZGF0YSBpc3N1ZXMvZXJyb3JzLCBhbmQgY2hlY2tpbmcgYXNzdW1wdGlvbnMgZm9yIGRpZmZlcmVudCBzdGF0aXN0aWNhbCBtb2RlbHMuCgpMT09LIEFUIFlPVVIgREFUQeKAlHRoaXMgaXMgb25lIG9mIHRoZSBtb3N0IG92ZXJsb29rZWQgc3RlcHMgaW4gZGF0YSBhbmFseXNpcyEKCiMgUHJlYW1ibGUKCiMjIEluc3RhbGwgTGlicmFyaWVzCgpgYGB7cn0KI2luc3RhbGwucGFja2FnZXMoInJlbW90ZXMiKQojcmVtb3Rlczo6aW5zdGFsbF9naXRodWIoIkRldlBzeUxhYi9wZXRlcnNlbmxhYiIpCmBgYAoKIyMgTG9hZCBMaWJyYXJpZXMKCmBgYHtyLCBtZXNzYWdlID0gRkFMU0UsIHdhcm5pbmcgPSBGQUxTRX0KbGlicmFyeSgicGV0ZXJzZW5sYWIiKQpsaWJyYXJ5KCJjYXIiKQpsaWJyYXJ5KCJ2aW9wbG90IikKbGlicmFyeSgiZWxsaXBzZSIpCmxpYnJhcnkoIm5sbWUiKQpsaWJyYXJ5KCJlZmZlY3RzIikKbGlicmFyeSgiY29ycnBsb3QiKQpsaWJyYXJ5KCJnZ3Bsb3QyIikKbGlicmFyeSgicHN5Y2giKQpsaWJyYXJ5KCJ0aWR5dmVyc2UiKQpsaWJyYXJ5KCJwdXJyciIpCmxpYnJhcnkoIm5hbmlhciIpCmxpYnJhcnkoIm12bm9ybXRlc3QiKQpsaWJyYXJ5KCJnZ0V4dHJhIikKbGlicmFyeSgiWElDT1IiKQpgYGAKCiMgU2ltdWxhdGUgRGF0YQoKYGBge3J9CnNldC5zZWVkKDUyMjQyKQoKbiA8LSAxMDAwCgpJRCA8LSByZXAoMToxMDAsIGVhY2ggPSAxMCkKcHJlZGljdG9yIDwtIHJiZXRhKG4sIDEuNSwgNSkgKiAxMDAKb3V0Y29tZSA8LSBwcmVkaWN0b3IgKyBybm9ybShuLCBtZWFuID0gMCwgc2QgPSAyMCkgKyA1MAoKcHJlZGljdG9yT3ZlcnBsb3QgPC0gc2FtcGxlKDE6NTAsIG4sIHJlcGxhY2UgPSBUUlVFKQpvdXRjb21lT3ZlcnBsb3QgPC0gcHJlZGljdG9yT3ZlcnBsb3QgKyBzYW1wbGUoMTo3NSwgbiwgcmVwbGFjZSA9IFRSVUUpCgpjYXRlZ29yaWNhbDEgPC0gc2FtcGxlKDE6NSwgc2l6ZSA9IG4sIHJlcGxhY2UgPSBUUlVFKQpjYXRlZ29yaWNhbDIgPC0gc2FtcGxlKDE6NSwgc2l6ZSA9IG4sIHJlcGxhY2UgPSBUUlVFKQoKbXlkYXRhIDwtIGRhdGEuZnJhbWUoCiAgSUQgPSBJRCwKICBwcmVkaWN0b3IgPSBwcmVkaWN0b3IsCiAgb3V0Y29tZSA9IG91dGNvbWUsCiAgcHJlZGljdG9yT3ZlcnBsb3QgPSBwcmVkaWN0b3JPdmVycGxvdCwKICBvdXRjb21lT3ZlcnBsb3QgPSBvdXRjb21lT3ZlcnBsb3QsCiAgY2F0ZWdvcmljYWwxID0gY2F0ZWdvcmljYWwxLAogIGNhdGVnb3JpY2FsMiA9IGNhdGVnb3JpY2FsMikKCm15ZGF0YVtzYW1wbGUoMTpuLCBzaXplID0gMTApLCAicHJlZGljdG9yIl0gPC0gTkEKbXlkYXRhW3NhbXBsZSgxOm4sIHNpemUgPSAxMCksICJvdXRjb21lIl0gPC0gTkEKbXlkYXRhW3NhbXBsZSgxOm4sIHNpemUgPSAxMCksICJwcmVkaWN0b3JPdmVycGxvdCJdIDwtIE5BCm15ZGF0YVtzYW1wbGUoMTpuLCBzaXplID0gMTApLCAib3V0Y29tZU92ZXJwbG90Il0gPC0gTkEKbXlkYXRhW3NhbXBsZSgxOm4sIHNpemUgPSAzMCksICJjYXRlZ29yaWNhbDEiXSA8LSBOQQpteWRhdGFbc2FtcGxlKDE6biwgc2l6ZSA9IDcwKSwgImNhdGVnb3JpY2FsMiJdIDwtIE5BCmBgYAoKIyBEZXNjcmlwdGl2ZSBzdGF0aXN0aWNzCgpgYGB7cn0Kcm91bmQoZGF0YS5mcmFtZShwc3ljaDo6ZGVzY3JpYmUobXlkYXRhKSksIDIpCmBgYAoKIyMgU2FtcGxlCgotIENoZWNrIHRoZSBzYW1wbGUgc2l6ZSAoKk4qKQogIC0gSXMgdGhlIHNhbXBsZSBzaXplIGluIHRoZSBkYXRhIHRoZSBleHBlY3RlZCBzYW1wbGUgc2l6ZT8KICBBcmUgdGhlcmUgY2FzZXMgKHBhcnRpY2lwYW50cykgdGhhdCBhcmUgbWlzc2luZz8KICBBcmUgdGhlcmUgY2FzZXMgdGhhdCBzaG91bGQgbm90IGJlIHRoZXJlPwogIC0gSGVyZSBpcyB0aGUgc2FtcGxlIHNpemU6CiAgCmBgYHtyfQpsZW5ndGgodW5pcXVlKG15ZGF0YSRJRCkpCmBgYCAgIAoKLSBDaGVjayB0aGUgZXh0ZW50IG9mIG1pc3NpbmduZXNzCiAgICAtIEhvdyBtdWNoIGRhdGEgYXJlIG1pc3NpbmcgaW4gdGhlIG1vZGVsIHZhcmlhYmxlc+KAlGluY2x1ZGluZyB0aGUgcHJlZGljdG9yLCBvdXRjb21lLCBhbmQgY292YXJpYXRlcz8KICAgIC0gSGVyZSBhcmUgdGhlIHByb3BvcnRpb24gb2YgbWlzc2luZyBkYXRhIGluIGVhY2ggdmFyaWFibGU6CiAgICAKYGBge3J9Cm1hcChteWRhdGEsIH5tZWFuKGlzLm5hKC4pKSkgJT4lIHQgJT4lIHQKYGBgCgojIyBEaXN0cmlidXRpb24KCi0gRnJlcXVlbmNpZXMKICAgIC0gRXhhbWluZSB0aGUgZnJlcXVlbmNpZXMgb2YgY2F0ZWdvcmljYWwgdmFyaWFibGVzOgoKYGBge3J9Cm15ZGF0YSAlPiUgCiAgc2VsZWN0KGNhdGVnb3JpY2FsMSwgY2F0ZWdvcmljYWwyKSAlPiUKICBzYXBwbHkoZnVuY3Rpb24oeCkgdGFibGUoeCwgdXNlTkEgPSAiYWx3YXlzIikpICU+JSAKICB0KCkKYGBgCgojIyBDZW50cmFsIFRlbmRlbmN5CgotIE1lYW4KCmBgYHtyfQpyb3VuZChjb2xNZWFucyhteWRhdGEsIG5hLnJtID0gVFJVRSksIDIpCnJvdW5kKGFwcGx5KG15ZGF0YSwgMiwgZnVuY3Rpb24oeCkgbWVhbih4LCBuYS5ybSA9IFRSVUUpKSwgMikKCm15ZGF0YSAlPiUgCiAgc3VtbWFyaXNlKGFjcm9zcyhldmVyeXRoaW5nKCksCiAgICAgICAgICAgICAgICAgICAuZm5zID0gbGlzdChtZWFuID0gfiBtZWFuKC4sIG5hLnJtID0gVFJVRSkpKSkgJT4lIAogIHJvdW5kKC4sIDIpCmBgYAoKLSBNZWRpYW4KCmBgYHtyfQpyb3VuZChhcHBseShteWRhdGEsIDIsIGZ1bmN0aW9uKHgpIG1lZGlhbih4LCBuYS5ybSA9IFRSVUUpKSwgMikKCm15ZGF0YSAlPiUgCiAgc3VtbWFyaXNlKGFjcm9zcyhldmVyeXRoaW5nKCksCiAgICAgICAgICAgICAgICAgICAuZm5zID0gbGlzdChtZWRpYW4gPSB+IG1lZGlhbiguLCBuYS5ybSA9IFRSVUUpKSkpICU+JSAKICByb3VuZCguLCAyKQpgYGAKCi0gTW9kZQoKYGBge3J9CnJvdW5kKGFwcGx5KG15ZGF0YSwgMiwgZnVuY3Rpb24oeCkgTW9kZSh4LCBtdWx0aXBsZU1vZGVzID0gIm1lYW4iKSksIDIpCgpteWRhdGEgJT4lIAogIHN1bW1hcmlzZShhY3Jvc3MoZXZlcnl0aGluZygpLAogICAgICAgICAgICAgICAgICAgLmZucyA9IGxpc3QobW9kZSA9IH4gTW9kZSguLCBtdWx0aXBsZU1vZGVzID0gIm1lYW4iKSkpKSAlPiUgCiAgcm91bmQoLiwgMikKYGBgCgpDb21wdXRlIGFsbCBvZiB0aGVzZSBtZWFzdXJlcyBvZiBjZW50cmFsIHRlbmRlbmN5OgoKYGBge3J9Cm15ZGF0YSAlPiUgCiAgc3VtbWFyaXNlKGFjcm9zcyhldmVyeXRoaW5nKCksCiAgICAgICAgICAgICAgICAgICAuZm5zID0gbGlzdChtZWFuID0gfiBtZWFuKC4sIG5hLnJtID0gVFJVRSksCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBtZWRpYW4gPSB+IG1lZGlhbiguLCBuYS5ybSA9IFRSVUUpLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbW9kZSA9IH4gTW9kZSguLCBtdWx0aXBsZU1vZGVzID0gIm1lYW4iKSksCiAgICAgICAgICAgICAgICAgICAubmFtZXMgPSAiey5jb2x9LnsuZm59IikpICU+JSAKICByb3VuZCguLCAyKSAlPiUgCiAgcGl2b3RfbG9uZ2VyKGNvbHMgPSBldmVyeXRoaW5nKCksCiAgICAgICAgICAgICAgIG5hbWVzX3RvID0gYygidmFyaWFibGUiLCJpbmRleCIpLAogICAgICAgICAgICAgICBuYW1lc19zZXAgPSAiXFwuIikgJT4lIAogIHBpdm90X3dpZGVyKG5hbWVzX2Zyb20gPSBpbmRleCwKICAgICAgICAgICAgICB2YWx1ZXNfZnJvbSA9IHZhbHVlKQpgYGAKCiMjIERpc3BlcnNpb24KCi0gU3RhbmRhcmQgZGV2aWF0aW9uCi0gT2JzZXJ2ZWQgbWluaW11bSBhbmQgbWF4aW11bSAodmlzLcOgLXZpcyBwb3NzaWJsZSBtaW5pbXVtIGFuZCBtYXhpbXVtKQotIFNrZXduZXNzCi0gS3VydG9zaXMKCkNvbXB1dGUgYWxsIG9mIHRoZXNlIG1lYXN1cmVzIG9mIGRpc3BlcnNpb246CgpgYGB7cn0KbXlkYXRhICU+JSAKICBzdW1tYXJpc2UoYWNyb3NzKGV2ZXJ5dGhpbmcoKSwKICAgICAgICAgICAgICAgICAgIC5mbnMgPSBsaXN0KFNEID0gfiBzZCguLCBuYS5ybSA9IFRSVUUpLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbWluID0gfiBtaW4oLiwgbmEucm0gPSBUUlVFKSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIG1heCA9IH4gbWF4KC4sIG5hLnJtID0gVFJVRSksCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBza2V3bmVzcyA9IH4gc2tldyguLCBuYS5ybSA9IFRSVUUpLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAga3VydG9zaXMgPSB+IGt1cnRvc2koLiwgbmEucm0gPSBUUlVFKSksCiAgICAgICAgICAgICAgICAgICAubmFtZXMgPSAiey5jb2x9LnsuZm59IikpICU+JSAKICByb3VuZCguLCAyKSAlPiUgCiAgcGl2b3RfbG9uZ2VyKGNvbHMgPSBldmVyeXRoaW5nKCksCiAgICAgICAgICAgICAgIG5hbWVzX3RvID0gYygidmFyaWFibGUiLCJpbmRleCIpLAogICAgICAgICAgICAgICBuYW1lc19zZXAgPSAiXFwuIikgJT4lIAogIHBpdm90X3dpZGVyKG5hbWVzX2Zyb20gPSBpbmRleCwKICAgICAgICAgICAgICB2YWx1ZXNfZnJvbSA9IHZhbHVlKQpgYGAKCkNvbnNpZGVyIHRyYW5zZm9ybWluZyBkYXRhIGlmIHNrZXduZXNzID4gfDAuOHwgb3IgaWYga3VydG9zaXMgPiB8My4wfC4KCiMjIFN1bW1hcnkgU3RhdGlzdGljcyB7I3N1bW1hcnlTdGF0c30KCkFkZCBzdW1tYXJ5IHN0YXRpc3RpY3MgdG8gdGhlIGJvdHRvbSBvZiBjb3JyZWxhdGlvbiBtYXRyaWNlcyBpbiBwYXBlcnM6CgpgYGB7cn0KY29yLnRhYmxlKG15ZGF0YSwgdHlwZSA9ICJtYW51c2NyaXB0IikKCnN1bW1hcnlUYWJsZSA8LSBteWRhdGEgJT4lIAogIHN1bW1hcmlzZShhY3Jvc3MoZXZlcnl0aGluZygpLAogICAgICAgICAgICAgICAgICAgLmZucyA9IGxpc3QobiA9IH4gbGVuZ3RoKG5hLm9taXQoLikpLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbWlzc2luZ25lc3MgPSB+IG1lYW4oaXMubmEoLikpICogMTAwLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgTSA9IH4gbWVhbiguLCBuYS5ybSA9IFRSVUUpLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgU0QgPSB+IHNkKC4sIG5hLnJtID0gVFJVRSksCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBtaW4gPSB+IG1pbiguLCBuYS5ybSA9IFRSVUUpLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbWF4ID0gfiBtYXgoLiwgbmEucm0gPSBUUlVFKSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHNrZXduZXNzID0gfiBza2V3KC4sIG5hLnJtID0gVFJVRSksCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBrdXJ0b3NpcyA9IH4ga3VydG9zaSguLCBuYS5ybSA9IFRSVUUpKSwKICAgICAgICAgICAgICAgICAgIC5uYW1lcyA9ICJ7LmNvbH0uey5mbn0iKSkgJT4lICAKICBwaXZvdF9sb25nZXIoY29scyA9IGV2ZXJ5dGhpbmcoKSwKICAgICAgICAgICAgICAgbmFtZXNfdG8gPSBjKCJ2YXJpYWJsZSIsImluZGV4IiksCiAgICAgICAgICAgICAgIG5hbWVzX3NlcCA9ICJcXC4iKSAlPiUgCiAgcGl2b3Rfd2lkZXIobmFtZXNfZnJvbSA9IGluZGV4LAogICAgICAgICAgICAgIHZhbHVlc19mcm9tID0gdmFsdWUpCgpzdW1tYXJ5VGFibGVUcmFuc3Bvc2VkIDwtIHN1bW1hcnlUYWJsZVstMV0gJT4lIAogIHQoKSAlPiUgCiAgYXMuZGF0YS5mcmFtZSgpICU+JSAKICBzZXROYW1lcyhzdW1tYXJ5VGFibGUkdmFyaWFibGUpICU+JSAKICByb3VuZCguLCBkaWdpdHMgPSAyKQoKc3VtbWFyeVRhYmxlVHJhbnNwb3NlZApgYGAKCiMjIERpc3RyaWJ1dGlvbiBQbG90cwoKU2VlIFtoZXJlXShmaWd1cmVzLmh0bWwpIGZvciByZXNvdXJjZXMgZm9yIGNyZWF0aW5nIGZpZ3VyZXMgaW4gUi4KCiMjIyBIaXN0b2dyYW0KCiMjIyMgQmFzZSBSCgpgYGB7cn0KaGlzdChteWRhdGEkb3V0Y29tZSkKYGBgCgojIyMjIGBnZ3Bsb3QyYAoKYGBge3J9CmdncGxvdChteWRhdGEsIGFlcyh4ID0gb3V0Y29tZSkpICsKICBnZW9tX2hpc3RvZ3JhbShjb2xvciA9IDEpCmBgYAoKIyMjIEhpc3RvZ3JhbSBvdmVybGFpZCB3aXRoIGRlbnNpdHkgcGxvdCBhbmQgcnVnIHBsb3QKCiMjIyMgQmFzZSBSCgpgYGB7cn0KaGlzdChteWRhdGEkb3V0Y29tZSwgcHJvYiA9IFRSVUUpCmxpbmVzKGRlbnNpdHkobXlkYXRhJG91dGNvbWUsIG5hLnJtID0gVFJVRSkpCnJ1ZyhteWRhdGEkb3V0Y29tZSkKYGBgCgojIyMjIGBnZ3Bsb3QyYAoKYGBge3J9CmdncGxvdChteWRhdGEsIGFlcyh4ID0gb3V0Y29tZSkpICsKICBnZW9tX2hpc3RvZ3JhbShhZXMoeSA9IGFmdGVyX3N0YXQoZGVuc2l0eSkpLCBjb2xvciA9IDEpICsKICBnZW9tX2RlbnNpdHkoKSArCiAgZ2VvbV9ydWcoKQpgYGAKCiMjIyBEZW5zaXR5IFBsb3QKCiMjIyMgQmFzZSBSCgpgYGB7cn0KcGxvdChkZW5zaXR5KG15ZGF0YSRvdXRjb21lLCBuYS5ybSA9IFRSVUUpKQpgYGAKCiMjIyMgYGdncGxvdDJgCgpgYGB7cn0KZ2dwbG90KG15ZGF0YSwgYWVzKHggPSBvdXRjb21lKSkgKwogIGdlb21fZGVuc2l0eSgpCmBgYAoKIyMjIEJveCBhbmQgd2hpc2tlciBwbG90IChib3hwbG90KQoKIyMjIyBCYXNlIFIKCmBgYHtyfQpib3hwbG90KG15ZGF0YSRvdXRjb21lLCBob3Jpem9udGFsID0gVFJVRSkKYGBgCgojIyMjIGBnZ3Bsb3QyYAoKYGBge3J9CmdncGxvdChteWRhdGEsIGFlcyh4ID0gb3V0Y29tZSkpICsKICBnZW9tX2JveHBsb3QoKQpgYGAKCiMjIyBWaW9saW4gcGxvdAoKIyMjIyBCYXNlIFIKCmBgYHtyfQp2aW9wbG90KG5hLm9taXQobXlkYXRhJG91dGNvbWUpLCBob3Jpem9udGFsID0gVFJVRSkKYGBgCgojIyMjIGBnZ3Bsb3QyYAoKYGBge3J9CmdncGxvdChteWRhdGEsIGFlcyh4ID0gIiIsIHkgPSBvdXRjb21lKSkgKwogIGdlb21fdmlvbGluKCkKYGBgCgojIEJpdmFyaWF0ZSBBc3NvY2lhdGlvbnMKCkZvciBtb3JlIGFkdmFuY2VkIHNjYXR0ZXJwbG90cywgc2VlIFtoZXJlXShmaWd1cmVzLmh0bWwjbWFyZ2luYWxEaXN0cmlidXRpb25zKS4KCiMjIENvcnJlbGF0aW9uIENvZWZmaWNpZW50cwoKIyMjIFBlYXJzb24gQ29ycmVsYXRpb24KCmBgYHtyfQpjb3IobXlkYXRhLCB1c2UgPSAicGFpcndpc2UuY29tcGxldGUub2JzIikKY29yLnRlc3QoIH4gcHJlZGljdG9yICsgb3V0Y29tZSwgZGF0YSA9IG15ZGF0YSkKY29yLnRhYmxlKG15ZGF0YSkKYGBgCgojIyMgU3BlYXJtYW4gQ29ycmVsYXRpb24KCmBgYHtyfQpjb3IobXlkYXRhLCB1c2UgPSAicGFpcndpc2UuY29tcGxldGUub2JzIiwgbWV0aG9kID0gInNwZWFybWFuIikKY29yLnRlc3QoIH4gcHJlZGljdG9yICsgb3V0Y29tZSwgZGF0YSA9IG15ZGF0YSwgbWV0aG9kID0gInNwZWFybWFuIikKY29yLnRhYmxlKG15ZGF0YSwgY29ycmVsYXRpb24gPSAic3BlYXJtYW4iKQpgYGAKCiMjIyBYaSAoJFx4aSQpCgpYaSAoJFx4aSQpIGlzIGFuIGluZGV4IG9mIHRoZSBkZWdyZWUgb2YgZGVwZW5kZW5jZSBiZXR3ZWVuIHR3byB2YXJpYWJsZXMsIHdoaWNoIGlzIHVzZWZ1bCBhcyBhbiBpbmRleCBvZiBub25saW5lYXIgY29ycmVsYXRpb24uCgpDaGF0dGVyamVlLCBTLiAoMjAyMSkuIEEgbmV3IGNvZWZmaWNpZW50IG9mIGNvcnJlbGF0aW9uLiAqSm91cm5hbCBvZiB0aGUgQW1lcmljYW4gU3RhdGlzdGljYWwgQXNzb2NpYXRpb24sIDExNiooNTM2KSwgMjAwOS0yMDIyLiBodHRwczovL2RvaS5vcmcvMTAuMTA4MC8wMTYyMTQ1OS4yMDIwLjE3NTgxMTUKCmBgYHtyfQpjYWxjdWxhdGVYSSgKICBteWRhdGEkcHJlZGljdG9yLAogIG15ZGF0YSRvdXRjb21lKQpgYGAKCiMjIFNjYXR0ZXJwbG90CgojIyMgQmFzZSBSCgpgYGB7cn0KcGxvdCgKICBteWRhdGEkcHJlZGljdG9yLAogIG15ZGF0YSRvdXRjb21lKQphYmxpbmUobG0oCiAgb3V0Y29tZU92ZXJwbG90IH4gcHJlZGljdG9yT3ZlcnBsb3QsCiAgZGF0YSA9IG15ZGF0YSwKICBuYS5hY3Rpb24gPSAibmEuZXhjbHVkZSIpKQpgYGAKCiMjIyBgZ2dwbG90MmAKCmBgYHtyfQpnZ3Bsb3QobXlkYXRhLCBhZXMoeCA9IHByZWRpY3RvciwgeSA9IG91dGNvbWUpKSArIAogIGdlb21fcG9pbnQoKSArCiAgZ2VvbV9zbW9vdGgobWV0aG9kID0gImxtIiwgc2UgPSBUUlVFKQpgYGAKCiMjIFNjYXR0ZXJwbG90IHdpdGggTWFyZ2luYWwgRGVuc2l0eSBQbG90CgpgYGB7cn0Kc2NhdHRlcnBsb3QgPC0gCiAgZ2dwbG90KG15ZGF0YSwgYWVzKHggPSBwcmVkaWN0b3IsIHkgPSBvdXRjb21lKSkgKyAKICBnZW9tX3BvaW50KCkgKwogIGdlb21fc21vb3RoKG1ldGhvZCA9ICJsbSIsIHNlID0gVFJVRSkKYGBgCgpgYGB7cn0KZGVuc2l0eU1hcmdpbmFsIDwtIGdnTWFyZ2luYWwoCiAgc2NhdHRlcnBsb3QsCiAgdHlwZSA9ICJkZW5zaXR5IiwKICB4cGFyYW1zID0gbGlzdChmaWxsID0gImdyYXkiKSwKICB5cGFyYW1zID0gbGlzdChmaWxsID0gImdyYXkiKSkKYGBgCgpgYGB7cn0KcHJpbnQoZGVuc2l0eU1hcmdpbmFsLCBuZXdwYWdlID0gVFJVRSkKYGBgCgojIyBIaWdoIERlbnNpdHkgU2NhdHRlcnBsb3QKCmBgYHtyfQpnZ3Bsb3QobXlkYXRhLCBhZXMoeCA9IHByZWRpY3Rvck92ZXJwbG90LCB5ID0gb3V0Y29tZU92ZXJwbG90KSkgKyAKICBnZW9tX3BvaW50KHBvc2l0aW9uID0gImppdHRlciIsIGFscGhhID0gMC4zKSArIAogIGdlb21fZGVuc2l0eTJkKCkKCnNtb290aFNjYXR0ZXIobXlkYXRhJHByZWRpY3Rvck92ZXJwbG90LCBteWRhdGEkb3V0Y29tZU92ZXJwbG90KQpgYGAKCiMjIERhdGEgRWxsaXBzZQoKYGBge3J9Cm15ZGF0YV9ub21pc3NpbmcgPC0gbmEub21pdChteWRhdGFbLGMoInByZWRpY3RvciIsIm91dGNvbWUiKV0pCmRhdGFFbGxpcHNlKG15ZGF0YV9ub21pc3NpbmckcHJlZGljdG9yLCBteWRhdGFfbm9taXNzaW5nJG91dGNvbWUsIGxldmVscyA9IGMoMC41LCAuOTUpKQpgYGAKCiMjIFZpc3VhbGx5IFdlaWdodGVkIFJlZ3Jlc3Npb24KCmBgYHtyLCBtZXNzYWdlID0gRkFMU0UsIHJlc3VsdHMgPSAiaGlkZSJ9CnZ3UmVnKG91dGNvbWUgfiBwcmVkaWN0b3IsIGRhdGEgPSBteWRhdGEpCmBgYAoKIyBCYXNpYyBpbmZlcmVudGlhbCBzdGF0aXN0aWNzCgojIyBUZXN0cyBvZiBOb3JtYWxpdHkKCiMjIyBTaGFwaXJvLVdpbGsgdGVzdCBvZiBub3JtYWxpdHkKClRoZSBTaGFwaXJvLVdpbGsgdGVzdCBvZiBub3JtYWxpdHkgZG9lcyBub3QgYWNjZXB0IG1vcmUgdGhhbiA1MDAwIGNhc2VzIGJlY2F1c2UgaXQgd2lsbCByZWplY3QgdGhlIGh5cG90aGVzaXMgdGhhdCBkYXRhIGNvbWUgZnJvbSBhIG5vcm1hbCBkaXN0cmlidXRpb24gd2l0aCBldmVuIHNsaWdodCBkZXZpYXRpb25zIGZyb20gbm9ybWFsaXR5LgoKYGBge3J9CnNoYXBpcm8udGVzdChuYS5vbWl0KG15ZGF0YSRvdXRjb21lKSkgI3N1YnNldCB0byBrZWVwIG9ubHkgdGhlIGZpcnN0IDUwMDAgcm93czogbXlkYXRhJG91dGNvbWVbMTo1MDAwXQpgYGAKCiMjIyBUZXN0IG9mIG11bHRpdmFyaWF0ZSBub3JtYWxpdHkKCmBgYHtyfQpteWRhdGEgJT4lIAogIG5hLm9taXQgJT4lIAogIHQgJT4lIAogIG1zaGFwaXJvLnRlc3QKYGBgCgojIyBTdGF0aXN0aWNhbCBkZWNpc2lvbiB0cmVlCgpodHRwczovL3VwbG9hZC53aWtpbWVkaWEub3JnL3dpa2lwZWRpYS9jb21tb25zLzcvNzQvSW5mZXJlbnRpYWxTdGF0aXN0aWNhbERlY2lzaW9uTWFraW5nVHJlZXMucGRmIChhcmNoaXZlZCBhdCBodHRwczovL3Blcm1hLmNjL0wyUVItQUxGQSkKCiMjIFRlc3RzIG9mIHN5c3RlbWF0aWMgbWlzc2luZ25lc3MgKGkuZS4sIHdoZXRoZXIgbWlzc2luZ25lc3Mgb24gYSB2YXJpYWJsZSBkZXBlbmRzIG9uIG90aGVyIHZhcmlhYmxlcykKCi0gR2VuZXJhbGx5IHRlc3Q6CiAgICAtIFdoZXRoZXIgZGF0YSBhcmUgY29uc2lzdGVudCB3aXRoIGEgbWlzc2luZyBjb21wbGV0ZWx5IGF0IHJhbmRvbSAoTUNBUikgcGF0dGVybuKAlExpdHRsZSdzIE1DQVIgVGVzdAogICAgLSBXaGV0aGVyIG91dGNvbWUgdmFyaWFibGUocykgZGlmZmVyIGFzIGEgZnVuY3Rpb24gb2YgYW55IG1vZGVsIHZhcmlhYmxlcyAocHJlZGljdG9ycyBhbmQgY292YXJpYXRlcykgYW5kIGFzIGEgZnVuY3Rpb24gb2YgYW55IGtleSBkZW1vZ3JhcGhpYyBjaGFyYWN0ZXJpc3RpY3MgKGUuZy4sIHNleCwgZXRobmljaXR5LCBzb2Npb2Vjb25vbWljIHN0YXR1cykKICAgIC0gV2hldGhlciBmb2NhbCBwcmVkaWN0b3IgdmFyaWFibGUocykgZGlmZmVyIGFzIGEgZnVuY3Rpb24gb2YgYW55IG1vZGVsIHZhcmlhYmxlcyAoaW5jbHVkaW5nIG91dGNvbWUgdmFyaWFibGUpIGFuZCBhcyBhIGZ1bmN0aW9uIG9mIGFueSBrZXkgZGVtb2dyYXBoaWMgY2hhcmFjdGVyaXN0aWNzCi0gRm9yIGluc3RhbmNlOgogICAgLSBXaGV0aGVyIG1hbGVzIGFyZSBtb3JlIGxpa2VseSB0aGFuIGdpcmxzIHRvIGJlIG1pc3Npbmcgc2NvcmVzIG9uIHRoZSBkZXBlbmRlbnQgdmFyaWFibGUKICAgIC0gV2hldGhlciBsb25naXR1ZGluYWwgYXR0cml0aW9uIGlzIGdyZWF0ZXIgaW4gbG93ZXIgc29jaW9lY29ub21pYyBzdGF0dXMgZmFtaWxpZXMKLSBJZiBtaXNzaW5nbmVzcyBkaWZmZXJzIHN5c3RlbWF0aWNhbGx5IGFzIGEgZnVuY3Rpb24gb2Ygb3RoZXIgdmFyaWFibGVzLCB5b3UgY2FuIGluY2x1ZGUgdGhhdCB2YXJpYWJsZSBhcyBhIGNvbnRyb2wgdmFyaWFibGUgaW4gbW9kZWxzLCBhbmQvb3IgY2FuIGluY2x1ZGUgdGhhdCB2YXJpYWJsZSBpbiBtdWx0aXBsZSBpbXB1dGF0aW9uIHRvIGluZm9ybSBpbXB1dGVkIHNjb3JlcyBmb3IgbWlzc2luZyB2YWx1ZXMKCiMjIyBMaXR0bGUncyBNQ0FSIFRlc3QKCmBgYHtyfQptY2FyX3Rlc3QobXlkYXRhKQpgYGAKCiMjIE11bHRpdmFyaWF0ZSBBc3NvY2lhdGlvbnMKCiMjIyBDb3JyZWxhdGlvbiBNYXRyaXgKCiMjIyMgUGVhcnNvbiBDb3JyZWxhdGlvbnMKCmBgYHtyfQpjb3IudGFibGUobXlkYXRhWyxjKCJwcmVkaWN0b3IiLCJvdXRjb21lIiwicHJlZGljdG9yT3ZlcnBsb3QiLCJvdXRjb21lT3ZlcnBsb3QiKV0pCmNvci50YWJsZShteWRhdGFbLGMoInByZWRpY3RvciIsIm91dGNvbWUiLCJwcmVkaWN0b3JPdmVycGxvdCIsIm91dGNvbWVPdmVycGxvdCIpXSwgdHlwZSA9ICJtYW51c2NyaXB0IikKY29yLnRhYmxlKG15ZGF0YVssYygicHJlZGljdG9yIiwib3V0Y29tZSIsInByZWRpY3Rvck92ZXJwbG90Iiwib3V0Y29tZU92ZXJwbG90IildLCB0eXBlID0gIm1hbnVzY3JpcHRCaWciKQpgYGAKCiMjIyMgU3BlYXJtYW4gQ29ycmVsYXRpb25zCgpgYGB7cn0KY29yLnRhYmxlKG15ZGF0YVssYygicHJlZGljdG9yIiwib3V0Y29tZSIsInByZWRpY3Rvck92ZXJwbG90Iiwib3V0Y29tZU92ZXJwbG90IildLCBjb3JyZWxhdGlvbiA9ICJzcGVhcm1hbiIpCmNvci50YWJsZShteWRhdGFbLGMoInByZWRpY3RvciIsIm91dGNvbWUiLCJwcmVkaWN0b3JPdmVycGxvdCIsIm91dGNvbWVPdmVycGxvdCIpXSwgdHlwZSA9ICJtYW51c2NyaXB0IiwgY29ycmVsYXRpb24gPSAic3BlYXJtYW4iKQpjb3IudGFibGUobXlkYXRhWyxjKCJwcmVkaWN0b3IiLCJvdXRjb21lIiwicHJlZGljdG9yT3ZlcnBsb3QiLCJvdXRjb21lT3ZlcnBsb3QiKV0sIHR5cGUgPSAibWFudXNjcmlwdEJpZyIsIGNvcnJlbGF0aW9uID0gInNwZWFybWFuIikKYGBgCgojIyMjIFBhcnRpYWwgQ29ycmVsYXRpb25zCgpFeGFtaW5lIHRoZSBhc3NvY2lhdGlvbnMgYW1vbmcgdmFyaWFibGVzIGNvbnRyb2xsaW5nIGZvciBhIGNvdmFyaWF0ZSAoYG91dGNvbWVPdmVycGxvdGApLgoKYGBge3J9CnBhcnRpYWxjb3IudGFibGUobXlkYXRhWyxjKCJwcmVkaWN0b3IiLCJvdXRjb21lIiwicHJlZGljdG9yT3ZlcnBsb3QiKV0sIHogPSBteWRhdGFbLGMoIm91dGNvbWVPdmVycGxvdCIpXSkKcGFydGlhbGNvci50YWJsZShteWRhdGFbLGMoInByZWRpY3RvciIsIm91dGNvbWUiLCJwcmVkaWN0b3JPdmVycGxvdCIpXSwgeiA9IG15ZGF0YVssYygib3V0Y29tZU92ZXJwbG90IildLCB0eXBlID0gIm1hbnVzY3JpcHQiKQpwYXJ0aWFsY29yLnRhYmxlKG15ZGF0YVssYygicHJlZGljdG9yIiwib3V0Y29tZSIsInByZWRpY3Rvck92ZXJwbG90IildLCB6ID0gbXlkYXRhWyxjKCJvdXRjb21lT3ZlcnBsb3QiKV0sIHR5cGUgPSAibWFudXNjcmlwdEJpZyIpCmBgYAoKIyMjIENvcnJlbG9ncmFtCgpgYGB7cn0KY29ycnBsb3QoY29yKG15ZGF0YVssYygicHJlZGljdG9yIiwib3V0Y29tZSIsInByZWRpY3Rvck92ZXJwbG90Iiwib3V0Y29tZU92ZXJwbG90IildLCB1c2UgPSAicGFpcndpc2UuY29tcGxldGUub2JzIikpCmBgYAoKIyMjIFNjYXR0ZXJwbG90IG1hdHJpeAoKYGBge3J9CnNjYXR0ZXJwbG90TWF0cml4KH4gcHJlZGljdG9yICsgb3V0Y29tZSArIHByZWRpY3Rvck92ZXJwbG90ICsgb3V0Y29tZU92ZXJwbG90LCBkYXRhID0gbXlkYXRhLCB1c2UgPSAicGFpcndpc2UuY29tcGxldGUub2JzIikKYGBgCgojIyMgUGFpcnMgcGFuZWxzCgpgYGB7cn0KcGFpcnMucGFuZWxzKG15ZGF0YVssYygicHJlZGljdG9yIiwib3V0Y29tZSIsInByZWRpY3Rvck92ZXJwbG90Iiwib3V0Y29tZU92ZXJwbG90IildKQpgYGAKCiMjIEVmZmVjdCBQbG90cwoKIyMjIE11bHRpcGxlIFJlZ3Jlc3Npb24gTW9kZWwKCmBgYHtyfQptdWx0aXBsZVJlZ3Jlc3Npb25Nb2RlbCA8LSBsbShvdXRjb21lIH4gcHJlZGljdG9yICsgcHJlZGljdG9yT3ZlcnBsb3QsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGRhdGEgPSBteWRhdGEsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIG5hLmFjdGlvbiA9ICJuYS5leGNsdWRlIikKCmFsbEVmZmVjdHMobXVsdGlwbGVSZWdyZXNzaW9uTW9kZWwpCnBsb3QoYWxsRWZmZWN0cyhtdWx0aXBsZVJlZ3Jlc3Npb25Nb2RlbCkpCmBgYAoKIyMjIE11bHRpbGV2ZWwgUmVncmVzc2lvbiBNb2RlbAoKYGBge3J9Cm11bHRpbGV2ZWxSZWdyZXNzaW9uTW9kZWwgPC0gbG1lKG91dGNvbWUgfiBwcmVkaWN0b3IgKyBwcmVkaWN0b3JPdmVycGxvdCwgcmFuZG9tID0gfiAxfElELAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBtZXRob2QgPSAiTUwiLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBkYXRhID0gbXlkYXRhLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBuYS5hY3Rpb24gPSAibmEuZXhjbHVkZSIpCgphbGxFZmZlY3RzKG11bHRpbGV2ZWxSZWdyZXNzaW9uTW9kZWwpCnBsb3QoYWxsRWZmZWN0cyhtdWx0aWxldmVsUmVncmVzc2lvbk1vZGVsKSkKYGBgCgojIFNlc3Npb24gSW5mbwoKYGBge3IsIGNsYXNzLnNvdXJjZSA9ICJmb2xkLWhpZGUifQpzZXNzaW9uSW5mbygpCmBgYAo=
