% setwd("d:/dev/twang/{t/doc") % Sweave("twang.rnw"); system("texify twang.tex"); system("c:\\MiKTeX\\miktex\\bin\\yap.exe twang.dvi",wait=FALSE) \SweaveOpts{prefix.string=twang} \documentclass{article} \bibliographystyle{plain} \usepackage[active]{srcltx} \usepackage{url} \addtolength{\topmargin}{-0.5in} \addtolength{\textheight}{0.75in} \addtolength{\oddsidemargin}{-0.5in} \addtolength{\textwidth}{1in} \newcommand{\EV}{\mathrm{E}} \newcommand{\Var}{\mathrm{Var}} \newcommand{\aRule}{\begin{center} \rule{5in}{1mm} \end{center}} %% Tracking changes and notes code added by Dan Mc 6-30-2014 %% \usepackage[normalem]{ulem} % for "tracked changes" \usepackage{color} % for "tracked changes" \newcommand{\note}[1]{{\color{blue} #1}} \newcommand{\dele}[1]{\textcolor{red}{\sout{#1}}} % delet in an equation \newcommand{\ins}[1]{\textcolor{blue}{\bf #1}} % insert \title{Propensity scores for multiple treatments:\\A tutorial for the \texttt{mnps} function in the \texttt{twang} package} \author{Lane Burgette, Beth Ann Griffin and Dan McCaffrey \footnote{The development of this software and tutorial was funded by National Institute of Drug Abuse grants number 1R01DA015697 (PI: McCaffrey) and 1R01DA034065 (PIs: Griffin/McCaffrey).} \\RAND Corporation} %\VignetteIndexEntry{Propensity scores for multiple treatments: A tutorial for the mnps function} %\VignetteDepends{gbm,survey,xtable} %\VignetteKeywords{propensity score, nonresponse weighting} %\VignettePackage{twang} \newcommand{\mathgbf}[1]{{\mbox{\boldmath$#1$\unboldmath}}} \begin{document} \maketitle \section{Introduction} The Toolkit for Weighting and Analysis of Nonequivalent Groups, \texttt{twang}, was designed to make causal estimates in the binary treatment setting. In \texttt{twang} versions 1.3 and later, we have extended this software package to handle more than two treatment conditions through the \texttt{mnps} function, which stands for \underline{m}ulti\underline{n}omial \underline{p}ropensity \underline{s}cores. McCaffrey et al.\ (2013) describe the methodology behind the \texttt{mnps} function; the purpose of this document is to describe the syntax and features related to the implementation in \texttt{twang}. At a high level, the \texttt{mnps} function decomposes the propensity score estimation into several applications of the \texttt{ps} function, which was designed for the standard dichotomous treatment setting. For this reason, users who are new to \texttt{twang} are encouraged to learn about the \texttt{ps} function before using the \texttt{mnps} function. The other vignette that accompanies the package (Ridgeway et al., 2014) provides an extensive overview of the \texttt{ps} function, and much of that information will not be repeated here. \section{An ATE example} <>= options(width=80) @ To demonstrate the package we use a random subset of the data described in McCaffrey et al.\ (2013). This truncated dataset is called \texttt{AOD}, and is included in the package. There are three treatment groups in the study, and the data include records for 200 youths in each treatment group of an alcohol and other drug treatment evaluation. We begin by loading the package and the data.\footnote{Code used in this tutorial can be found in stand alone text file at \url{http://www.rand.org/statistics/twang/downloads.html/mnps_turorial_code.r.}} %Because there is a stochastic component to the %subsequent model fits, we also set the random seed to ensure full %replicability. %\note{DFM: Is there really a stochastic component anymore %by default the bag fraction is 1 so there is no stochastics component. %I removed this from the SAS tutorial and don't recall if it is still in %the R ps tutorial} <<>>= library(twang) data(AOD) set.seed(1) @ For the \texttt{AOD} dataset, the variable \texttt{treat} contains the treatment indicators, which have possible values \texttt{community}, \texttt{metcbt5}, and \texttt{scy}. The other variables included in the dataset are: \begin{itemize} \item \texttt{suf12}: outcome variable, substance use frequency at 12 month follow-up \item \texttt{illact}: pretreatment covariate, illicit activities scale \item \texttt{crimjust}: pretreatment covariate, criminal justice involvement \item \texttt{subprob}: pretreatment covariate, substance use problem scale \item \texttt{subdep}: pretreatment covariate, substance use dependence scale \item \texttt{white}: pretreatment covariate, indicator for non-Hispanic white youth \end{itemize} In such an observational study, there are several quantities that one may be interested in estimating. The estimands that are most commonly of interest are the average treatment effect on the population (ATE) and the average treatment effect on the treated (ATT). The differences between these quantities are explained at length in McCaffrey et al.\ (2013), but in brief the ATE answers the question of how, on average, the outcome of interest would change if everyone in the population of interest had been assigned to a particular treatment relative to if they had all received another single treatment. The ATT answers the question of how the average outcome would change if everyone who received one particular treatment had instead received another treatment. We first demonstrate the use of \texttt{mnps} when ATE is the effect of interest and then turn to using the function to support estimation of ATT. \subsection{Estimating the weights} The main argument for the \texttt{mnps} function is a formula with the treatment variable on the left-hand side of a tilde, and pretreatment variables on the right-hand side, separated by plus signs. Other key arguments are \texttt{data}, which simply tells the function the name of the dataframe that contains the variables for the propensity score estimation; the \texttt{estimand}, which can either be ``ATT'' or ``ATE''; and \texttt{verbose}, which if set as \texttt{TRUE} instructs the function to print updates on the model fitting process, which can take a few minutes. <<>>= mnps.AOD <- mnps(treat ~ illact + crimjust + subprob + subdep + white, data = AOD, estimand = "ATE", verbose = FALSE, stop.method = c("es.mean", "ks.mean"), n.trees = 3000) @ The \texttt{twang} methods rely on tree-based regression models that are built in an iterative fashion. As the iterations or number of regression trees added to the model increases, the model becomes more complex. However, at some point, more complex models typically result in worse balance on the pretreatment variables and therefore are less useful in a propensity score weighting context. The \texttt{n.trees} argument controls the maximum number of iterations. Another key choice is the measure of balance that one uses when fitting these models. This is specified in the \texttt{stop.method} argument. As with the \texttt{ps} function, four \texttt{stop.method} objects are included in the package. They are \texttt{es.mean}, \texttt{es.max}, \texttt{ks.mean}, and \texttt{ks.max}. The four stopping rules are defined by two components: a balance metric for covariates and rule for summarizing across covariates. The balance metric summarizes the difference between two univariate distributions of a single pretreatment variable (e.g., illicit activities scale). The default stopping rules in \texttt{twang} use two balance metrics: absolute standardized mean difference (ASMD; also referred to as the absolute standardized bias or the effect size (ES)) and the Kolmogorov-Smirnov (KS) statistic. The stopping rule use two different rules for summarizing across covariates: the mean of the covariate balance metrics (``mean'') or the maximum of the balance metrics (``max''). The first piece of the stopping rule name identifies the balance metric (ES or KS) and the second piece specifies the method for summarizing across balance metrics. For instance, \texttt{es.mean} uses the effect size or ASMD and summarizes across variables with the mean and the \texttt{ks.max} uses the KS statistics to assess balances and summarizes using the maximum across variables and the other two stopping rules use the remaining two combinations of balance metrics and summary statistics. In this example, we chose to examine both \texttt{es.mean} and \texttt{ks.mean}. After running the \texttt{mnps()} command, a useful first step is to make sure that we let the models run for a sufficiently large number of iterations in order to optimize the balance statistics of interest. We do this by seeing whether any of the balance measures of interest still appear to be decreasing after the number of iterations specified by the argument \texttt{n.trees} which we set to 3,000 for this example (10,000 iterations is the default). \begin{center} <>= plot(mnps.AOD, plots = 1) @ \end{center} As noted above, \texttt{mnps} estimates weights by repeated use of the \texttt{ps} function and comparing each treatment the pooled sample of other treatments. The figure has one plot corresponding to each of those fits. Each plot is then further divided into one panel for each stopping rule used in the estimation. Since we used the ``es.mean'' and ``ks.mean'' stopping rules there are two panels in each plot. By default the plots for the different treatments are plotted in a single row; setting the height and width of the graphics device can make the plots easier to view. In this figure, it appears that each of the balance measures are optimized with substantially fewer than 3,000 iterations, so we do not have evidence that we should re-run the \texttt{mnps()} call with a higher number of iterations or trees. A key assumption in propensity score analyses is that each experimental unit has a non-zero probability of receiving each treatment. The plausibility of this assumption may be assessed by examining the overlap of the empirical propensity score distributions. This diagnostic is available using the \texttt{plots = 2} argument in the \texttt{plot} function. We use the \texttt{subset} option to specify which stopping rule we wish present in the plot.\footnote{The value for the \texttt{subset} argument can be a character variable with the name of the stopping, as was used in the example code, or a number corresponding to the stopping rule. Stopping rules are numbered by the alphabetical ordering among the rules specified in the \texttt{mnps} call.} \pagebreak \begin{center} <>= plot(mnps.AOD, plots = 2, subset = "es.mean") @ \end{center} Here, the overlap assumption generally seems to be met, although there should be some concern that adolescents in the metcbt5 and scy conditions do not overlap well with the community group given the top most graphic. See McCaffrey et al.\ (2013) for more details on this issue. \subsection{Graphical assessments of balance} As with the \texttt{ps} function for the binary treatment setting, the default plotting function for \texttt{mnps}-class objects also displays information on commonly-used balance statistics. In particular, when the \texttt{plots} argument is set equal to 3, it provides comparisons of the absolute standardized mean differences (ASMD) between the treatment groups on the pretreatment covariates, before and after weighting. When the \texttt{plots} argument is set equal to 4, the display is of $t$- and chi-squared statistic $p$-values comparing the two groups before and after weighting. However, whereas there is a single plot for these balance diagnostics in the binary treatment setting, in the multiple treatment case, one can either examine a plot for each of the treatment conditions, or summarize the balance statistics in some way, across the treatment conditions. As a default, the \texttt{plot} function for an \texttt{mnps} object returns the maximum of the pairwise balance statistics across treatment groups for each of the covariates: <>= options(width=120) @ \begin{center} <>= plot(mnps.AOD, plots = 3) @ \end{center} As shown here, after weighting, the maximum ASMD decreases for all pretreatment covariates. The statistically significant difference (before taking the maximum across treatment groups) is indicated by the solid circle. One may see the balance plots for the individual fits by setting the \texttt{pairwiseMax} argument to \texttt{FALSE}. <>= options(width=120) @ \pagebreak \begin{center} <>= plot(mnps.AOD, plots = 3, pairwiseMax = FALSE, figureRows = 3) @ \end{center} %\note{Maybe I am being too simplistic but I couldn't understand %how the max of the pairwise std effect in the individual %pairwise plots was about 0.10 but the max in maximum plot was %greater than 0.20. What am I missing? We need to explain this %in the text. Using twang 1.4, the y-axis label on the max plot %did not include "(maximum pairwise)". Also, the pairwise plots %had side by side panels for the stopping rules and they had %a better aspect ratio.} The additional \texttt{figureRows} argument instructs the function to spread the plots over three rows; by default the plots would be arranged in a single row rather than a column. We note here that red lines represent pretreatment covariates for which the pairwise ASMDs increase after weighting. Setting the \texttt{plots} argument equal to 4 displays $t$-test or $\chi^2$ statistic pairwise minimum $p$-values for differences between each of the individual treatment groups and observations in all other treatment groups. \pagebreak \begin{center} <>= plot(mnps.AOD, plots = 4) @ \end{center} <>= options(width=80) @ As seen in this figure, the pairwise minimum $p$-values all increase after propensity score weighting. Some of the figures include many frames, which can result in figures that are too big or difficult to read for some methods of display. To control this, three controls are available. First, the \texttt{treatments} argument can be used to specify only comparisons that involve a specific treatment level or, in the ATE case, only comparisons between two specified treatment levels. Similarly, the \texttt{singlePlot} argument . For example, \texttt{singlePlot = 2} would display only the second frame of those produced by the plot command (see figure below). Finally, specifying \texttt{multiPage = TRUE} prints the frames in succession. If this option is used after specifying a file to plot to (e.g., using \texttt{pdf()}), the frames will be printed on separate pages. \begin{center} <>= plot(mnps.AOD, plots = 2, subset = "es.mean", singlePlot = 2) @ \end{center} \subsection{Tabular assessments of balance} Beyond graphics, there are several other functions that may be of interest to \texttt{mnps} users. The first is given by the \texttt{bal.table} function. For propensity score analyses with multiple treatments, this function returns a lot of information. The intention with this function is that its output be loaded into a spreadsheet software program. (E.g., one can write the output into a .csv file using the \texttt{write.csv} function and open the resulting file using a spreadsheet application.) For each outcome category, and each stopping rule (in addition to the unweighted analysis) the \texttt{bal.table} function gives balance statistics such as weighted and unweighted means by treatment group. <<>>= bal.table(mnps.AOD, digits = 2) @ %\note{DFM: would it be useful to show that we can save the %results of bal.table? Also, there if you use digits it only %save 2 digits.} As of version 1.4 of TWANG, the balance measures are given for all pairwise combinations. (Prior to that version the balance measures were reported for each treatment against all others; we feel that the pairwise comparisons give a fuller accounting of balance in ATE applications.) More parsimonious versions of the summaries are available using the \texttt{collapse.to} argument. Setting \texttt{collapse.to = 'covariate'} gives the maximum of the ASMD and the minimum of the $p$-value across all pairwise comparisons for each pretreatment covariate and stopping rule. <<>>= bal.table(mnps.AOD, collapse.to = 'covariate', digits = 4) @ As shown, for each pretreatment variable, the maximum ASMD has decreased and the minimum $p$-values have increased after applying weights that arise from either stop.method. Another useful summary table sets \texttt{collapse.to = 'stop.method'} which further collapses the results above so that we summarize balance across all covariates and all pairwise group comparisons. <<>>= bal.table(mnps.AOD, collapse.to = 'stop.method', digits = 4) @ Here we quickly see how the maximum ASMDs and minimum $p$-values have all moved in the desired direction after propensity score weighting. Rather than collapsing the values of the table as described above, there are also several options for subsetting the \texttt{bal.table} output. The arguments \texttt{subset.var} and \texttt{subset.stop.method} instruct the function to include only the covariates indicated, and stop.method results indicated, respectively. The \texttt{subset.treat} instructs the function to return only the pairwise comparisons including the specified treatment or, if two treatment levels are indicated, the pair-wise comparisons that include those two treatments. Note that \texttt{subset.treat} may not be used when \texttt{collapse.to} is specified as \texttt{'stop.method'} or \texttt{'covariate'}. Further, the table may be subset on the basis of ES and KS and the related $p$-values via the \texttt{es.cutoff}, \texttt{ks.cutoff}, \texttt{p.cutoff}, and \texttt{ks.p.cutoff} arguments. These cutoffs exclude rows that are well-balanced as measured by the corresponding . For example \texttt{p.cutoff = 0.1} would exclude rows with $p$-values greater than 10\%, and \texttt{es.cutoff = 0.2} excludes rows with ES values below 0.2 in absolute value. Examples of the use of these subsetting arguments are given below. <<>>= bal.table(mnps.AOD, subset.treat = c('community', 'metcbt5'), subset.var = c('white', 'illact', 'crimjust')) @ <<>>= bal.table(mnps.AOD, subset.stop.method = 'es.mean', collapse.to = 'covariate') @ <<>>= bal.table(mnps.AOD, es.cutoff = 0.1) @ Finally, there is also \texttt{summary} method for the \texttt{mnps} objects which gives the collapsed version of \texttt{bal.table()} as well as information about the effective sample sizes for each treatment group under each stop.method. The \texttt{summary} function for an mnps output object does not have a \texttt{digits} argument. <<>>= summary(mnps.AOD) @ <>= options(width=80) @ After examining the graphical and tabular diagnostics provided by \texttt{twang}, we can analyze the outcome variable using the propensity scores generated by the \texttt{mnps} function. Although two stop methods were specified initially (\texttt{es.mean} and \texttt{ks.mean}), at this point we have to commit to a single set of weights. From the \texttt{bal.table} call above, we see that the balance properties are very similar for the two stopping rules, and from the \texttt{summary} statement, we see that the effective sample sizes (ESS) are similar as well. Hence, we expect the two stop methods to give similar results; we choose to analyze the data with the \texttt{es.mean} weights. \subsection{Estimating treatment effects} In order to analyze the data using the weights, it is recommended that one use the \texttt{survey} package, which performs weighted analyses. We can add the weights to the dataset using the \texttt{get.weights} function and specify the survey design as follows: <<>>= library(survey) AOD$w <- get.weights(mnps.AOD, stop.method = "es.mean") design.mnps <- svydesign(ids=~1, weights=~w, data=AOD) @ As shown in the \texttt{ps} vignette, we can then perform the propensity score-adjusted regression using the \texttt{svyglm} function: <<>>= glm1 <- svyglm(suf12 ~ as.factor(treat), design = design.mnps) summary(glm1) @ By default, \texttt{svyglm} includes dummy variables for MET/CBT-5 and SCY, Community is the holdout group (the holdout is the group with the label that comes first alphabetically). Consequently, the estimated effect for MET/CBT-5 equals the weighted mean for the MET/CBT-5 sample less the weighted mean for the Community sample, where both means are weighted to match the overall sample. Similarly, the effect fro SCY equals the difference in the weighted means for the SCY and Community samples. The coefficients estimate the causal effects of MET/CBT-5 vs.\ Community and SCY vs.\ Community, respectively, assuming there are no unobserved confounders. Using this small subset of the data, we are unable to detect differences in the treatment group means. In the context of this application, the signs of the estimates correspond to higher substance use frequency for youths exposed to MET/CBT-5 or SCY relative to Community. More details on how to obtain all relevant pairwise differences can be found in McCaffrey et al.\ (2013). As an alternative to estimating the pairwise differences, we could also estimate the causal effect of each treatment relative to the average potential outcome of all the treatments. This estimate is easy to obtain using \texttt{svyglm} through the use of the \texttt{constrast} argument in the function. <<>>= glm2 <- svyglm(suf12 ~ treat, design = design.mnps, contrast=list(treat=contr.sum)) summary(glm2) @ The function now provides the estimates for Community and MET/CBT-5. It labels them ``treat1'' and ``treat2'' because it uses their numeric codings rather than the factor levels. We have seen previously that the factor levels for treatment are ``community'', ``metcbt5'', and ``scy'' as levels, 1, 2, and 3. Relative to the average of all the treatments, the weighted Community group has lower substance use and the weighted MET/CBT-5 group has higher use. The SCY estimate is not reported because it is a linear combination of the other to estimates. It can be found by: <<>>= -sum(coef(glm2)[-1]) @ The standard error of this estimate can be calculated using the covariance matrix for the estimated coefficients: <<>>= sqrt(c(-1,-1) %*% summary(glm2)$cov.scaled[-1,-1] %*% c(-1,-1)) @ The SCY mean is about equal to the average and the difference between them is very small relative to its standard error. %\end{document} \section{An ATT example} \subsection{Estimating the weights} It is also possible to explore treatment effects on the treated (ATTs) using the \texttt{mnps} function. A key difference in the multiple treatment setting is that we must be clear as to which treatment condition ``the treated'' refers to. This is done through the \texttt{treatATT} argument. Here, we define the treatment group of interest to be the community group; thus, we are trying to draw inferences about the relative effectiveness of the three treatment groups for individuals like those who were enrolled in the community program. <<>>= mnps.AOD.ATT <- mnps(treat ~ illact + crimjust + subprob + subdep + white, data = AOD, estimand = "ATT", treatATT = "community", verbose = FALSE, n.trees = 3000, stop.method = c("es.mean", "ks.mean")) @ \subsection{Graphical assessments of balance} The same basic graphical descriptions are available as in the ATE case, though it is important to note that these comparisons all assess balance relative to the ``treatment'' group rather than by comparing balance for all possibly pairwise treatment group comparisons as is done with ATE. \begin{center} <>= plot(mnps.AOD.ATT, plots = 1) @ \end{center} \pagebreak \begin{center} <>= plot(mnps.AOD.ATT, plots = 3) @ \end{center} \begin{center} <>= plot(mnps.AOD.ATT, plots = 3, pairwiseMax = FALSE) @ \end{center} %\end{document} \begin{center} <>= plot(mnps.AOD.ATT, plots = 4) @ \end{center} <>= options(width=85) @ \subsection{Tabular assessments of balance} The \texttt{bal.table} output is similar to the ATE case. However, for ATT, we only report pairwise comparisons that include the \texttt{treatATT} category. <<>>= bal.table(mnps.AOD.ATT, digits = 2) @ <<>>= bal.table(mnps.AOD.ATT, digits = 2, collapse.to = "covariate") @ <<>>= bal.table(mnps.AOD.ATT, digits = 3, collapse.to = "stop.method") @ <>= options(width=80) @ \subsection{Estimating treatment effects} The process to analyze the outcome variable is also similar: <<>>= require(survey) AOD$w.ATT <- get.weights(mnps.AOD.ATT, stop.method = "es.mean") design.mnps.ATT <- svydesign(ids=~1, weights=~w.ATT, data=AOD) @ <<>>= glm1 <- svyglm(suf12 ~ as.factor(treat), design = design.mnps.ATT) summary(glm1) @ Note in this case that the estimated treatment effect of community on those exposed to the community treatment is slightly stronger than in the ATE case (high numbers are bad for the outcome variable). Although not statistically significant, such differences are compatible with the notion that the youths who actually received the community treatment responded more favorably to it than the ``average'' youth would have (where the average is taken across the whole collection of youths enrolled in the study). The discussion in McCaffrey et al.\ (2013) may be useful for determining whether the ATE or ATT is of greater interest in a particular application. \section{Conclusion} Often, more than two treatments are available to study participants. If the study is not randomized, analysts may be interested in using a propensity score approach. Previously, few tools existed to aide the analysis of such data, perhaps tempting analysts to ignore all but two of the treatment conditions. We hope that this extension to the \texttt{twang} package will encourage more appropriate analyses of observational data with more than two treatment conditions. \section*{Acknowledgements} The random subset of data was supported by the Center for Substance Abuse Treatment (CSAT), Substance Abuse and Mental Health Services Administration (SAMHSA) contract \# 270-07-0191 using data provided by the following grantees: Adolescent Treatment Model (Study: ATM: CSAT/SAMHSA contracts \# 270-98-7047, \# 270-97-7011, \#2 77-00-6500, \# 270-2003-00006 and grantees: TI-11894, TI-11874, TI-11892), the Effective Adolescent Treatment (Study: EAT; CSAT/SAMHSA contract \# 270-2003-00006 and grantees: TI-15413, TI-15433, TI-15447, TI-15461, TI-15467, TI-15475, TI-15478, TI-15479, TI-15481, TI-15483, TI-15486, TI-15511, TI15514, TI-15545, TI-15562, TI-15670, TI-15671, TI-15672, TI-15674, TI-15678, TI-15682, TI-15686, TI-15415, TI-15421, TI-15438, TI-15446, TI-15458, TI-15466, TI-15469, TI-15485, TI-15489, TI-15524, TI-15527, TI-15577, TI-15584, TI-15586, TI-15677), and the Strengthening Communities-Youth (Study: SCY; CSAT/SAMHSA contracts \# 277-00-6500, \# 270-2003-00006 and grantees: TI-13305, TI-13308, TI-13313, TI-13322, TI-13323, TI-13344, TI-13345, TI-13354). The authors thank these grantees and their participants for agreeing to share their data to support the development of the \texttt{mnps} functionality. \begin{thebibliography}{77} % start the bibliography \small % put the bibliography in a small font \bibitem{psChapt} Burgette, L.F., D.F. McCaffrey, B.A.\ Griffin (forthcoming). ``Propensity score estimation with boosted regression.'' In W.\ Pan and H.\ Bai (Eds.) {\em Propensity Score Analysis: Fundamentals, Developments and Extensions}. New York: Guilford Publications, Inc. \bibitem{mccaf} McCaffrey, D.F., B.A. Griffin, D.\ Almirall, M.E.\ Slaughter, R.\ Ramchand, and L.F.\ Burgette (2013). ``A tutorial on propensity score estimation for multiple treatments using generalized boosted models.'' Forthcoming at {\em Statistics in Medicine}. \bibitem{psVig} Ridgeway, G., D.\ McCaffrey, B.A.\ Griffin, and L.\ Burgette (2014). ``twang: Toolkit for weighting and analysis of non-equivalent groups.'' Available at http://cran.r-project.org/web/packages/twang/vignettes/twang.pdf. \end{thebibliography} % end the bibliography \end{document} \texttt{means.table} which provides a simple summary of balance across the groups. When \texttt{estimand} is set as \texttt{'ATE'}, the table shows the population means for each pretreatment covariate in the first column as well as each treatment group's unweighted and ATE weighted means and corresponding unweighted and weighted population standardized mean differences. As shown in the table below, incorporation of the ATE propensity score weights improves each treatment groups overall balance with the population means for each pretreatment covariate. The function also includes an argument called \texttt{includeSD} whose default is \texttt{FALSE}; changing it to \texttt{TRUE} returns standard deviations for each of the treatment conditions (not shown). <>= options(width=85) @ <<>>= means.table(mnps.AOD, stop.method = "es.mean", digits = 3) @ More extensive balance information is Although the same basic graphical descriptions are available as in the ATE case, note that the population means above are replaced with the means of the \texttt{treatATT} category in the \texttt{means.table} call. <<>>= means.table(mnps.AOD.ATT, digits = 3) @