\documentclass[nojss]{jss}
\usepackage{thumbpdf}
%% need no \usepackage{Sweave}

\newcommand{\update}[1]{\emph{\marginpar{UPDATE} (#1)}}

%% new environments for exams
\newenvironment{question}{\item \textbf{Problem}\newline}{}
\newenvironment{solution}{\textbf{Solution}\newline}{}

\author{Bettina Gr\"un\\Wirtschaftsuniversit{\"a}t Wien
   \And Achim Zeileis\\Universit\"at Innsbruck}
\Plainauthor{Bettina Gr\"un, Achim Zeileis}

\title{Automatic Generation of Exams in \proglang{R}}
\Plaintitle{Automatic Generation of Exams in R}

\Keywords{exams, multiple choice, arithmetic problems, Sweave, {\LaTeX}, \proglang{R}}
\Plainkeywords{exams, multiple choice, arithmetic problems, Sweave, LaTeX, R}

\Abstract{ 
  This introduction to version~1 of the \proglang{R}/\pkg{exams} package is a (slightly)
  modified version of \cite{exams:Gruen+Zeileis:2009}, published in the
  \emph{Journal of Statistical Software}. It describes how to produce PDF
  files from exercises in Sweave format. Meanwhile, \proglang{R}/\pkg{exams} has been
  considerably extended by \cite{exams:Zeileis+Umlauf+Leisch:2014} and beyond to
  also produce HTML output or e-learning exams for \pkg{Moodle}, \pkg{Canvas}, \pkg{OpenOlat}, etc.
  This has resulted in some small changes that are not fully backward-compatible
  and which are marked with ``UPDATE'' in the text below.
  
  For a current and more general introduction to the \proglang{R}/\pkg{exams} package
  see the materials at the official web page: \url{https://www.R-exams.org/}.

  Package \pkg{exams} provides a framework for automatic
  generation of standardized statistical exams which is especially
  useful for large-scale exams. To employ the tools, users just need
  to supply a pool of exercises and a master file controlling the
  layout of the final PDF document.  The exercises are specified in
  separate Sweave files (containing \proglang{R} code for data
  generation and {\LaTeX} code for problem and solution description)
  and the master file is a {\LaTeX} document with some additional
  control commands.  This paper gives an overview of the main design
  aims and principles as well as strategies for adaptation and
  extension. Hands-on illustrations -- based on example exercises and
  control files provided in the package -- are presented to get new
  users started easily.
}

\Address{
  Bettina Gr\"un\\
  Wirtschaftsuniversit{\"a}t Wien\\
  Institute for Statistics and Mathematics\\
  Welthandelsplatz~1, 1020 Wien, Austria\\
  E-mail: \email{Bettina.Gruen@wu.ac.at}\\
  URL: \url{https://statmath.wu.ac.at/~gruen/}\\

  Achim Zeileis\\
  Universit\"at Innsbruck\\
  Department of Statistics\\
  Universit\"atsstr.~15, 6020 Innsbruck, Austria\\
  E-mail: \email{Achim.Zeileis@R-project.org}\\
  URL: \url{https://www.zeileis.org/}
}

\begin{document}

\SweaveOpts{engine = R, eps = FALSE, keep.source = TRUE}
%\VignetteIndexEntry{Automatic Generation of Exams in R}
%\VignetteDepends{stats, graphics, tools, utils}
%\VignetteKeywords{exams, multiple choice, arithmetic problems, Sweave, LaTeX, R}
%\VignettePackage{exams}

<<preliminaries, echo=FALSE, results=hide>>=
options(width = 70, prompt = "R> ", continue = "+  ")
library("exams")
combine <- function(x, sep, width) {
  cs <- cumsum(nchar(x))
  remaining <- if (any(cs[-1] > width)) combine(x[c(FALSE, cs[-1] > width)], sep, width)
  c(paste(x[c(TRUE, cs[-1] <= width)], collapse= sep), remaining)
}
prettyPrint <- function(x, sep = " ", linebreak = "\n\t", width = getOption("width")) {
  x <- strsplit(x, sep)[[1]]
  paste(combine(x, sep, width), collapse = paste(sep, linebreak, collapse = ""))
}
@

\section{Introduction} \label{sec:intro}

The introductory statistics lecture at the Wirtschaftsuniversit\"at
Wien (WU Wien, \url{https://www.wu.ac.at/}) is
attended each semester by about 1,000--1,500 students (mostly first-year
business students). Several lecturers from the Department of
Statistics and Mathematics teach this course in parallel. In order to
ensure an efficient, consistent, and transparent organization, the course
format and all its teaching materials (presentation slides, collections
of exercises, exams, etc.) were re-designed in a collaborative
effort during 2006/7. Among many other aspects -- such as specification
of a topic list or definition of learning outcomes, etc. -- this
re-design encompassed several technological challenges.
Hence, the \pkg{exams} package was designed to address these challenges
and thus facilitate the discussions about the content of the new course.
More specifically, \pkg{exams} aims to provide software infrastructure for:
\begin{itemize}
  \item \textit{Scalable exams:} Automatic generation of a large number
        of different exams in order to provide an individual test to each student.
  \item \textit{Associated self-study materials:} Collections of exercises
        and solutions from the same pool of examples.
  \item \textit{Joint development:} Development and maintenance of a large
        pool of exercises in a multi-author and cross-platform setting.
\end{itemize}
Specifically, at WU Wien about 10--15 lecturers were working in small
teams of 2--4 people on different chapters for the presentation
slides.  For each chapter, the corresponding team would also provide
suitable exercise templates that could be used for self-study
materials, exams, and solutions.

The pool of exercises does not only need to contain different types of
exercises, but also variants of the same type to avoid that students
learn the solutions ``by heart''. Correction should be fast and
easy. This restricts the suitable types of exercises to those which
either have a single number as result which only needs to be checked to
assess the correctness, multiple-choice questions, or potentially
questions which require only a short text answer. These requirements on maintenance,
variation, and correction of exercises led to the following design
principles for package \pkg{exams}:
\begin{itemize}
  \item \textit{Maintenance:} Each exercise template is a single file
        (also just called ``exercise'').
  \item \textit{Variation:} Exercises are dynamic documents, containing a
        problem/solution along with a data-generating process (DGP) so that
	random samples can be drawn easily.
      \item \textit{Correction:} Solutions for exercises are either
        multiple-choice answers (logical vectors), numeric values
        (e.g., a test statistic or a confidence interval), short
        text answers (e.g., the appropriate null hypothesis corresponding to a
        given problem), or combinations of these.
\end{itemize}
Thus, the DGP of an exercise controls the distribution of possible
solutions and can be utilized to make them (approximately) evenly
distributed and difficult to ``guess'' or ``learn by heart''.  In
addition to the variability within an exercise, one can add further
variation by providing several exercise templates for the same type of
problem. Depending on the flexibility of the DGP, the pool of
exercises can thus be rather small or needs to be somewhat larger.

Mixing problems/solutions and DGPs for exam generation poses challenges 
that are similar to those of making data analysis reproducible.
Thus, \pkg{exams} employs many ideas from literate data analysis
\citep{exams:Rossini:2001,exams:Leisch+Rossini:2003}, literate programming
\citep{exams:Knuth:1992} and reproducible research \citep{exams:deLeeuw:2001}. 
Specifically, it makes extensive use of Sweave \citep{exams:Leisch:2002a}
for mixing DGPs written in the \proglang{R} system for statistical computing
\citep{exams:R:2008} and problem/solution descriptions written in
the typesetting system {\LaTeX} \citep{exams:Knuth:1984,exams:Lamport:1994}. 
Thus, the implementation in \pkg{exams} is based on
(1)~independent Sweave files for each exercise interweaving \proglang{R} and {\LaTeX} code,
(2)~different master {\LaTeX} files controlling the appearance and
(3)~an implementation of a minimal markup for communication with \proglang{R}
plus \proglang{R} functions tying everything together. 

The package \pkg{exams} that emerged from the developments at WU Wien
is available from the Comprehensive \proglang{R} Archive Network
at \url{https://CRAN.R-project.org/package=exams}. To utilize it for
exam generation, users need to generate a pool of exercises and supply
a master {\LaTeX} file. The package provides several examples for both
types of files along with documentation (like this paper) and hands-on
examples to get new users started.\footnote{In version 2 of the \pkg{exams}
package a new function \texttt{exams\_skeleton()} has been added to facilitate
starting new exams projects. See Section~2.3 of \cite{exams:Zeileis+Umlauf+Leisch:2014}
or \code{vignette("exams2", package = "exams")} for more details.}
Hence, the remainder of this paper is organized as follows: Section~\ref{sec:exercises} explains the
structure of the Sweave files defining the exercises and demonstrates
how the final output file is derived from the original \proglang{R}
and {\LaTeX} code. In Section~\ref{sec:master}, the structure of the
master {\LaTeX} file for constructing the final PDF file from the
exercise files is presented. Finally Section~\ref{sec:customization}
illustrates how to use the package in practice and how \pkg{exams} can
be customized. Experience with the development of exercises and the
application of the package are discussed in
Section~\ref{sec:discussion}. Section~\ref{sec:summary} concludes the
paper with a summary.

\section{Exercises} \label{sec:exercises}

Each exercise is contained in a separate Sweave file and typically
represents an exemplary application of a statistical procedure.  It
needs to contain (1)~a question and a solution description which are
encapsulated in {\LaTeX} environments of the corresponding names and
(2)~some meta-information about the exercise such as type and the
solution.  To allow for variability, the data for the exercise can be
generated within the Sweave file in a code chunk (typically suppressed
in the final document). Meta-information for the examples needs to be
provided, e.g., for computing with the solution within \proglang{R} or
for generating lists with solutions.

In Table~\ref{tab:Rnw}, the Sweave file for a simple exercise asking
students to compute a one-sample $t$~test statistic is shown for
illustration. The four different blocks (\proglang{R} code with DGP,
\code{question} environment, \code{solution} environment,
meta-information) can be easily distinguished. The {\LaTeX} file
generated from a call to \code{Sweave()} is shown in
Table~\ref{tab:tex}, and the final compiled PDF output is provided in
Table~\ref{tab:pdf}.

The \proglang{R} code for the DGP is used as in any other Sweave
document: The code chunk is framed by \verb|<<>>=| and \code{@} and
options in the header control formatting (typically suppressing the
display in the final document, as in this example). For further
information about the application of Sweave and a detailed description
of the different options see
\cite{exams:Leisch:2008a,exams:Leisch:2008}.

The core of an exercise are two blocks for the question and the
solution in their accordingly named environments. The values generated
by the DGP are typically included using \verb|\|\verb|Sexpr{}|
statements. Furthermore, question/solution might contain graphics and/or
\proglang{R} output created by Sweave. To include the {\LaTeX} output
(see Table~\ref{tab:tex}) in some master {\LaTeX} file, this needs to define
the \code{question} and \code{solution} environment. This mechanism can be
used to control the display, e.g., to suppress the solution in an exam
but to display it in associated self-study material
(see Section~\ref{sec:master} for further details).

%% from Rnw
\begin{table}[p!]
\begin{center}
\makebox[\textwidth]{\hrulefill}

<<echo=FALSE, results=tex>>=
invisible(file.copy(system.file("exercises", "tstat.Rnw", package = "exams"), "tstat.Rnw"))
Rnw <- readLines("tstat.Rnw")
cat(c("\\begin{verbatim}", Rnw, "\\end{verbatim}"), sep = "\n")
@

\makebox[\textwidth]{\hrulefill}
\caption{\label{tab:Rnw} A simple Sweave exercise: \code{tstat.Rnw}.}
\end{center}
\end{table}

%% to LaTeX
\begin{table}[t!]
\begin{center}
\makebox[\textwidth]{\hrulefill}

<<echo=FALSE, results=hide>>=
set.seed(1090)
Sweave("tstat.Rnw")
tex <- readLines("tstat.tex")
file.remove(c("tstat.Rnw", "tstat.tex"))
@
<<echo=FALSE, results=tex>>=
cat(c("\\begin{verbatim}", tex, "\\end{verbatim}"), sep = "\n")
@

\makebox[\textwidth]{\hrulefill}
\caption{\label{tab:tex} {\LaTeX} output of \code{Sweave("tstat.Rnw"}).}
\end{center}
\end{table}

%% to PDF
\begin{table}[t!]
\makebox[\textwidth]{\hrulefill}
\begin{enumerate}

<<echo=FALSE, results=tex>>=
cat(tex, sep = "\n")
@

\end{enumerate}
\makebox[\textwidth]{\hrulefill}
\begin{center}
\caption{\label{tab:pdf} Display of processed exercise from \code{exams2pdf("tstat.pdf")}.}
\end{center}
\end{table}

Finally, some meta-information needs to be provided. All meta-information
commands are in {\LaTeX} style but are actually commented out and hidden in the final
output file. There are four commands\footnote{Note that version~2 provides many additional
commands, especially for e-learning output and other custom interfaces.}:
\begin{itemize}
  \item \verb|\extype{}|: type of question. It may be either \code{mchoice}
        for multiple-choice questions, \code{num} for questions with a
        numeric result or \code{string} for questions with a (short) text answer.
  \item \verb|\exsolution{}|: solution. It contains the numeric
        solution, a string of zeros and ones for multiple-choice
        questions or a character string. For confidence intervals, it
        may also specify two numeric solutions of type
        \verb|\exsolution{|\emph{lower}\verb|}{|\emph{upper}\verb|}|.
  \item \verb|\exname{}|: a name or description. This is some
        short informative text, only used for printing of solutions
        within \proglang{R}.
  \item \verb|\extol{}|: optional tolerance limit(s). For numeric solutions
        a tolerance limit can be specified; by default it is assumed to
        be 0. This can be useful for the automatic evaluation of numeric
	solutions if some deviation from the correct answer is allowed
	(e.g., due to rounding errors). If a single tolerance value is supplied,
	a symmetric interval around the true value is used. \update{Version~1
	of the \pkg{exams} package also supported asymmetric tolerance intervals
	when two tolerance values were provided. However, this has not been
	implemented for any of the version~2 interfaces.}
\end{itemize}
The meta-information is read from the {\LaTeX} file (e.g., as in Table~\ref{tab:tex})
and hence all values can be dynamically computed using \verb|\|\verb|Sexpr{}|.
Typically, this will be necessary for the correct solution, but similarly the
descriptive name, the tolerance allowed, or even the type of the question could
be data-driven and determined by the DGP.

The user does not have to go from the Sweave file (Table~\ref{tab:Rnw})
to the {\LaTeX} file (Table~\ref{tab:tex}) to the PDF file (Table~\ref{tab:pdf})
``by hand''. The function \code{exams2pdf()} from package \pkg{exams} carries
out all of these steps automatically. \update{In version~1 of the package
the function \texttt{exams()} was provided. While this is still contained in the
package, it is recommended to use the newer and more flexible \texttt{exams2pdf()}
for new projects.}
%
<<tstat-interacive, echo=TRUE, eval=FALSE>>=
library("exams")
set.seed(1090)
tstat_ex <- exams2pdf("tstat.Rnw")
@ 
<<tstat-non-interactive, echo=FALSE, eval=TRUE>>=
set.seed(1090)
dir.create(tdir <- tempfile())
tstat_ex <- exams2pdf("tstat.Rnw", dir = tdir)
tstat_sol <- exams_metainfo(tstat_ex)
@ 
%
After loading the package, \code{exams2pdf()} is called with the Sweave file name.
The final PDF file is displayed on screen by default (for further options see
Section~\ref{sec:customization}). The PDF viewer used depends on the operating
system: On Windows, the application specified in the system's file
associations is used. On other systems (e.g., Linux or Mac OS),
the PDF viewer specified in the options (see \code{getOption("pdfviewer")})
is started. \code{exams2pdf()} returns a list of exams which is a list of
exercises \citep[see][for details]{exams:Zeileis+Umlauf+Leisch:2014}. To extract an
``\code{\Sexpr{class(tstat_sol)}}'' object for pretty printing in \proglang{R},
the function \code{exams_metainfo()} can be used:
%
<<>>=
exams_metainfo(tstat_ex)
@ 

\section[Combining exercises: The master LaTeX file]{Combining
  exercises: The master {\LaTeX} file} \label{sec:master}

To produce the PDF for an exam, the exercises need to be weaved,
tied together in the master {\LaTeX} file and subsequently processed
to PDF. More precisely, \code{exams2pdf()} takes the following approach: 
\begin{enumerate}
  \item Collect all Sweave files for the exercises, the master {\LaTeX}
        file(s) and potentially  additionally specified input files.
  \item Copy all files to a (temporary, by default) directory.
  \item Run \code{Sweave()} for each exercise.
  \item Produce a copy of the master {\LaTeX} file(s) in which certain
    control structures are substituted by dynamically generated
    {\LaTeX} commands (e.g., for including the exercises).
  \item Run \code{texi2dvi()} for each master {\LaTeX} file.
  \item Store the resulting PDF file(s) in an output directory
        or display it on the screen (for a single file only, by default).
\end{enumerate}
Thus, the only thing needed to combine several independent exercises
to an exam is a master {\LaTeX} file. Its basic structure is rather
straightforward but for more elaborate layouts it can become quite
complex. Package \pkg{exams} provides various examples of master
{\LaTeX} files in the \code{inst/tex/} directory of the source
package.  The simplest one is \code{plain.tex} (see
Table~\ref{tab:master}) which is described here to provide a first
overview. \update{Compared to the original paper, this {\LaTeX} file
grew somewhat for two reasons: First, instead of the dependency on the
\code{Sweave.sty} file, the necessary packages and environments are defined
directly. Second, several commands are needed for compatibility with
output from \pkg{pandoc} for \proglang{R}/Markdown-based exercises.}

The structure of \code{plain.tex} is as follows:
First, the document class is defined and then necessary add-on
packages are loaded.
Then the environments for \code{question}, \code{solution}, and \code{answerlist} are defined
(as part of an itemize or enumerate list). Moreover, some commands and environments
for compatibility with both \code{Sweave()} and \pkg{pandoc} output are provided.
Finally, there is a placeholder for further header commands that can be inserted
and then the document starts and consists only of an enumerate list. In Step~4 of the algorithm described above,
the control command \verb|%% \exinput{exercises}| is replaced by
(a sequence of) \verb|\input{|{\it filename}\verb|}|, e.g., \verb|\inp|\verb|ut{exercise1.tex}|\footnote{Leading
zeros are added if \code{n} has more than one digit, e.g., \texttt{exercise01.tex} if 
there are between ten and 99 exercises.}
for the first exercise etc. Optionally, the \verb|%% \exinput{exercises}| comment
can also be omitted in the template. Then, the user can place his/her own\linebreak
\texttt{\textbackslash input\{exercise1.tex\}}, \texttt{\textbackslash input\{exercise2.tex\}}, \dots
in the master {\LaTeX} file. This requires knowing the correct number of exercises but
adds the flexibility of including arbitrary text/formatting between the exercises.

\begin{table}[t!]
\begin{center}
\makebox[\textwidth]{\hrulefill}

<<echo=FALSE, results=tex>>=
tex <- readLines(system.file("tex", "plain.tex", package = "exams"))
a <- grep("newenvironment{answerlist}", tex, fixed = TRUE)
tex <- c(tex[1:(a-1)], substr(tex[a], 1, 72), paste(" ", substr(tex[a], 73, nchar(tex[a]))), tex[-(1:a)])
cat(c("\\begin{verbatim}", tex, "\\end{verbatim}"), sep = "\n")
@

\makebox[\textwidth]{\hrulefill}
\caption{\label{tab:master} A basic master {\LaTeX} file: \code{plain.tex}.}
\end{center}
\end{table}

Thus, all appearance options can be controlled in the {\LaTeX} code of
the master file, e.g., by changing the definitions of the \code{question}/\code{solution}
environments or by modifying the code around the \verb|\exinput{}| command.
As an example, one might want to show only the questions and hide the
solutions in the PDF. This is easily obtained by changing the definition
of the \code{solution} environment to
\verb|\newenvironment{solution}{\comment}{\endcomment}|.

<<echo=FALSE, results=hide>>=
tstat_char <- strsplit(gsub("\\.", "", as.character(tstat_sol[[1]][[1]]$solution)), "")[[1]]
tstat_exnum <- rep("", 9)
tstat_exnum[(10 - length(tstat_char)):9] <- tstat_char
tstat_exnum <- paste("{", tstat_exnum, "}", sep = "", collapse = "")
@

The structure described so far is completely static, except for the
\verb|\exinput{}| commands. In fact, \code{exercises} is not the only argument
allowed; at the moment two further arguments
can be used to modify aspects of the master {\LaTeX} file dynamically:
\begin{itemize}
\item \verb|\exinput{exercises}|: Inclusion of exercises.\\
      Replaced by: \verb|\input{|{\it filename}\verb|}| (one for each exercise).\\
      Example: \verb|\input{exercise1.tex}|.
\item \verb|\exinput{questionnaire}|: Inclusion of questionnaires, e.g., for cover sheets.\\
      Replaced by: \verb|\exnum{|{\it \dots}\verb|}|, 
                   \verb|\exmchoice{|{\it \dots}\verb|}|, or
                   \verb|\exstring{|{\it \dots}\verb|}|,
                   respectively (one for each exercise).\\
      Example: \verb|\exnum\Sexpr{tstat_exnum}|.
\item \verb|\exinput{header}|: Further commands and definitions.\\
      Replaced by: \verb|\|{\it command}\verb|{|{\it value}\verb|}| (one for each header command).\\
      Example: \verb|\Date{\Sexpr{Sys.Date()}}|.
\end{itemize}
For the latter two, the master {\LaTeX} file has to define the commands that are
used in the replacement step, e.g., \verb|\exnum| has to be a command that takes nine
arguments (corresponding to six and three digits before and after the decimal point for numeric
solutions), \verb|\exmchoice| has to take one required and one or more optional arguments
such as \verb|\exmchoice{X}[][X][][]| (corresponding to the logical
multiple-choice answers) and \verb|\exstring| has to take only one argument (corresponding 
to the solution string). \update{The implementation of \texttt{mchoice} was slightly
different in version~1 of the package where only a fixed number of five multiple choices
was supported.} For the header, arbitrary commands can be set up, see
Section~\ref{sec:customization} for details.

Two master {\LaTeX} files that illustrate all of the commands above are
provided in \pkg{exams}: \code{exam.tex} and \code{solution.tex}.
\code{exam.tex} can be used
to generate exams including a cover sheet for students to fill in
their names as well as their answers to the problems (hiding solutions
for the exercises, obviously). \code{solution.tex} produces PDF files containing a
cover sheet similar to \code{exam.tex} but with the correct answers
already filled in. Furthermore, the \code{solution} environments are displayed
for each exercise.

\section[Application and customization]{Application and customization} \label{sec:customization}

In the simple case where only a single Sweave exercise is processed
running \code{exams2pdf()} essentially corresponds to first calling
\code{Sweave()} and then \code{texi2dvi()} on the file after
including it in a master {\LaTeX} file. This is quite convenient,
especially for non-experts (in \proglang{R} and/or {\LaTeX}), but
beyond that not much simplification is gained by \code{exams2pdf()}.
The main advantages of the function, however, are its flexibility
and customizability: It controls the (dynamic) combination of the
Sweave and {\LaTeX} files and allows for
\begin{itemize}
  \item construction of exams with stratified sampling of exercises,
  \item automatic generation of multiple copies (potentially of
        multiple layouts) with suitable names and storage,
  \item inclusion of a suitable cover page with answer fields,
  \item collection of meta-information for problems and solutions in
        an \proglang{R} object.
\end{itemize}
In the following, we illustrate how \code{exams2pdf()} can be employed
to exploit all of these features. First, the interface of \code{exams2pdf()}
is briefly described before using it to generate a small set of exams
with corresponding solutions.
Function \code{exams2pdf()} has the following arguments:
\begin{verbatim}
<<results=tex, echo=false>>=
cat(prettyPrint(prompt(exams2pdf, filename = NA)$usage[[2]], sep = ", ", 
  linebreak = paste("\n", paste(rep(" ", nchar("exams2pdf") + 1), collapse = ""), sep= ""),
  width = 60))
@ 
\end{verbatim}
where \code{file} specifies a list/vector of exercise Sweave files
(see Section~\ref{sec:exercises}), \code{template} is the name of the
master {\LaTeX} file (see Section~\ref{sec:master}), and \code{n} is
the number of random replications. The remaining arguments control
details of the processing, e.g., the directories for input/output
files. A technical manual is available on the help page \code{?exams2pdf},
a brief hands-on introduction to selected arguments is given below:
\begin{description}
  \item[\normalfont \code{file}:] This is either a character vector containing the
        file names of Sweave exercises or a list of such vectors, e.g.,
<<>>=
myexam <- list("boxplots",
               c("confint", "ttest", "tstat"),
               c("anova", "regression"),
               "scatterplot",
               "relfreq")
@ 
	Exams generated from \code{myexam} always have five exercises: \code{"boxplots"},
	\code{"scatterplot"}, and \code{"relfreq"} are always included but the
	second exercise is randomly drawn from \code{"confint"}, \code{"ttest"},
	\code{"tstat"}. Similarly, the third exercise is randomly chosen from
	\code{"anova"} and \code{"regression"}. This stratified sampling strategy
	is useful if there are several exercises related to the same topic,
	or several exercises for the same statistical techniques with different
	``stories''. If only a single vector (rather than a list)
	of file names is specified, each exercise is always included in the final output file.       	
	The extension \code{.Rnw} can be omitted from the file names
        and the corresponding files should either be in the local directory,
        the \code{edir} directory or in the \code{exercises} directory of the
        installed package. 
  
  \item[\normalfont \code{n}:] Number of randomly generated exams.
    
  \item[\normalfont \code{nsamp}:] Number of elements sampled from each
        list element of \code{file}. Sampling without replacement is used (if possible).
  
  \item[\normalfont \code{dir}:] Output directory for storing the resulting PDF files (and
        meta-information). If only a single PDF file is generated, this can
	be omitted (and the result is displayed directly on the screen), otherwise
	it has to be specified. 

  \item[\normalfont \code{template}:] A (vector of) master {\LaTeX} file(s). If more than
        one \code{template} is specified, one PDF output file is created for
	each in each of the \code{n} runs. The extension \code{.tex} can be
	omitted in the file name and the corresponding files should either
	be in the local directory (or provided with the full path) or in the
	\code{tex} directory of the installed package.
	
  \item[\normalfont \code{header}:] Additional {\LaTeX} commands for
        replacement of \verb|\exinput{header}| in the master {\LaTeX}
        file.  It has to be a \code{list()} of \textit{command}\code{
        = }\textit{value} pairs, where \textit{value} can either be a
        static string or a function computing a string from the index
        $i$ of the $i$th exam. 
        
  \item[\normalfont Further arguments:] \code{inputs} can specify a list of files
        needed during the {\LaTeX} compilation, e.g., private \code{.sty} files
	etc. \code{name} is the (vector of) prefix(es) for the final PDF
	files. \code{quiet = TRUE} suppresses output when calling
        \code{Sweave()} and \code{texi2dvi()}. \code{edir} is the path
	to the exercise directory (defaulting to the current working directory).
	\code{tdir} is the temporary (by default) directory into which
	all files are copied and where \code{Sweave()} and \code{texi2dvi()}
	are called. \code{sdir} is the directory (tempoarary by default)
	in which supplementary files (such as graphics) are stored.
	\code{verbose = TRUE} displays some progress information and	
	\code{control} allows a few further control options
	(see also below).
\end{description}
%
For the subsequent example, we use a temporary output directory (but the reader
could easily change \code{odir} to some local directory).
<<>>=
odir <- tempfile()
@ 
%
Assume that we want a different ID for each exam. To accomplish this, we 
define a function which is used to substitute a different \verb|\ID{myexam|\textit{ii}\verb|}|
command in the header of each exam. (The {\LaTeX} command \verb|\ID{}|
has to be defined in the master {\LaTeX} files.)
<<>>=
getID <- function(i) 
  paste("myexam", gsub(" ", "0", format(i, width = 2)), sep = "")
getID(1)
@ 
%
Using these arguments, a set of exams can be easily produced:
<<>>=
set.seed(1090)
ex <- exams2pdf(myexam, n = 5, nsamp = c(1, 2, 1, 1, 1), dir = odir, 
 template = c("exam", "solution"), 
 header = list(ID = getID, Date = Sys.Date()))
@ 
This takes the exam \code{myexam} (for which the corresponding Sweave files
are all provided in the \code{exercises} directory of the package) and
produces five exams (from the \code{exam.tex} {\LaTeX} file) with associated
solutions (from the \code{solution.tex} {\LaTeX} file). The templates are
both provided in the \code{tex} directory of the package and allow for
specification of \verb|\ID{}| and \verb|\Date{}|. All output files
are stored in \code{odir}
<<>>=
list.files(odir)
@ 
and can now be easily inspected by the reader. If in addition to the
PDF exams and solutions, further information about the exams is needed
the \code{ex} list returned could be stored for future reference.
This contains all {\LaTeX} code for questions/solutions along with
the metainformation about the correct answers etc. The latter can
also be extracted and printed within \proglang{R}:
<<>>=
sol <- exams_metainfo(ex)
print(sol, 1)
print(sol, "exam5")
@
%
For numeric results, the solution is displayed, possibly including
tolerance limits (if non-zero). For multiple-choice answers,
the true statements (coded with 1--5) are listed and the false
ones are omitted. Consequently, if a multiple-choice question
has no true statements, no numbers are displayed.

 
\section{Discussion} \label{sec:discussion}

\subsubsection*{Infrastructure vs.\ content}

Package \pkg{exams} provides the technological framework for the
generation of structured exams, especially for large-lecture courses.
Given a relatively simple structure of exams (sequence of ``stand-alone'' exercises
with multiple-choice/numeric/short text solutions), it is designed
to be as flexible as possible. Thus, users should not have to worry
about implementation details and can focus on the specification of the
content and the development of the pool of exercises. While the package's
structure can aid the design and development of the exercises, the package can, of course,
not assure that ``good'' exercises (from an educational point of view)
are generated. As this is beyond the scope of \pkg{exams}, some brief pointers
to the relevant literature on statistical education and assessment are given here:
\cite{exams:Gal+Garfield:1997} and \cite{exams:Garfield+Chance:2000}
discuss issues such as topics covered and skills developed in
statistics courses as well as suitable ways of assessment. 
The development of effective multiple-choice
questions which force the student to understand underlying statistical
concepts is crucial and often not straightforward. Strategies for
good multiple-choice questions, especially if they are also used for
self-study materials, are suggested by \cite{exams:Klinke:2004}.

\subsubsection*{Strategies for setting up exercises}

As pointed out above, \pkg{exams} does not really address the problem
of designing ``good'' exercises. However, the infrastructure supports
several commonly-used strategies for setting up different types of exercises
which in turn can support the decisions about the content. For example,
in the case of multiple-choice two typical strategies are: Either
only one option is correct or any one option can be correct or false.
In the former case, the evaluation typically penalizes incorrect answers
(to avoid random guessing) while in the latter case the number of potential
patterns is sufficiently large that no penalization is necessary. In
both cases, it is good practice to check the DGP employed for generating
the answers and look at the resulting distribution of answer patterns.
For exercises with numerical solutions, on the other hand, a typical
problem in practice is how to set the tolerance. While the instructor
will want to catch typical mistakes (e.g., wrong standardization, etc.),
certain imprecisions (e.g., rounding errors in intermediate steps)
might be tolerable. The simpler and somewhat more rigid approach to
this problem is to practice with the students how to avoid the imprecisions
and just require a certain accuracy of the solution (as in the $t$~test example in
Tables~\ref{tab:Rnw} and \ref{tab:tex}). A more flexible approach
would be to derive the tolerance dynamically (as part of the DGP)
assuring that common small imprecisions lead to results within
tolerance limits while typical mistakes are not. 

\subsubsection*{Experiences at WU Wien}

During 2007, \pkg{exams} was employed by about 10--15 lecturers at the
Department of Statistics and Mathematics of WU Wien for jointly
developing materials for the basic statistics lecture. With package
\pkg{exams} we were able to address the technological requirements and
facilitate the development process. All lecturers involved were
familiar with \proglang{R} and {\LaTeX} (which are both available on
all standard platforms) and could thus contribute to the pool of
exercises. To do so, they just needed to know the structure of the
exercise Sweave files while different master {\LaTeX} files (for the
department's exams, exercise collections, etc.) have been written by
the authors of the \pkg{exams} package.  As an additional tool we
decided to use \pkg{Subversion}
\citep[SVN,][]{exams:Pilato+Collins-Sussman+Fitzpatrick:2004} for
version control in order to provide all lecturers involved with access
to all resources. In combination with the package this approach proved
to be rather successful in addressing the needs of multi-author and
cross-platform development.
Since Spring 2008, \pkg{exams} is used at WU Wien for generating
collections of exercises and trial exams (both available prior to the
actual exam) as well as the exams and associated solutions (which are
e-mailed to the students individually after correction of the test).
Exams containing numeric answers are still corrected ``by hand'' but
for exams consisting entirely of multiple/single-choice answers automatic
scanning using optical character recognition (OCR) is used.\footnote{We also
have an \proglang{R} implementation for scanning multiple/single-choice
exams. However, this is somewhat geared towards the exam sheets used
at WU Wien and Universit\"at Innsbruck and hence not part of the \pkg{exams}
package. Readers interested in this code should contact the package
authors.}

\section{Summary} \label{sec:summary}

Package \pkg{exams} provides a framework for automatic generation of
simple (statistical) exams and associated self-study materials.
It is based on independent exercises in Sweave format which can be
compiled in exams (or other collections of exercises) by providing 
one (or more) master {\LaTeX} template(s). Because contributing to the
pool of exercises just requires knowledge of Sweave and minimal markup
for meta-information, \pkg{exams} facilitates joint development of
lecture materials. An extension of the package to e-learning exams,
e.g., for \pkg{Moodle} or \pkg{OLAT}/\pkg{OpenOlat} is described
in \cite{exams:Zeileis+Umlauf+Leisch:2014}.

\section*{Acknowledgments}

We are indebted to our colleagues at the Department of Statistics and
Mathematics at WU Wien -- in particular Regina T\"uchler and Josef
Leydold -- for testing and challenging the code and making suggestions
for improvement. We would like to thank two anonymous referees and one
associate editor for their valuable comments which led to several
improvements. This research was partially supported by the Austrian
Science Foundation (FWF) under Hertha-Firnberg grant T351.

\bibliography{exams}

\end{document}