---
title: "Compliance Documentation"
author: "Peter Hurford"
date: "`r Sys.Date()`"
output:
  rmarkdown::html_vignette:
    fig_caption: yes  
vignette: >
  %\VignetteIndexEntry{Compliance Documentation}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

**Compliance documentation** is a premium add-on product to DataRobot. It allows users to automatically generate and download documentation to assist with deploying models in highly regulated industries.


## Connect to DataRobot

To explore compliance documentation, let's first connect to DataRobot. First, you must load the DataRobot R package library.

If you have set up a credentials file, `library(datarobot)` will initialize a connection to DataRobot automatically. Otherwise, you can specify your `endpoint` and `apiToken` as in this example to connect to DataRobot directly. For more information on connecting to DataRobot, see the "Introduction to DataRobot" vignette.

```{r results = "asis", message = FALSE, warning = FALSE, eval = FALSE}
library(datarobot)
endpoint <- "https://<YOUR DATAROBOT URL GOES HERE>/api/v2"
apiToken <- "<YOUR API TOKEN GOES HERE>"
ConnectToDataRobot(endpoint = endpoint, token = apiToken)
```


## Download Compliance Documentation

To download compliance documentation for a particular model, call `DownloadComplianceDocumentation` on a particular model and specify a filepath to download the documentation to. Note that it downloads in DOCX format.

```{r results = "asis", message = FALSE, warning = FALSE, eval = FALSE}
DownloadComplianceDocumentation(model, "path/to/filename.docx")
```


## Creating a Custom Template

You can also use your own custom compliance documentation templates.

#### The Default Template

First, let's get the default template. This can be done just by using `GetComplianceDocTemplate`. It downloads as a JSON file.

```{r results = "asis", message = FALSE, warning = FALSE, eval = FALSE}
GetComplianceDocTemplate("path/to/filename.json")
```

#### Updating the Default Template

A common workflow for building your own template is downloading the default template and modifying it.

```{r results = "asis", message = FALSE, warning = FALSE, eval = FALSE}
DownloadComplianceDocTemplate("path/to/filename.json")
# ...then modify the compliance doc template in your favorite editor.
UploadComplianceDocTemplate(name = "myNewTemplate", filename = "path/to/modified_file.json")
```

Alternatively, you can construct a template via a list:

```{r results = "asis", message = FALSE, warning = FALSE, eval = FALSE}
sections <- list(list("title" = "Missing Values Report",
                      "highlightedText" = "NOTICE",
                      "regularText" = "This dataset had a lot of Missing Values. See the chart below: {{missingValues}}",
                      "type" = "user"),
                 list("title" = "Blueprints",
                      "regularText" = "{{blueprintDiagram}} /n Blueprint for this model",
                      "type" = "user"))
UploadComplianceDocTemplate(name = "myNewTemplateFromSections", sections = sections)
```

You can then get and download your template:

```{r results = "asis", message = FALSE, warning = FALSE, eval = FALSE}
myTemplate <- ListComplianceDocTemplates(namePart = "myNewTemplateFromSections")[[1]]
DownloadComplianceDocTemplate(myTemplate)
```

#### Creating Custom Compliance Documentation from Custom Template

Once you have a custom template made, you can use it to create custom compliance documentation:

```{r results = "asis", message = FALSE, warning = FALSE, eval = FALSE}
myTemplate <- ListComplianceDocTemplates(namePart = "myNewTemplate")[[1]]
CreateComplianceDocumentation(model, myTemplate)
```

#### Keyword Tags

Custom keyword tags are supported for templates, embedding DataRobot generated content into the template. Each tag looks like {{ keyword }} and on generation will be replaced with corresponding content. Below you can find a table of currently supported tags:

```
+--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
| Tag                            | Type           | Content                                              | Web Application UI Analog                                      |
+--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
| {{ blueprint_diagram }}        | Image          | Graphical representation of the modeling pipeline.   | Leaderboard >> Model >> Describe >> Blueprint                  |
+--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
| {{ alternative_models }}       | Table          | Comparison of the model with alternatives            | Leaderboard                                                    |
|                                |                | built in the same project.                           |                                                                |
|                                |                | Also known as challenger models.                     |                                                                |
+--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
| {{ model_features }}           | Table          | Description of the model features                    | Data >> Project Data                                           |
|                                |                | and corresponding EDA statistics.                    |                                                                |
+--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
| {{ missing_values }}           | Table          | Description of the missing values and their          | Leaderboard >> Model >> Describe >> Missing Values             |
|                                |                | processing in the model.                             |                                                                |
+--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
| {{ partitioning }}             | Image          | Graphical representation of the data partitioning.   | Data >> Show Advanced Options >> Partitioning                  |
|                                |                |                                                      | (only available before project start)                          |
+--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
| {{ model_scores }}             | Table          | Metric scores of the model on different data sources | Leaderboard >> Model                                           |
+--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
| {{ lift_chart }}               | Image          | Lift chart.                                          | Leaderboard >> Model >> Evaluate >> Lift Chart                 |
+--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
| {{ feature_impact }}           | Image          | Feature Impact chart.                                | Leaderboard >> Model >> Understand >> Feature Impact           |
+--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
| {{ feature_impact_table }}     | Table          | Table representation of Feature Impact data.         | Leaderboard >> Model >> Understand >> Feature Impact >> Export |
+--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
| {{ feature_effects }}          | List of images | Feature Effects charts for the top 3 features.       | Leaderboard >> Model >> Understand >> Feature Effects          |
+--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
| {{ accuracy_over_time }}       | Image          | Accuracy over time chart.                            | Leaderboard >> Model >> Evaluate >> Accuracy Over Time         |
|                                |                | Available only for datetime partitioned projects.    |                                                                |
+--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
| {{ cv_scores }}                | Table          | Project metric scores for each fold.                 | Currently unavailable in the UI                                |
|                                |                | Available only for projects with cross validation.   |                                                                |
+--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
| {{ roc_curve }}                |                | ROC Curve.                                           | Leaderboard >> Model >> Evaluate >> ROC Curve                  |
|                                | Image          | Available only for binary classification projects.   |                                                                |
+--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
| {{ confusion_matrix_summary }} | Table          | Confusion matrix summary for the threshold with      | Leaderboard >> Model >> Evaluate >> ROC Curve                  |
|                                |                | maximal F1 score value (default suggestion in UI).   |                                                                |
|                                |                | Available only for binary classification projects.   |                                                                |
+--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
| {{ prediction_distribution }}  | Image          | Prediction distribution.                             | Leaderboard >> Model >> Evaluate >> ROC Curve                  |
|                                |                | Available only for binary classification projects.   |                                                                |
+--------------------------------+----------------+------------------------------------------------------+----------------------------------------------------------------+
```