--- title: "Summarize and secure harmonized outputs" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Summarize and secure harmonized outputs} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r, include = FALSE} knitr::opts_chunk$set( collapse = TRUE, comment = "#>" ) ``` Once you have processed inputs and generated harmonized outputs, you can examine the harmonized datasets and extract different outputs. This vignette shows you how to access and use harmonized outputs from the harmonized dossier. ## Load packages ```{r eval=FALSE} # Load relevant packages library(Rmonize) library(fabR) # Additional functions to help with importing data library(tidyverse) # Collection of R packages for data science ``` ## Get the harmonized dossier The harmonized dossier produced by `harmo_process()` is a list containing the harmonized datasets and associated metadata (including the DataSchema, DPEs, and harmonized data dictionaries), which can be further examined and summarized. We will use a harmonized dossier that is provided with the examples in the packages, which includes harmonized outputs from five study datasets (and is the equivalent of the harmonized dossier generated in the vignette Process inputs and generate harmonized outputs). ```{r eval=FALSE} # Load harmonized dossier harmonized_dossier <- Rmonize_examples$harmonized_dossier # See information about the harmonized dossier in the RStudio viewer View(harmonized_dossier) # See additional attributes associated with the harmonized dossier summary(attributes(harmonized_dossier)) ``` ```{r,fig.cap="Information about the example harmonized dossier provided in the RStudio viewer.", out.width="80%", fig.align="center",echo=FALSE} knitr::include_graphics("images/vig5_fig01.png") ``` ```{r,fig.cap="Additional Rmonize attributes are also associated with the harmonized dossier.", out.width="80%", fig.align="center",echo=FALSE} knitr::include_graphics("images/vig5_fig02.png") ``` ## Summarize harmonized datasets You can get more detailed summaries and descriptive statistics about the harmonized datasets and variables. By default, summaries of a harmonized dossier are generated for pooled data and grouped by dataset. The output of dataset summaries can easily be exported as Excel files. ```{r eval=FALSE} # Produce dataset and variable summaries summary_report_harmonized_dossier <- harmonized_dossier_summarize(harmonized_dossier) # Note: This report is also provided in Rmonize_examples$summary_report_harmonized_dossier. # Export to Excel # WARNING: This script creates a folder 'tmp'. output_path <- paste0('tmp/',basename(tempdir())) dir.create(output_path) write_excel_allsheets( summary_report_harmonized_dossier, paste0(output_path,"/summary_report_harmonized_dossier.xlsx")) ``` ```{r,fig.cap="Information about the example harmonized dossier summary report provided in the RStudio viewer.", out.width="80%", fig.align="center",echo=FALSE} knitr::include_graphics("images/vig5_fig03.png") ``` You can get visual summaries of the harmonized datasets and variables, which by default are produced for the pooled data and grouped by dataset. A categorical variable 'Rmonize::harmonized_col_dataset' is created based on the dataset names and included in the report. ```{r eval=FALSE} # Produce a visual report of the harmonized datasets and variables # You must specify a folder to contain the visual report files, and the # folder name must not already exist. # WARNING: This script creates a folder 'tmp'. bookdown_path <- paste0('tmp/',basename(tempdir())) if(dir.exists(bookdown_path)) file.remove(bookdown_path) harmonized_dossier_visualize( harmonized_dossier, bookdown_path = bookdown_path, harmonized_dossier_summary = summary_report_harmonized_dossier ) # Open the visual report in a browser. bookdown_open(bookdown_path) ``` To open the visual report outside of R, go to the folder containing the bookdown files, open the folder 'docs', and open the file 'index'. *Screenshots from the interactive html visual report of the harmonized dossier:* ```{r, fig.cap="a) the Overview page", out.width="80%", fig.align="center",echo=FALSE} knitr::include_graphics("images/vig5_fig04a.png") ``` ```{r, fig.cap="b) summary figure for the variable lsb_alc_binge_m_preg.", out.width="80%", fig.align="center",echo=FALSE} knitr::include_graphics("images/vig5_fig04b.png") ``` While there should be no technical data processing errors at this point, the harmonized datasets should always be validated. Data summaries and visual reports can help further verify that there were no remaining issues in data processing or data content. You can also use the reports to compare distributions for the pooled dataset and individual datasets. ## Extract harmonized data dictionary A harmonized data dictionary with useful metadata about the datasets and variables can be extracted from the harmonized dossier. ```{r eval=FALSE} # Extract the data dictionary data_dict_harmonized_dossier <- harmonized_dossier %>% lapply(data_dict_extract) ``` > **INFO:** The function `data_dict_extract()` will create a data dictionary from metadata associated with any dataset. From a harmonized dataset generated from `harmo_process()`, the data dictionary will include all fields from the DataSchema, certain fields from the DPE, and the R script used to generate the variable for each dataset. ## Create pooled harmonized dataset The harmonized dossier contains individual harmonized datasets as named data frames contained in a list. They can be explicitly combined into a pooled harmonized dataset, which includes much of the associated metadata. ```{r eval=FALSE} # Generate one pooled harmonized dataset from a harmonized dossier pooled_harmonized_dataset <- pooled_harmonized_dataset_create( harmonized_dossier = harmonized_dossier) ``` > **TIP:** The parameter 'harmonized_col_dataset' can be used to indicate the variable that identifies each dataset, if such a variable is available (e.g., in this example, the values in the column adm_study_id identify each dataset). If 'harmonized_col_dataset' is empty, a new column will be created in the pooled dataset using the dataset names from the harmonized dossier as values, and a warning will be printed in the console. This is to ensure that source datasets are indicated if needed in further data analysis, and that participants (each row of the pooled dataset) can always be uniquely identified in case there is any overlap in IDs. It is good practice to include a categorical variable in the DataSchema that identifies the dataset. ## Export harmonized datasets to other formats You can work with the harmonized datasets in R or export them into other data formats. Data dictionaries can be exported as Excel files. ```{r eval=FALSE} # Export harmonized datasets without metadata to Excel # WARNING: This script creates a folder 'tmp'. output_path <- paste0('tmp/',basename(tempdir())) dir.create(output_path) for(i in names(harmonized_dossier)){ filename <- paste0(output_path,"/outputs/harmonized_datasets/harmonized_",i,".xlsx") write_excel_allsheets(harmonized_dossier[[i]],filename)} # Export harmonized data dictionaries to Excel for(i in names(data_dict_harmonized_dossier)){ filename <- paste0(output_path, "/outputs/harmonized_data_dictionaries/harmonized_data_dictionary_",i,".xlsx") write_excel_allsheets(data_dict_harmonized_dossier[[i]],filename)} ``` Once harmonized outputs are produced, they can be explored with any tools and preserved in any file formats you would typically use in R.