--- title: "Basics" author: "John D Harrison" date: "`r Sys.Date()`" output: rmarkdown::html_vignette: toc: true toc_depth: 3 vignette: > %\VignetteIndexEntry{Basics} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- The goal of this vignette is to describe the basic functionality of the `binman` package. ## Introduction `binman` (Binary Manager) is an R package that allows the user to manage the downloading of third party binaries. The downloading of binaries is split into three parts: the pre-download, the download and the post-download. Each part of the download is controlled by an appropriate function: **predlfunction**, **dlfunction** and **postdlfunction** respectively. ### Pre-Download Function The pre-download function should return up-to its final point a named list of data.frames. The name of each data.frame corresponds to a "platform". Each individual data.frame should be composed of three columns namely "version", "url" and "file". This named list of data.frames can then be passed to the `assign_directory` function. ```R mypredl_fun <- function(...) { # code to return named list of data.frames assign_directory(mynamedlist, "myappname") } ``` #### How directories are assigned Directories are assigned using the `rappdirs` package. See `rappdirs::user_data_dir` for more OS specific information. ### Download Function The download function should take as its primary input the output given by `assign_directory` that is a named list of data.frames with column names "version", "url", "file", "dir" and "exists". The download function should process the list downloading the "url" to the given directory "dir" assigning it the file-name "file". Whether the download is attempted may be dictated on the value of the "exists" variable (For example an overwrite argument in the download function). The download function should return a single data.frame with columns "platform", "file" and "processed". The "platform" column indicates what platform the file downloaded relates to. The "file" column gives the full path to the downloaded file. The "processed" column indicates whether the file was downloaded/processed. ```R mydl_fun <- function(dllist, ...) { # dllist is output from a predlfunction with final output from assign_directory # CODE here to perform download data.frame( platform = platform, file = file, processed = processed, stringsAsFactors = FALSE ) } ``` ### Post-Download Function The post-download function takes as its primary input the output given by a "download function" that is a data.frame with column names "platform", "file" and "processed". The purpose of the post-download function is to process the downloaded files after the download. This could involve changing the file properties such as the mode, unzip/untar a downloaded zip/tar file etc. ### Application YAML file The three functions mentioned previously are combined together to define a process for downloading the binary file. The user stipulates the required functions and arguments by way of a YAML format file. For example: ```yaml name: superduperapp predlfunction: "superduperapp::predl_superduper": appname: "superduperapp" dlfunction: "superduperapp::download_superduper": [] postdlfunction: "superduperapp::postdl_superduper": [] ``` The defined YAML file is then processed by the `process_yaml` function. The YAML file needs to define the three required functions with any arguments and also define a name. ## Browser Mob Proxy example The following is an example of using binman to get the github assets from a project. The project is https://github.com/lightbody/browsermob-proxy/releases . When a new version is released a zipped binary is added as an "asset". A JSON representation of the project releases is available at https://api.github.com/repos/lightbody/browsermob-proxy/releases. We shall breakdown the process into its three functional parts. ### BMP Pre-Download Firstly we note we have JSON data so we parse the JSON from the URL: ```R bmpURL <- "https://api.github.com/repos/lightbody/browsermob-proxy/releases" bmpData <- jsonlite::fromJSON(bmpURL) ``` This gives us a list of 10 releases. A GitHub release should have a version number. ```R version <- bmpData[["tag_name"]] version ``` ``` [1] "browsermob-proxy-2.1.2" "browsermob-proxy-2.1.1" [3] "browsermob-proxy-2.1.0" "browsermob-proxy-2.1.0-beta-6" [5] "browsermob-proxy-2.1.0-beta-5" "browsermob-proxy-2.1.0-beta-4" [7] "browsermob-proxy-2.1.0-beta-3" "browsermob-proxy-2.1.0-beta-2" [9] "browsermob-proxy-2.1.0-beta-1" "browsermob-proxy-2.0.0" ``` This is our version data but we may want to tidy it up so: ```R versionregex <- c("browsermob-proxy-(.*)$", "\\1") version <- gsub(versionregex[1], versionregex[2], version) version ``` ``` [1] "2.1.2" "2.1.1" "2.1.0" "2.1.0-beta-6" "2.1.0-beta-5" [6] "2.1.0-beta-4" "2.1.0-beta-3" "2.1.0-beta-2" "2.1.0-beta-1" "2.0.0" ``` Each release has "assets" associated with it (a zip file). ```R assets <- bmpData[["assets"]] length(assets) ``` ``` [1] 10 ``` The assets will contain the other information we need for our pre-download function. From each assets item we would like the file-name: ```R bmpFiles <- vapply(assets, "[[", character(1), "name") bmpFiles ``` ``` [1] "browsermob-proxy-2.1.2-bin.zip" "browsermob-proxy-2.1.1-bin.zip" [3] "browsermob-proxy-2.1.0-bin.zip" "browsermob-proxy-2.1.0-beta-6-bin.zip" [5] "browsermob-proxy-2.1.0-beta-5-bin.zip" "browsermob-proxy-2.1.0-beta-4-bin.zip" [7] "browsermob-proxy-2.1.0-beta-3-bin.zip" "browsermob-proxy-2.1.0-beta-2-bin.zip" [9] "browsermob-proxy-2.1.0-beta-1-bin.zip" "browsermob-proxy-2.0.0-bin.zip" ``` and the URL for that file: ```R bmpURLs <- vapply(assets, "[[", character(1), "browser_download_url") bmpURLs ``` ``` [1] "https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.2/browsermob-proxy-2.1.2-bin.zip" [2] "https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.1/browsermob-proxy-2.1.1-bin.zip" [3] "https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.0/browsermob-proxy-2.1.0-bin.zip" [4] "https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.0-beta-6/browsermob-proxy-2.1.0-beta-6-bin.zip" [5] "https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.0-beta-5/browsermob-proxy-2.1.0-beta-5-bin.zip" [6] "https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.0-beta-4/browsermob-proxy-2.1.0-beta-4-bin.zip" [7] "https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.0-beta-3/browsermob-proxy-2.1.0-beta-3-bin.zip" [8] "https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.0-beta-2/browsermob-proxy-2.1.0-beta-2-bin.zip" [9] "https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.0-beta-1/browsermob-proxy-2.1.0-beta-1-bin.zip" [10] "https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.0.0/browsermob-proxy-2.0.0-bin.zip" ``` In the case of BMP we have a single platform which we could denote "generic". So we could pass the following: ```R dllist <- list( "generic" = data.frame( version = version, url = bmpURLs, file = bmpFiles, stringsAsFactors = FALSE ) ) ``` Then finally we pass this list to the `assign_directory` function: ```R predlOut <- assign_directory(dllist, "bmpApp") str(predlOut, max.level = 2) ``` ``` List of 1 $ generic:'data.frame': 10 obs. of 5 variables: ..$ version: chr [1:10] "2.1.2" "2.1.1" "2.1.0" "2.1.0-beta-6" ... ..$ url : chr [1:10] "https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.2/browsermob-proxy-2.1.2-bin.zip" "https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.1/browsermob-proxy-2.1.1-bin.zip" "https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.0/browsermob-proxy-2.1.0-bin.zip" "https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.0-beta-6/browsermob-proxy-2.1.0-beta-6-bin"| __truncated__ ... ..$ file : chr [1:10] "browsermob-proxy-2.1.2-bin.zip" "browsermob-proxy-2.1.1-bin.zip" "browsermob-proxy-2.1.0-bin.zip" "browsermob-proxy-2.1.0-beta-6-bin.zip" ... ..$ dir :List of 10 ..$ exists : logi [1:10] FALSE FALSE FALSE FALSE FALSE FALSE ... ``` In the `binman` package there is a function template for github asset type downloads: ```R dllist <- predl_github_assets( url = "https://api.github.com/repos/lightbody/browsermob-proxy/releases", history = 3L, platform = "generic", appname = "bmproxy", platformregex = "browsermob-proxy", versionregex = c("browsermob-proxy-(.*)$", "\\1") ) ``` There are some extra arguments for generality but it basically performs the operations we outlined above. ### BMP Download The download for BMP is straightforward: ```R dllist[["generic"]][1,] ``` ``` version 1 2.1.2 url 1 https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.2/browsermob-proxy-2.1.2-bin.zip file dir 1 browsermob-proxy-2.1.2-bin.zip /home/john/.local/share/binman_bmproxy/generic/2.1.2 exists 1 FALSE ``` For each platform (there is only one "generic" platform in this case) we want to loop over the data.frame downloading the data at "url" to the "file" in the "dir". If it already "exists" we may not perform the download. In `binman` there is a simple template function for this: ```R dlfiles <- download_files(dllist) ``` So we simply need to pass the list from the pre download function to this download function. ### BMP Post Download The files we receive from the Browser mob Proxy project are zipped. As a post processing operation we would like to unzip these files. The result of unzipping the files is a directory structure. In this case we do not change the mode of the ultimate binary which is contained in this directory structure. Again `binman` has a simple template function to unzip/untar a downloaded file so we pass our output from the download to this: ```R dlres <- unziptar_dlfiles(dlfiles) ``` ### BMP YAML We can incorporate our three functional calls into a YAML file for the Browser Mob Proxy application as follows: ```yaml name: bmproxy predlfunction: "binman::predl_github_assets": url: "https://api.github.com/repos/lightbody/browsermob-proxy/releases" platform: - generic history: 3 appname: "bmproxy" platformregex: - "browsermob-proxy" versionregex: - "browsermob-proxy-(.*)$" - "\\1" dlfunction: "binman::download_files": [] postdlfunction: "binman::unziptar_dlfiles": [] ``` We have saved this file in the `binman` package: ```R ymlfile <- system.file("examples", "yaml", "bmproxy.yml", package = "binman") ``` We can now use the `process-yaml` function to download the BMP binaries: ```R process_yaml(ymlfile) ``` ``` BEGIN: PREDOWNLOAD BEGIN: DOWNLOAD Creating directory: /home/john/.local/share/binman_bmproxy/generic/2.1.2 Downloading binary: https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.2/browsermob-proxy-2.1.2-... Creating directory: /home/john/.local/share/binman_bmproxy/generic/2.1.1 Downloading binary: https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.1/browsermob-proxy-2.1.1-... Creating directory: /home/john/.local/share/binman_bmproxy/generic/2.1.0 Downloading binary: https://github.com/lightbody/browsermob-proxy/releases/download/browsermob-proxy-2.1.0/browsermob-proxy-2.1.0-... BEGIN: POSTDOWNLOAD ``` Looking at the resulting directory structure (this is for Linux the location of directories will be OS dependent): ```sh tree -d /home/john/.local/share/binman_bmproxy ``` ``` /home/john/.local/share/binman_bmproxy └── generic ├── 2.1.0 │ └── browsermob-proxy-2.1.0 │ ├── bin │ │ └── conf │ ├── lib │ └── ssl-support ├── 2.1.1 │ └── browsermob-proxy-2.1.1 │ ├── bin │ │ └── conf │ ├── lib │ └── ssl-support └── 2.1.2 └── browsermob-proxy-2.1.2 ├── bin │ └── conf ├── lib └── ssl-support ``` We see that `binman` has downloaded and unzipped the binaries to appropriate directories.