--- title: "Overview of the inverseRegex Package" author: "Jasper Watson" date: "`r Sys.Date()`" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Overview} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r setup, include = FALSE} knitr::opts_chunk$set( collapse = TRUE, comment = "#>" ) ``` ```{r, echo = FALSE} library(inverseRegex) ``` # inverseRegex The inverseRegex package allows users to reverse engineer regular expression patterns for R objects. Individual characters that make up an object are categorised into common groups and encoded into run-lengths. For example, the phrase "Hello World!" can be translated to `"[[:upper:]][[:lower:]]{4} [[:upper:]][[:lower:]]{4}!"`. This could be useful to summarise a dataset without viewing all individual entries or to aid in data cleaning. One could check that a column of dates all follow a "nnnn-nn-nn" format or that a column of strings consisted entirely of alphabetic characters (no zeros entered instead of the letter O for example). ## Usage The main function to use is `inverseRegex(x)` which will identify the different characters that make up the input object `x`. The different groups that can be identified are - `'[[:digit:]]'` - `'[[:lower:]]'` - `'[[:upper:]]'` - `'[[:alpha:]]'` - `'[[:alnum:]]'` - `'[[:space:]]'` - `'[[:punct:]]'` See `?regex` for an explanation of their meanings. By default the only groups that will be identified are `[[:digit:]]`, `[[:upper:]]`, and `[[:lower:]]`, with any other characters being left as is. This can altered with the following arguments: - `combineCases`: Use `'[[:alpha:]]'` instead of `'[[:lower:]]'` and `'[[:upper:]]'`. - `combineAlphanumeric`: Use '`[[:alnum:]]`' instead of '`[[:digit:]]`', '`[[:lower:]]`', '`[[:upper:]]`', and '`[[:alpha:]]`'. - `combinePunctuation`: Use '`[[:punct:]]`' instead of leaving punctuation characters as is. - `combineSpace`: Use '`[[:space:]]`' instead of leaving space characters as is. Some examples of these arguments are below: ```{r, echo = TRUE} inverseRegex('1aA') inverseRegex('1aA', combineCases = TRUE) inverseRegex('1aA', combineAlphanumeric = TRUE) inverseRegex('Hello World!') inverseRegex('Hello World!', combineSpace = TRUE, combinePunctuation = TRUE) ``` Users can also specify the different run lengths that will be identified. The `inverseRegex` function has an argument called `numbersToKeep` which allows the user to specify what lengths of repeated sequences should be identified explicitly. The default value is `c(2, 3, 4, 5, 10)`. Run lengths not requested will be identified with a `+`. ```{r, echo = TRUE} inverseRegex('abcd1234567') inverseRegex('abcd1234567', numbersToKeep = NULL) inverseRegex('abcd1234567', numbersToKeep = 1:10) ``` ## Priority The priority argument allows users to specify characters that will take precedence over the regex grouping patterns, or to make them a lower priority than the generic `.` placeholder. For example below the `a` and `1` characters are treated separately to the `[[:lower:]]` and `[[:upper:]]` groups, whilst the `?` is treated as a lower priority than the `.`, unlike the `!`. ```{r, echo = TRUE} inverseRegex('abc123?!', priority = c('a', '1', '!', '[[:lower:]]', '[[:digit:]]', '.')) ``` ## Non-character Inputs Many objects with a class other than `character` are supported, including `logical`, `integer`, `numeric`, `Date`, `POSIXct`, `factor`, `matrix`, `data.frame`, and `tibble`. They are all (except `logical`) converted to characters first and then the collection of regex patterns returned either as character vectors or as the same class as the input object if it was a matrix, data frame, or tibble. See `?inverseRegex` for a full description of how they are treated. If users need a different character conversion method they can do it themselves prior to calling `inverseRegex`. Special mention of numerics and data frames will be given here: ### Inputs of Class `numeric` An attempt has been made to convert numeric values into characters as directly as possible without losing or adding any information. When passed a numeric vector `inverseRegex` will convert it to character using: `vapply(x, format, character(1), nsmall = 1)`. This will force at least one decimal place for all entries but will not add extra decimal places beyond that unless they were present in the individual input element; it will however remove trailing decimal zeros. For example: ```{r, echo = TRUE} vapply(c(1, 1.0, 1.10, 1.12, 1.123), format, character(1), nsmall = 1) inverseRegex(c(1, 1.0, 1.10, 1.12, 1.123), numbersToKeep = 2:10) ## Vectors of class integer are just converted using as.character. inverseRegex(1L) ``` Numerics are treated differently if they are present in a matrix, data frame, or tibble. In the case of a matrix if it has a mode of numeric then the entire object will be converted to character using `trimws(format(x))`. For data frames and tibbles each column of type numeric will be converted using `trimws(format(x))`. This means that unlike for numeric vectors described above, all numeric entries in matrices, data frames, and tibbles will have the same number of decimal places. ```{r, echo = TRUE} inverseRegex(c(1, 1.0, 1.10, 1.12, 1.123)) inverseRegex(data.frame(a = c(1, 1.0, 1.10, 1.12, 1.123))) ``` ### Inputs of Class `data.frame` When giving a data frame `inverseRegex` will return a data frame of similar dimensions with each column representing an individual call to inverseRegex. ```{r, echo = TRUE} unique(inverseRegex(iris, numbersToKeep = 2:10)) ``` ## Identifying Rare Patterns One of the main use cases of the package is to identify irregular entries in a dataset. To this end there is a function `occurrencesLessThan` which will call `inverseRegex` and return logical values with `TRUE` giving the location of any regex patterns that occur less than a certain number of times. ```{r, echo = TRUE} occurrencesLessThan(c(LETTERS, 1)) ## When called on a data frame occurrencesLessThan will assess each column individually. x <- iris x$Species <- as.character(x$Species) x[27, 'Species'] <- 'set0sa' unique(occurrencesLessThan(x)) ``` What constitutes a "rare" pattern can be specified with the `fraction` or `n` arguments. See `?occurrencesLessThan` for a full description.