‘fedregs’: Text Analysis of the US Code of Federal Regulations
The goal of fedregs is to allow for easy exploration and analysis of the Code of Federal Regulation.
Installation
You can install fedregs using:
install.packages("fedregs") # Or: devtools::install_github("NOAA-EDAB/fedregs")
Example
The Code of Federal Regulation is organized according to a consistent hierarchy: title, chapter, part, subpart, section, and subsection. Each title within the CFR is (somewhat haphazardly) divided into volumes and over time each chapter isn’t consistently in the same volume. The cfr_text() function is the main function in the package and it will return the text for a specified part, including the associated subparts and sections. Behind the scenes, cfr_text() and associated helper functions gather the volumes for a given title/year combination and parses XML to determine the chapters, parts, and subparts associated with each volume. Next, the text is extracted for each subpart. The return_tidytext = TRUE argument will return a tibble with the text in a tidytext format. If ngrams are your game, set token = "ngrams" and specify n.
library(fedregs) library(dplyr) library(tidyr) library(ggplot2) library(quanteda) regs <- cfr_text(year = 2017, title_number = 50, chapter = 6, part = 648, #token = "ngrams", # uncomment for ngrams of length 2 #n = 2, # uncomment for ngrams of length 2 return_tidytext = TRUE, verbose = FALSE) head(regs) ## # A tibble: 6 x 6 ## # Groups: subpart, year, title_number, chapter, part [6] ## subpart year title_number chapter part data ## <chr> <dbl> <dbl> <chr> <dbl> <list> ## 1 Subpart A—General Provisions 2017 50 VI 648 <tibble [~ ## 2 Subpart B—Management Measures for~ 2017 50 VI 648 <tibble [~ ## 3 Subpart C—Management Measures for~ 2017 50 VI 648 <tibble [~ ## 4 Subpart D—Management Measures for~ 2017 50 VI 648 <tibble [~ ## 5 Subpart E—Management Measures for~ 2017 50 VI 648 <tibble [~ ## 6 Subpart F—Management Measures for~ 2017 50 VI 648 <tibble [~
Now, we can unnest the tibble and take a peek at the data to see what data we have to play with.
regs %>% unnest(cols = c(data)) %>% head(20) %>% pull(word) ## [1] "c" "a" "this" "part" "implements" ## [6] "the" "fishery" "management" "plans" "fmps" ## [11] "for" "the" "atlantic" "mackerel" "squid" ## [16] "and" "butterfish" "fisheries" "atlantic" "mackerel"
Not entirely unexpected, but there are quite a few common words that don’t mean anything. These “stop words” typically don’t have important significance and and are filtered out from search queries.
There are some other messes like punctuation, numbers, iths, Roman Numerals, web sites, and random letters (probably from indexed lists) that can be removed with some simple regex-ing. We can also convert the raw words to word stems to further aggregate our data.
stop_words <- tibble(word = stopwords("english")) clean_words <- regs %>% unnest(cols = c(data)) %>% mutate(word = gsub("[[:punct:]]", "", word), # remove any remaining punctuation word = gsub("^[[:digit:]]*", "", word)) %>% # remove digits (e.g., 1st, 1881a, 15th, etc) anti_join(stop_words, by = "word") %>% # remove "stop words" filter(is.na(as.numeric(word)), !grepl("^m{0,4}(cm|cd|d?c{0,3})(xc|xl|l?x{0,3})(ix|iv|v?i{0,3})$", word), # adios Roman Numerals !grepl("\\b[a-z]{1}\\b", word), # get rid of one letter words !grepl("\\bwww*.", word)) %>% # get rid of web addresses mutate(word = tokens(word), word = as.character(tokens_wordstem(word))) head(clean_words) ## # A tibble: 6 x 9 ## # Groups: subpart, year, title_number, chapter, part [1] ## subpart year title_number chapter part SECTION_NAME SECTION_NUMBER values ## <chr> <dbl> <dbl> <chr> <dbl> <chr> <chr> <chr> ## 1 Subpar~ 2017 50 VI 648 Purpose and~ §<U+2009>648.1 648.1 ## 2 Subpar~ 2017 50 VI 648 Purpose and~ §<U+2009>648.1 648.1 ## 3 Subpar~ 2017 50 VI 648 Purpose and~ §<U+2009>648.1 648.1 ## 4 Subpar~ 2017 50 VI 648 Purpose and~ §<U+2009>648.1 648.1 ## 5 Subpar~ 2017 50 VI 648 Purpose and~ §<U+2009>648.1 648.1 ## 6 Subpar~ 2017 50 VI 648 Purpose and~ §<U+2009>648.1 648.1 ## # ... with 1 more variable: word <chr>
Now we can look at binning and plotting the words
count_words <- clean_words %>% group_by(word) %>% summarise(n = n()) %>% ungroup() %>% arrange(-n) %>% top_n(n = 50, wt = n) %>% mutate(word = reorder(word, n))
ggplot(count_words, aes(word, n)) + geom_col() + labs(xlab = NULL, title = "Code of Federal Regulations", subtitle = "Title 50, Chapter VI, Part 648", caption = sprintf("Data accessed on %s from:\n https://www.gpo.gov/fdsys/browse/collectionCfr.action?collectionCode=CFR", format(Sys.Date(), "%d %B %Y"))) + theme(axis.text.x = element_text(angle = 45, hjust = 1), legend.direction = "horizontal", legend.position = "bottom", text = element_text(size = 8)) + coord_flip() + theme_minimal()
This repository is a scientific product and is not official communication of the National Oceanic and Atmospheric Administration, or the United States Department of Commerce. All NOAA GitHub project code is provided on an ‘as is’ basis and the user assumes responsibility for its use. Any claims against the Department of Commerce or Department of Commerce bureaus stemming from the use of this GitHub project will be governed by all applicable Federal law. Any reference to specific commercial products, processes, or services by service mark, trademark, manufacturer, or otherwise, does not constitute or imply their endorsement, recommendation or favoring by the Department of Commerce. The Department of Commerce seal and logo, or the seal and logo of a DOC bureau, shall not be used in any manner to imply endorsement of any commercial product or activity by DOC or the United States Government.
