Unit Tests

Ron Klasky

2024-06-05

Source: vignettes/UnitTests.Rmd

Source: vignettes/UnitTests.Rmd

Unit Tests

Unit Tests are written using the testthat R package. The tests are stored in the file: tests/tests_rpath.R. They may be run explicitly by the user clicking the Test or Check buttons under the Build tab in RStudio. They also run automatically via git actions when the user does a push or pull request. The output of the unit tests is a pdf file with a variety of comparison plots, examples of which are given below in Figures 1-3.

Details

Unit Tests consist of comparing a “current” model value (i.e., string, table) to a previously stored “baseline” value. Differences are shown in the testing output as well as reflected in the generated plots which are written to: Rplots.pdf. Difference “tolerances” are defined by the variable:

TOLERANCE <- 1e-5

which can be modified as necessary.

Regenerating Baseline Files

If the user needs to regenerate the baseline files, the following variable should be set to TRUE and the tests re-run manually (by clicking the Test button under the Build tab). This will regenerate all of the baseline data files.

CREATE_BASELINE_FILES <- TRUE

After the baseline files have been generated, the user should reset the variable back:

CREATE_BASELINE_FILES <- FALSE

List of Tests

The following unit tests have been developed. Note that some tests consist of additional/finer tests that compare out_Biomass, out_Catch, and out_Gear_Catch* for AB (Adams-Bashforth) and RK4 (4th Order Runge Kutta) numerical differential equation methods.

Unit Tests

Is model balanced?
  1. Looks for “Balanced” string in Rpath summary file output
Does model run silent?
  1. Tests if model runs without producing print statements, warnings, or error messages.
Compare Rpath summaries
  1. Is the baseline Rpath object the same as the current Rpath object?
Compare Rpath summaries
  1. Is the baseline Rpath summary the same as the current Rpath summary?
Compare baseline AB sim run to current AB sim run
  1. Checks if baseline Rpath AB sim run is same as current AB sim run (checks top level data structure only)
  2. Compare out_Biomass from baseline and current AB sim runs
  3. Compare out_Catch from baseline and current AB sim runs
  4. Compare out_Gear_Catch from baseline and current AB sim runs
Compare baseline RK4 sim run to current RK4 sim run
  1. Checks if baseline RK4 sim run same as current RK4 sim run (checks top level data structure only)
  2. Compare out_Biomass from baseline and current RK4 sim runs
  3. Compare out_Catch from baseline and current RK4 sim runs
  4. Compare out_Gear_Catch from baseline and current RK4 sim runs
Compare baseline AB sim run to current RK4 sim run
  1. Checks if baseline AB sim run same as current RK4 sim run (checks top level data structure only)
  2. Compare out_Biomass from baseline AB run and current RK4 sim run
  3. Compare out_Catch from baseline AB run and current RK4 sim run
  4. Compare out_Gear_Catch from baseline AB run and current RK4 sim run
Compare sim runs with random noise (i.e., jitter) added to $forcing$ForcedBio
  1. Compare out_Biomass from baseline AB run and current AB run with noise added to ForcedBio
  2. Compare out_Catch from baseline AB run and current AB run with noise added to ForcedBio
  3. Compare out_Gear_Catch from baseline AB run and current AB run with noise added to ForcedBio
  4. Compare out_Biomass from baseline RK4 run and current RK4 run with noise added to ForcedBio
  5. Compare out_Catch from baseline RK4 run and current RK4 run with noise added to ForcedBio
  6. Compare out_Gear_Catch from baseline RK4 run and current RK4 run with noise added to ForcedBio
Compare sim runs with random noise (i.e., jitter) added to $forcing$ForcedMigrate
  1. Compare out_Biomass from baseline AB run and current AB run with noise added to ForcedMigrate
  2. Compare out_Catch from baseline AB run and current AB run with noise added to ForcedMigrate
  3. Compare out_Gear_Catch from baseline AB run and current AB run with noise added to ForcedMigrate
  4. Compare out_Biomass from baseline RK4 run and current RK4 run with noise added to ForcedMigrate
  5. Compare out_Catch from baseline RK4 run and current RK4 run with noise added to ForcedMigrate
  6. Compare out_Gear_Catch from baseline RK4 run and current RK4 run with noise added to ForcedMigrate
Compare sim runs with stepped noise added to $forcing$ForcedBio
  1. Compare out_Biomass from baseline AB run and current AB run with stepped noise added to ForcedBio
  2. Compare out_Catch from baseline AB run and current AB run with stepped noise added to ForcedBio
  3. Compare out_Gear_Catch from baseline AB run and current AB run with stepped noise added to ForcedBio
  4. Compare out_Biomass from baseline RK4 run and current RK4 run with stepped noise added to ForcedBio
  5. Compare out_Catch from baseline RK4 run and current RK4 run with stepped noise added to ForcedBio
  6. Compare out_Gear_Catch from baseline RK4 run and current RK4 run with stepped noise added to ForcedBio
Compare sim runs with stepped noise added to $forcing$ForcedMigrate
  1. Compare out_Biomass from baseline AB run and current AB run with stepped noise added to ForcedMigrate
  2. Compare out_Catch from baseline AB run and current AB run with stepped noise added to ForcedMigrate
  3. Compare out_Gear_Catch from baseline AB run and current AB run with stepped noise added to ForcedMigrate
  4. Compare out_Biomass from baseline RK4 run and current RK4 run with stepped noise added to ForcedMigrate
  5. Compare out_Catch from baseline RK4 run and current RK4 run with stepped noise added to ForcedMigrate
  6. Compare out_Gear_Catch from baseline RK4 run and current RK4 run with stepped noise added to ForcedMigrate
Compare sim runs with random noise (i.e., jitter) added to $fishing$ForcedEffort
  1. Compare out_Biomass from baseline AB run and current AB run with noise added to ForcedEffort
  2. Compare out_Catch from baseline AB run and current AB run with noise added to ForcedEffort
  3. Compare out_Gear_Catch from baseline AB run and current AB run with noise added to ForcedEffort
  4. Compare out_Biomass from baseline RK4 run and current RK4 run with noise added to ForcedEffort
  5. Compare out_Catch from baseline RK4 run and current RK4 run with noise added to ForcedEffort
  6. Compare out_Gear_Catch from baseline RK4 run and current RK4 run with noise added to ForcedEffort
Compare sim runs with random noise (i.e., jitter) added to $fishing$ForcedFRate
  1. Compare out_Biomass from baseline AB run and current AB run with noise added to ForcedFRate
  2. Compare out_Catch from baseline AB run and current AB run with noise added to ForcedFRate
  3. Compare out_Gear_Catch from baseline AB run and current AB run with noise added to ForcedFRate
  4. Compare out_Biomass from baseline RK4 run and current RK4 run with noise added to ForcedFRate
  5. Compare out_Catch from baseline RK4 run and current RK4 run with noise added to ForcedFRate
  6. Compare out_Gear_Catch from baseline RK4 run and current RK4 run with noise added to ForcedFRate
Compare sim runs with random noise (i.e., jitter) added to $fishing$ForcedCatch
  1. Compare out_Biomass from baseline AB run and current AB run with noise added to ForcedCatch
  2. Compare out_Catch from baseline AB run and current AB run with noise added to ForcedCatch
  3. Compare out_Gear_Catch from baseline AB run and current AB run with noise added to ForcedCatch
  4. Compare out_Biomass from baseline RK4 run and current RK4 run with noise added to ForcedCatch
  5. Compare out_Catch from baseline RK4 run and current RK4 run with noise added to ForcedCatch
  6. Compare out_Gear_Catch from baseline RK4 run and current RK4 run with noise added to ForcedCatch
Compare sim runs with stepped noise added to $fishing$ForcedEffort
  1. Compare out_Biomass from baseline AB run and current AB run with stepped noise added to ForcedEffort
  2. Compare out_Catch from baseline AB run and current AB run with stepped noise added to ForcedEffort
  3. Compare out_Gear_Catch from baseline AB run and current AB run with stepped noise added to ForcedEffort
  4. Compare out_Biomass from baseline RK4 run and current RK4 run with stepped noise added to ForcedEffort
  5. Compare out_Catch from baseline RK4 run and current RK4 run with stepped noise added to ForcedEffort
  6. Compare out_Gear_Catch from baseline RK4 run and current RK4 run with stepped noise added to ForcedEffort
Compare sim runs with stepped noise added to $fishing$ForcedFRate
  1. Compare out_Biomass from baseline AB run and current AB run with stepped noise added to ForcedFRate
  2. Compare out_Catch from baseline AB run and current AB run with stepped noise added to ForcedFRate
  3. Compare out_Gear_Catch from baseline AB run and current AB run with stepped noise added to ForcedFRate
  4. Compare out_Biomass from baseline RK4 run and current RK4 run with stepped noise added to ForcedFRate
  5. Compare out_Catch from baseline RK4 run and current RK4 run with stepped noise added to ForcedFRate
  6. Compare out_Gear_Catch from baseline RK4 run and current RK4 run with stepped noise added to ForcedFRate
Compare sim runs with stepped noise added to $fishing$ForcedCatch
  1. Compare out_Biomass from baseline AB run and current AB run with stepped noise added to ForcedCatch
  2. Compare out_Catch from baseline AB run and current AB run with stepped noise added to ForcedCatch
  3. Compare out_Gear_Catch from baseline AB run and current AB run with stepped noise added to ForcedCatch
  4. Compare out_Biomass from baseline RK4 run and current RK4 run with stepped noise added to ForcedCatch
  5. Compare out_Catch from baseline RK4 run and current RK4 run with stepped noise added to ForcedCatch
  6. Compare out_Gear_Catch from baseline RK4 run and current RK4 run with stepped noise added to ForcedCatch


*The out_Gear_Catch tests are currently awaiting issue #74 to be resolved.

Figures

The Unit Tests compare current runs with baseline runs and produce plots in a pdf file. These plots may be useful for diagnostic purposes. The plots depict the output of a current run in blue and of the baseline run in red. If there are no errors, then there should be no red visible on the plots. There are two types of plots available, also set by a variable in tests/tests_rpath.R:

PLOT_TYPE <- 1

If PLOT_TYPE is set to 1, then the plot depicts the current model in blue superimposed over the baseline model in red. If PLOT_TYPE is set to 2, then the difference (current-baseline) is displayed.

Example plots:

Figure 1 depicts an Rpath Simulation run of Biomass Density vs Time for 3 species. The REcosystem_scene$forcing$ForcedBio table, where

  REcosystem_scene <- rsim.scenario(REco, REco.params, 1:50)

has been modified with random noise. Both the baseline and current model runs use the AB (Adams-Bashforth) numerical differential equation method. Notice that there is no red line visible in this plot. That is, the current (blue) line is identical (and covers completely) the baseline (red) plot.

Figure 1. No Difference in Baseline vs Current Models

Figure 1. No Difference in Baseline vs Current Models



Figure 2 depicts the same information as Figure 1 with the difference being that the randomization factor was slightly changed in the current model from what was used in the baseline model. This was done to only to highlight how the plot would appear if the current model deviated from the baseline. Notice that the underlying baseline plot (in red) is now clearly visible.

Figure 2. Difference in red of Baseline vs Current Models

Figure 2. Difference in red of Baseline vs Current Models



Figure 3 shows the difference of the current and baseline models. If the models were identical, the difference would be 0, and the plots should be completely flat lines. This plot shows the current-baseline plots for the data shown in Figure 2. Notice that the plot lines are not completely flat, denoting that the difference is not 0. The scale of the y-axis may be modified by changing the following line in tests/tests_rpath.R:

YLIMIT_DIFFERENCE_PLOTS <- 0.05
Figure 3. Current-Baseline Result

Figure 3. Current-Baseline Result