Model life cycle

(NRC, 2007)

What makes a good model?

• Differs by life stage

• Each builds on the next

• Common themes

Best practices 1: Why do we need a model? Identify the problem

Problem identification ↔ Appropriateness of model for problem

Define the problem, and why this model is appropriate.

• What processes need to be modeled to address the problem?

• What focal species, spatial, and temporal resolution are needed to address the problem?

Problem identification ↔ Appropriateness of model for problem

A different example problem (ICES WKIRISH, Bentley, et al. (2021)):

For the Irish Sea EwE model, key outputs will be used to determine where the catch advice should be within the MSY range for each species. Therefore, outputs defining ecosystem conditions and both ecosystem and species productivity under the prevailing conditions are most important.

Best Practices 2: Is the scientific basis of the model sound?

Previously used models often skip this step, reasoning that if the science was sound previously it still is.

(Discuss...)

ICES WGSAM has provided model framework reviews for the LeMans ensemble (2016), FLBEIA (2017), and a multispecies state-space model (2017). “Constructed Model” evaluation issues and best practices apply here:

1. Spatial and temporal resolution
2. Algorithm choices
3. Assumptions (scientific basis, computational infrastructure; adequacy of conceptual model)
4. Data availability/software tools
5. Quality assurance/quality control (code testing)
6. Test scenarios (for example, stability; Kaplan, et al. (2016)):
   1. All functional groups persist in an unfished unperturbed run
   2. Model stabilizes for the last ~20 years of an unfished, unperturbed 80-100 year run
7. Corroboration with observations
8. Uncertainty/sensitivity analysis
9. Peer review

Note linkages between problem identification and scientific basis

Best Practices 3: Is the input data quality and parameterization sufficient for the problem?

See above defining the problem. Which datasets are adequate, which could be improved, and which are missing?

Show the input data as a simple chart: beginning and end of time series, gaps, different length of time series, spatial resolution of data.

Best Practices 3: Is the input data quality and parameterization sufficient for the problem?

Give information on input data pedigree/quality, reference for where it comes from, whether it is survey data or comes from other model output, whether confidence intervals or other uncertainty measures are available and used in the model.

Categorize the assumptions behind modeled ecological or biological processes. Emphasize those related to species interactions (predation, competition), environmental pressures, and also fleet dynamics if needed to address the problem. If the model is spatial, how do these processes happen in space?

Is the parameterization consistent with scientific knowledge (e.g. (PREBAL) diagnostics (Link, 2010) for general relationships across trophic levels, sizes, etc).

Best Practices 4: Does model output compare well with observations? (Kaplan, et al., 2016)

Characterize the reference dataset used for comparisons. Has the data been used to construct this model? Is the reference dataset from another model? Describe referece data source(s).

1. Define the hindcast time period where agreement with other data sources or assessments is needed. Error ranges of model output and reference datasets are needed for comparison.
2. Focal species should match biomass and catch trends over the hindcast time period. For full system models, species comprising a majority of biomass should also match general hindcast trends. Suggested tests include modeling efficiency, RMSE, etc. (Sterman, 1984; Stow, et al., 2009; Allen, et al., 2009; Lehuta, et al., 2013; Lehuta, et al., 2016; Olsen, et al., 2016)
3. Patterns of temporal variability should be captured as appropriate to the problem (emergent or forced with e.g. recruitment time series)
4. Productivity for focal species (or groups totaling ~80% of system biomass in full system models) should qualitatively match life history expectations (prebal diagnostics)
5. Natural mortality decreases with age for majority of groups
6. Age and length structure qualitatively matches expectations for majority of groups
7. Diet predicted qualitatively matches empirical diet comp for majority of groups
8. Spatial distribution of outputs match reference datasets for spatial models (most important if output required at spatial resolution of model, comment if a match in aggregate but not at higher resolution)
9. Ecosystem indicators (relationship between abundance and body size, pelagic to demersal, Large Fish Indicator) match reference data if needed for problem

Intermission: Why include species interactions? ...and why fit criteria alone are not sufficient

Ignore predation at your peril: results from multispecies state-space modeling (Trijoulet, et al., 2020)

Ignoring trophic interactions that occur in marine ecosystems induces bias in stock assessment outputs and results in low model predictive ability with subsequently biased reference points.

Best Practices 5: Uncertainty

Has uncertainty been assessed in the output of interest? Has sensitivity analysis been performed and how does it affect those outputs?

Show estimates of uncertainty in the output quantity of interest. Uncertainty analysis estimating confidence intervals is ideal. Otherwise list key sources of uncertainty and expected bounds on outputs based on those (possibly from sensitivity analysis)--i.e. design sensitivity analysis to approximate uncertainty analysis.

Specific analyses:

1. Retrospective analysis (5 year peel of all input data, example)
2. Forecast uncertainty: remove last 3-5 years of survey index only to see how well the model works in forecast mode, given the catch that actually happened.
3. Sensitivity to stomach data and other key or low-confidence data sources
4. Sensitivity to key parameters: consumption rates, residual mortality (M1, M0)
5. Sensitivity to initial conditions

Simpler investigation of uncertainty can be appropriate for complex models with long runtimes (Kaplan, et al., 2016).

Retain multiple parameterizations that meet the above criteria to allow scenario testing across a range of parameterizations. A simple method uses bounding (e.g. base, low bound, and high bound productivity scenarios; (Saltelli, et al., 2010).

Best Practices 6: Peer review

Are any Atlantic herring harvest control rules good for both fisheries and predators?

Problem identification: New England Herring MSE

Appropriateness of model for problem

How do changes in herring populations affect predators?

Availble models:

Multispecies catch at age (Curti, et al., 2013)

Multispecies catch at length (Gaichas, et al., 2017)

Multispecies production (Gamble, et al., 2009)

all model predation mortality but not prey effects on predators!

Food web (Link, et al., 2008; Lucey, et al., 2020)

models 2 way interactions but aggregated species groups and fishery

MICE for MSE: Addressing stakeholder objectives (Deroba, et al., 2018)

MICE for MSE Predators: deterministic delay-difference population models with herring-predator links

Uncertainty bracketed by herring population and assessment assumptions

Managing tradeoffs under uncertainty: What control rules give us 90% of everything we want?

Tern productivity 1.0 > 90% of the time, Herring biomass > 90% of SSBmsy, and Fishery yield > 90% of MSY

All three of the conditions on the left, AND fishery closures (F=0) less than 1% of the time

Peer review, March 2017

Report

The Panel agreed that the NEFSC technical team constructed a series of models (Atlantic herring, predator, and economic) appropriate for evaluating ABC control rules for the Atlantic herring fishery in the context of herring’s role as a forage fish. The Panel detailed areas of strength and areas for improvement in the MSE workshop process, modeling, and synthesis. The Panel concluded that the data, methods, and results of the MSE are sufficient for the Council to use when identifying and analyzing a range of ABC control rule alternatives for the Atlantic Herring Fishery Management Plan. Overall, the Panel concluded that the Atlantic herring MSE represents the best available science at this time for evaluating the performance of herring control rules and their potential impact on key predators.

But, predator models were simple and may not capture all important effects, combination of approaches may be better, integrate impacts of predators on herring, consider predator's alternative prey, include process error in predator models.

Epilogue: Herring MSE food web modeling supplemental results

• Tradeoffs between forage groups and mixed impacts to predators apparent when multiple species and full predator prey interaction feedbacks can be included

Conclusions: What makes a good multispecies model?

References

Additional resources

Thank you!