Skill assessment framework for a length based multispecies model

class: right, middle, my-title, title-slide

.title[
# Skill assessment framework for a length based multispecies model
]
.subtitle[
## ICES WGSAM ToR C Skill Assessment
]
.author[
### Sarah Gaichas Northeast Fisheries Science Center Gavin Fay and Cristina Perez University of Massachusetts Dartmouth Many thanks to: (atlantisom development): Christine Stawitz, Kelli Johnson, Alexander Keth, Allan Hicks, Sean Lucey, Emma Hodgson, Gavin Fay (Atlantis): Isaac Kaplan, Cecilie Hanson, Beth Fulton (atlantisom use): Bai Li, Alphonso Perez Rodriguez, Howard Townsend (skill assessment design): Patrick Lynch
]

---

class: top, left

## Outline

.pull-left[
* Management decisions and models

* Assessment model testing

* What do models need to do?
    * Challenges with what models need to do  
    * Addressing challenges with Atlantis

* Skill assessment with Atlantis "data"

1. R package `atlantisom`
    1. Single species assessment
    1. Multispecies assessment
    1. Other models? Ensembles?

]

.pull-right[
* Discussion: what's next
 
    + Automation of other model inputs!
    * Atlantis output verification
    * New outputs?
    + Best practices for 
        + dataset creation
        + input parameter estimation

![:img poseidon](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/poseidon-1024x6391.jpg)
]

---
background-image: url("https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/noaafisheries.png")
background-size: 900px
background-position: bottom center

# We do a lot of assessments

.center[
![:img NOAAlogo, 20%](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/noaa-logo.PNG) 

]

---
background-image: url("https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/ecomods.png")
background-size: 900px
background-position: bottom center

# With a wide range of models

.center[

]

---
background-image: url("https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/modeling_study.png")
background-size: 500px
background-position: right bottom

# How do we know they are right?

* Fits to historical data (hindcast)
 
 * Influence of data over time (retrospective diagnostics)
 
 * Keep as simple and focused as possible
 
 * Simulation testing

# But, what if

##   data are noisy?

##   we need to model complex interactions?

##   conditions change over time?

.footnote[
https://xkcd.com/2323/
]

---
## Skill assessment background (see [Journal of Marine Systems Special Issue](https://www.sciencedirect.com/journal/journal-of-marine-systems/vol/76/issue/1))

.pull-left-40[
[Stow et al. 2009](https://www.sciencedirect.com/science/article/abs/pii/S0924796308001103?via%3Dihub)
![:img Stow et al 2009 Fig 1](https://ars.els-cdn.com/content/image/1-s2.0-S0924796308001103-gr1.jpg)
]
.pull-right-60[
[Olsen et al. 2016](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0146467)
![:img Olsen et al Fig 1](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/pone.0146467.g001.png) 
]
???
"Both our model predictions and the observations reside in a halo of uncertainty and the true state of the system is assumed to be unknown, but lie within the observational uncertainty (Fig. 1a). A model starts to have skill when the observational and predictive uncertainty halos overlap, in the ideal case the halos overlap completely (Fig. 1b). Thus, skill assessment requires a set of quantitative metrics and procedures for comparing model output with observational data in a manner appropriate to the particular application."

---
## Skill assessment with ecological interactions... fit criteria alone are not sufficient
*Ignore predation at your peril: results from multispecies state-space modeling  *  [Trijoulet et al. 2020](https://besjournals.onlinelibrary.wiley.com/doi/full/10.1111/1365-2664.13515)
>Ignoring trophic interactions that occur in marine ecosystems induces bias in stock assessment outputs and results in low model predictive ability with subsequently biased reference points.

.pull-left-40[
![VanessaPaper](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/VanessaPaper.png)

EM1: multispecies state space

EM2: multispecies, no process error

EM3: single sp. state space, constant M

EM4: single sp. state space, age-varying M

*note difference in scale of bias for single species!*

]
.pull-right-60[
![modcomp](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/jpe13515-fig-0004-m.jpg)
]

???
This is an important paper both because it demonstrates the importance of addressing strong species interactions, and it shows that measures of fit do not indicate good model predictive performance. Ignoring process error caused bias, but much smaller than ignoring species interactions.
See also Vanessa's earlier paper evaluating diet data interactions with multispecies models

---
## Virtual worlds with adequate complexity: end-to-end ecosystem models

Atlantis modeling framework: [Fulton et al. 2011](https://onlinelibrary.wiley.com/doi/full/10.1111/j.1467-2979.2011.00412.x), [Fulton and Smith 2004](https://www.ajol.info/index.php/ajms/article/view/33182)

.pull-left[
**Norwegian-Barents Sea**

[Hansen et al. 2016](https://www.imr.no/filarkiv/2016/04/fh-2-2016_noba_atlantis_model_til_web.pdf/nn-no), 
[2018](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0210419)

![NOBA scale 70%](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/journal.pone.0210419.g001.png)
]
.pull-right[
**California Current**

[Marshall et al. 2017](https://onlinelibrary.wiley.com/doi/full/10.1111/gcb.13594), 
[Kaplan et al. 2017](https://www.sciencedirect.com/science/article/pii/S0304380016308262?via%3Dihub)

![CCAspatial scale 80%](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/CCAspatial.png)
]

Building on global change projections:  [Hodgson et al. 2018](https://www.sciencedirect.com/science/article/pii/S0304380018301856?via%3Dihub), 
[Olsen et al. 2018](https://www.frontiersin.org/articles/10.3389/fmars.2018.00064/full)

???
---
## Design: Ecosystem model scenario (climate and fishing)

???

.pull-left[
* Recruitment variability in the operating model

* Specify uncertainty in assessment inputs using `atlantisom`
]
.pull-right[
![sardinerec scale 100%](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/CCAsardineRecVar.png)
]

---
## Overview of `atlantisom` R package: [link](https://sgaichas.github.io/poseidon-dev/atlantisom_landingpage.html)

.pull-left[
Started at the 2015 Atlantis Summit
![atlantisom intro](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/atlantisomlanding-top.png)
]
.pull-right[
![atlantisom get started](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/atlantisomlanding-getstarted.png)
]

---
background-image: url("https://github.com/sgaichas/poseidon-dev/raw/master/docs/images/atlantisomDataFlow_truth-link.png")
background-size: 850px
background-position: right bottom

## `atlantisom` workflow: get "truth"

* locate files
* run `om_init`
* select species
* run `om_species`

---
background-image: url("https://github.com/sgaichas/poseidon-dev/raw/master/docs/images/atlantisomDataFlow_data.png")
background-size: 625px
background-position: right top

## `atlantisom` workflow: get "data"

*  specify surveys 
    + (can now have many, file for each)
    + area/time subsetting
    + efficiency (q) by species
    + selectivity by species
    + biological sample size
    + index cv
    + length at age cv
    + max size bin
*  specify survey diet sampling (new)
    + requires detaileddietcheck.txt
    + diet sampling parameters
*  specify fishery
    + area/time subsetting
    + biological sample size
    + catch cv
*  run `om_index`
*  run `om_comps`
*  run `om_diet`
*  environmental data functions too

---
## `atlantisom` outputs, survey biomass index, [link](https://sgaichas.github.io/poseidon-dev/msSurveysTest.html)

.pull-left[
Perfect information (one Season)
![NOBA fall survey 1](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/NOBAsurvey-perfect.png)
]
.pull-right[
Survey with catchability and selectivity
![NOBA fall survey 2](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/atlantisomNOBAsurvey-qsel.png)
]

---
## `atlantisom` outputs, age and length compositions, [link](https://sgaichas.github.io/poseidon-dev/msSurveysTest.html)

.pull-left[
.center[
![:img capelin lengths, 75%](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/NOBAcapelinlengths.png)
![:img halibut lengths, 75%](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/NOBAhalibutlengths.png)
]
]
.pull-right[
.center[
![:img capelin ages, 75%](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/NOBAcapelinage.png)
![:img halibut ages, 75%](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/NOBAhalibutage.png)
]
]
---
## `atlantisom` outputs, diet compositions, [link](https://sgaichas.github.io/poseidon-dev/SurveyDietCompTest.html)

.pull-left[
![true diets](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/NOBAcoddiettrue.png)
]
.pull-right[
![seasonal survey diets](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/NOBAcodspringfallsurvdiet.png)
]

---
## Testing a simple "sardine" assessment, CC Atlantis [Kaplan et al. 2021](https://www.frontiersin.org/articles/10.3389/fmars.2021.624355/full)
.pull-left[
![:img Kaplan et al Fig 2](https://www.frontiersin.org/files/Articles/624355/fmars-08-624355-HTML/image_m/fmars-08-624355-g002.jpg)

]
.pull-right[

Will revisit with newer CC model; issues with different growth than assumed in SS setup?

![:img workinprogress, 30%](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/work-in-progress.png)
]

---
## Cod assessment based on NOBA Atlantis (Li, WIP)

https://github.com/Bai-Li-NOAA/poseidon-dev/blob/nobacod/NOBA_cod_files/README.MD

.pull-left[
Conversion from SAM to SS successful
![](https://github.com/Bai-Li-NOAA/poseidon-dev/raw/nobacod/NOBA_cod_files/README_files/figure-gfm/sam_ss_comparison-1.png)
]
.pull-right[
Fitting to NOBA data more problematic
![](https://github.com/Bai-Li-NOAA/poseidon-dev/raw/nobacod/NOBA_cod_files/README_files/figure-gfm/unnamed-chunk-1-1.png)

]
---
## Multispecies assessment based on NOBA Atlantis (Townsend et al, WIP)

.pull-left[
Stepwise development process of self fitting, fitting to atlantis output, then skill assessment using atlantis output

![](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/MSSPMfittoNOBA.png)

Profiles for estimated parameters; but 
*what to compare K values to?*

![](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/MSSPMfittoNOBAdiagK.png)
]

.pull-right[
Can test model diagnostic tools as well
![](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/MSSPMfittoNOBAretro.png)
Using simulated data in [`mskeyrun` package](https://noaa-edab.github.io/ms-keyrun/), available to all
]

---
## ms-keyrun simulated data

---
## Initial results for Hydra: Example fits to biomass (left) and catch (right)

.pull-left[
![](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/NOBA05bin_5klogs1Bfit.png)
![](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/NOBA05bin_5klogs2Bfit.png)
]

.pull-right[
![](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/NOBA05bin_5kfleet1fit.png)
![](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/NOBA05bin_5kfleet2fit.png)
]

---
## Initial results for Hydra: Example fit to biomass vs biomass skill

.pull-left[
![](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/NOBA05bin_5klogs1Bfit.png)
]

.pull-right[
![](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/NOBA05bin_5klogBskill.png)
]

.center[
*MODELS SHOWN ARE EXAMPLE TRIAL FITS, NOT FINISHED OR GOOD MODELS*
]
---
## Initial results for Hydra: Biomass Skill across models

.pull-left[
![](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/5binBskill.png)
]

.pull-right[
![](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/10binBskill.png)
]
.center[
*MODELS SHOWN ARE EXAMPLE TRIAL FITS, NOT FINISHED OR GOOD MODELS*
]
---
## Initial results for Hydra: Biomass Skill across models

.pull-left[
![](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/5binlogBskill.png)
]

.pull-right[
![](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/10binlogBskill.png)
]
.center[
*MODELS SHOWN ARE EXAMPLE TRIAL FITS, NOT FINISHED OR GOOD MODELS*
]
---
## Initial results for Hydra: example skill summary statistics, 5 bin models

.pull-left[
![](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/skill5binlog.png)
]

.pull-right[
![](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/MEF5bin.png)
]
.center[
*MODELS SHOWN ARE EXAMPLE TRIAL FITS, NOT FINISHED OR GOOD MODELS*
]
---
## P.S. What else could we test?
.center[
![:img xkcd_ensemble_model_2x, 70%](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/xkcd_ensemble_model_2x.png)
]
.footnote[
https://xkcd.com/1885/
]
---
## Multispecies production model ensemble assessment--use Atlantis instead

.pull-left[
![:img Hydra OM setup](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/HydraOM3msprods.png)
]
.pull-right[
![:img ensemble results](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/ensembleMSprodskill.png)
]
---
background-image: url("https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/scenario_4.png")
background-size: 500px
background-position: right
## Difficulties so far
.pull-left-60[
Atlantis related

*  Understanding Atlantis outputs (much improved with documentation since 2015)
*  Reconciling different Atlantis outputs, which to use?
*  Calculations correct? attempted M, per capita consumption
    
Skill assessment related

*  Running stock assessment models is difficult to automate. A lot of decisions are made by iterative running and diagnostic checks.
*  Generating input parameters for models that are consistent with Atlantis can be time consuming (Atlantis is a little too realistic...)
    + Fit vonB length at age models to atlantisom output for input to length based model, not all converge (!)
    + What is M (see above)
    
    
.footnote[
Slides available at https://noaa-edab.github.io/presentations

Contact: <Sarah.Gaichas@noaa.gov>
]

]
.pull-right-40[
.center[
.footnote[
https://xkcd.com/2289/
]
]
]

???
    + `atlantisom` is using outputs not often used in other applications
    + I don't run Atlantis so putting print statements in code not an option
    + could be more efficient with targeted group work

+ should we expect numbers in one output to match those in others?
    + diet comp from detailed file matches diet comp in simpler output
    + catch in numbers not always matching between standard and annual age outputs
    + YOY output
    + ... others that have been encountered

+ estimating per capita consumption from detaileddiet.txt results in lower numbers than expected
    + still can't get reasonable mortality estimates from outputs--understand this is an issue