Operating Model and Simulated Dataset Overview

class: right, middle, my-title, title-slide

.title[
# Operating Model and Simulated Dataset Overview
]
.subtitle[
## ICES WGSAM ToR C Skill Assessment
]
.author[
### Sarah Gaichas, Sean Lucey, Andy Beet Northeast Fisheries Science Center Many thanks to: (Atlantis): Joe Caracappa, Isaac Kaplan, Cecilie Hanson, Beth Fulton (atlantisom development): Christine Stawitz, Kelli Johnson, Alexander Keth, Allan Hicks, Sean Lucey, Emma Hodgson, Gavin Fay (atlantisom use): Bai Li, Alfonso Perez Rodriguez, Howard Townsend (WGSAM discussion): Grant Adams, Valerio Bartolino, Gustav Delius, Caroline Mckeon, Mike Spence, Vanessa Trijoulet (Hydra development): Gavin Fay, Cristina Perez
]

---

class: top, left

## Outline

.pull-left[

* Skill assessment with Atlantis "data"

1. Quick Atlantis overview
    1. R package [`atlantisom`](https://github.com/r4atlantis/atlantisom)
    1. R package [`mskeyrun`](https://github.com/NOAA-EDAB/ms-keyrun) dataset
    1. Multispecies assessment
    1. Food web models: WIP

.center[
![:img atlantis icon](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/atlantis-logo2.jpg)

]
]

.pull-right[
* Discussion: what's next
 
    + Progress from modeling groups
    + Discuss plans for 
        + common input parameters
        + common outputs
        + skill comparisons

![:img poseidon](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/poseidon-1024x6391.jpg)
]

---

## Multispecies model skill assessment

.pull-left-40[
[Stow et al. 2009](https://www.sciencedirect.com/science/article/abs/pii/S0924796308001103?via%3Dihub)
![:img Stow et al 2009 Fig 1, 90%](https://ars.els-cdn.com/content/image/1-s2.0-S0924796308001103-gr1.jpg)
]
.pull-right-60[
[Olsen et al. 2016](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0146467)
![:img Olsen et al Fig 1, 95%](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/pone.0146467.g001.png) 
]
???
"Both our model predictions and the observations reside in a halo of uncertainty and the true state of the system is assumed to be unknown, but lie within the observational uncertainty (Fig. 1a). A model starts to have skill when the observational and predictive uncertainty halos overlap, in the ideal case the halos overlap completely (Fig. 1b). Thus, skill assessment requires a set of quantitative metrics and procedures for comparing model output with observational data in a manner appropriate to the particular application."

---
## Virtual worlds with adequate complexity: end-to-end ecosystem models

Atlantis modeling framework: [Fulton et al. 2011](https://onlinelibrary.wiley.com/doi/full/10.1111/j.1467-2979.2011.00412.x), [Fulton and Smith 2004](https://www.ajol.info/index.php/ajms/article/view/33182)

.pull-left[
**Norwegian-Barents Sea**

[Hansen et al. 2016](https://www.imr.no/filarkiv/2016/04/fh-2-2016_noba_atlantis_model_til_web.pdf/nn-no), 
[2018](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0210419)

![NOBA scale 70%](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/journal.pone.0210419.g001.png)
]
.pull-right[
**California Current**

[Marshall et al. 2017](https://onlinelibrary.wiley.com/doi/full/10.1111/gcb.13594), 
[Kaplan et al. 2017](https://www.sciencedirect.com/science/article/pii/S0304380016308262?via%3Dihub)

![CCAspatial scale 80%](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/CCAspatial.png)
]

Building on global change projections:  [Hodgson et al. 2018](https://www.sciencedirect.com/science/article/pii/S0304380018301856?via%3Dihub), 
[Olsen et al. 2018](https://www.frontiersin.org/articles/10.3389/fmars.2018.00064/full)

???
---
## Atlantis Basics: [Audzijonyte et al. 2019](https://besjournals.onlinelibrary.wiley.com/doi/full/10.1111/2041-210X.13272), [Atlantis Ecosystem Model GitHub](https://github.com/Atlantis-Ecosystem-Model/Atlantis_example_and_instructions)

.pull-left-40[
3D ocean model

![](https://besjournals.onlinelibrary.wiley.com/cms/asset/e309c5c9-c3c5-4eaf-81d6-9f8aaaea0f34/mee313272-fig-0002-m.jpg)
.contrib[
Model timestep: 12-24 hours  
Model output timestep: Set by modeler 
*  Here, 73 days (5x/year)

Living groups: 
*  Biomass pools
*  Age structured

Model tracers: 
*  Nitrogen (reserve + structural)
*  Numbers at age class
]

.footnote[
[Audzijonyte et al. 2019 Fig 2](https://besjournals.onlinelibrary.wiley.com/doi/full/10.1111/2041-210X.13272)
]
]

.pull-right-60[
Submodels for processes

![](https://besjournals.onlinelibrary.wiley.com/cms/asset/d1ad86fd-282e-4297-890b-188ebad264b8/mee313272-fig-0001-m.jpg)
.contrib[
Deterministic physics and ecology  
Here, harvest forced F by fleet with age selectivity; not using assessment or economics submodels
]

.footnote[
[Audzijonyte et al. 2019 Fig 1](https://besjournals.onlinelibrary.wiley.com/doi/full/10.1111/2041-210X.13272)
]
]

---
## Design: Ecosystem model scenario (climate and fishing)

???

.pull-left[
* Recruitment variability in the operating model

* Specify uncertainty in assessment inputs using `atlantisom`
]
.pull-right[
![sardinerec scale 100%](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/CCAsardineRecVar.png)
]

---
## Overview of `atlantisom` R package: [link](https://sgaichas.github.io/poseidon-dev/atlantisom_landingpage.html)

.pull-left[
Started at the 2015 Atlantis Summit
![atlantisom intro](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/atlantisomlanding-top.png)
]
.pull-right[
![atlantisom get started](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/atlantisomlanding-getstarted.png)
]

---
background-image: url("https://github.com/sgaichas/poseidon-dev/raw/master/docs/images/atlantisomDataFlow_truth-link.png")
background-size: 850px
background-position: right bottom

## `atlantisom` workflow: get "truth"

* locate files
* run `om_init`
* select species
* run `om_species`

## additions!

* consumption
* subannual catch
* biomass pools

---
background-image: url("https://github.com/sgaichas/poseidon-dev/raw/master/docs/images/atlantisomDataFlow_data.png")
background-size: 625px
background-position: right top

## `atlantisom` workflow: get "data"

*  specify surveys 
    + (can now have many, file for each)
    + area/time subsetting
    + efficiency (q) by species
    + selectivity by species
    + biological sample size
    + index cv
    + length at age cv
    + max size bin
*  specify survey diet sampling (new)
    + requires detaileddietcheck.txt
    + diet sampling parameters
*  specify fishery
    + area/time subsetting
    + biological sample size
    + catch cv
*  run `om_index`
*  run `om_comps`
*  run `om_diet`
*  environmental data functions too

---
## `atlantisom` outputs, survey biomass index, [link](https://sgaichas.github.io/poseidon-dev/msSurveysTest.html)

.pull-left[
Perfect information (one Season)
![NOBA fall survey 1](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/NOBAsurvey-perfect.png)
]
.pull-right[
Survey with catchability and selectivity
![NOBA fall survey 2](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/atlantisomNOBAsurvey-qsel.png)
]

---
## `atlantisom` outputs, age and length compositions, [link](https://sgaichas.github.io/poseidon-dev/msSurveysTest.html)

.pull-left[
.center[
![:img capelin lengths, 75%](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/NOBAcapelinlengths.png)
![:img halibut lengths, 75%](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/NOBAhalibutlengths.png)
]
]
.pull-right[
.center[
![:img capelin ages, 75%](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/NOBAcapelinage.png)
![:img halibut ages, 75%](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/NOBAhalibutage.png)
]
]

---
## Testing a simple "sardine" assessment, CC Atlantis [Kaplan et al. 2021](https://www.frontiersin.org/articles/10.3389/fmars.2021.624355/full)
.pull-left[
![:img Kaplan et al Fig 2](https://www.frontiersin.org/files/Articles/624355/fmars-08-624355-HTML/image_m/fmars-08-624355-g002.jpg)

]
.pull-right[

Will revisit with newer CC model; issues with different growth than assumed in SS setup?

![:img workinprogress, 30%](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/work-in-progress.png)
]

---
## `atlantisom` outputs, diet compositions, [link](https://sgaichas.github.io/poseidon-dev/SurveyDietCompTest.html)

.pull-left[
![true diets](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/NOBAcoddiettrue.png)
]
.pull-right[
![seasonal survey diets](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/NOBAcodspringfallsurvdiet.png)
]

---
## [ms-keyrun simulated data](https://noaa-edab.github.io/ms-keyrun/articles/SimData.html)

---
.pull-left-40[
## Dataset additions

Subannual fishery data: 
catch, length, age, weight at age
![:img example 5x year catch outputs](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/mskeyrun_subannualcatch.png)
Biological parameters:  
*  Age class to age  
*  Recruitment timing  
*  Maturity ogives  
*  Gape width (pred/prey wt ratio)
]

.pull-right-60[
Food web model group biomass, catch, diet
![:img example food web group B](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/mskeyrun_surveyBfoodweb.png)
]

---
## Example initial results for Hydra: Biomass Skill across model structures

.pull-left[
![](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/5binlogBskill.png)
]

.pull-right[
![](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/10binlogBskill.png)
]
.center[
*FROM LAST YEAR: MODELS SHOWN ARE EXAMPLE TRIAL FITS, NOT FINISHED OR GOOD MODELS*
]

???
Base run completed for 5 and 10 bins, now the work begins.

---
## Initial results for Hydra: example skill summary statistics, 5 bin model structures

.pull-left[
![](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/skill5binlog.png)
]

.pull-right[
![](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/MEF5bin.png)
]
.center[
*FROM LAST YEAR: MODELS SHOWN ARE EXAMPLE TRIAL FITS, NOT FINISHED OR GOOD MODELS*
]

---
## Multispecies model skill comparison, ICES WGSAM 2023-2024

Do the same thing with Gadget, CEATTLE, Mizer, LeMans, Rpath... more participants welcome!

.pull-left[
[Simulated datasets](https://github.com/NOAA-EDAB/ms-keyrun/issues/56) have been added to mskeyrun

![](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/SimDataIssueMSkeyrun.png)

]

.pull-right[
[Discussion ongoing](https://github.com/NOAA-EDAB/ms-keyrun/discussions) for planned comparisons

![](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/MSkeyrunDIscussions.png)

]

---
## P.S. What else could we test?
.center[
![:img xkcd_ensemble_model_2x, 70%](https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/xkcd_ensemble_model_2x.png)
]
.footnote[
https://xkcd.com/1885/
]

---
background-image: url("https://github.com/NOAA-EDAB/presentations/raw/master/docs/EDAB_images/scenario_4.png")
background-size: 500px
background-position: right
## Discussion and plans for this week
.pull-left-60[
Atlantis related

*  Calculations correct? attempted M, per capita consumption
    
Skill assessment related

*  Running stock assessment models is difficult to automate. A lot of decisions are made by iterative running and diagnostic checks. 
*  Do we want to automate to have more repetitions? Or tailor a small set of fits to a single dataset?  
*  Generating input parameters for models that are consistent with Atlantis can be time consuming (Atlantis is a little too realistic...)
    + Fit vonB length at age models to atlantisom output for input to length based model, not all converge (!)
    + What is M (see above)
    
    
.footnote[
Slides available at https://noaa-edab.github.io/presentations

Contact: <Sarah.Gaichas@noaa.gov>
]

]
.pull-right-40[
.center[
.footnote[
https://xkcd.com/2289/
]
]
]

???
    + `atlantisom` is using outputs not often used in other applications
    + I don't run Atlantis so putting print statements in code not an option
    + could be more efficient with targeted group work

+ should we expect numbers in one output to match those in others?
    + diet comp from detailed file matches diet comp in simpler output
    + catch in numbers not always matching between standard and annual age outputs
    + YOY output
    + ... others that have been encountered

+ estimating per capita consumption from detaileddiet.txt results in lower numbers than expected
    + still can't get reasonable mortality estimates from outputs--understand this is an issue