+ - 0:00:00
Notes for current slide
Notes for next slide

Operating Model and
Simulated Dataset Overview

ICES WGSAM ToR C Skill Assessment

Sarah Gaichas, Sean Lucey, Andy Beet
Northeast Fisheries Science Center



Many thanks to:
(Atlantis): Joe Caracappa, Isaac Kaplan, Cecilie Hanson, Beth Fulton
(atlantisom development): Christine Stawitz, Kelli Johnson, Alexander Keth, Allan Hicks, Sean Lucey, Emma Hodgson, Gavin Fay
(atlantisom use): Bai Li, Alfonso Perez Rodriguez, Howard Townsend
(WGSAM discussion): Grant Adams, Valerio Bartolino, Gustav Delius, Caroline Mckeon, Mike Spence, Vanessa Trijoulet
(Hydra development): Gavin Fay, Cristina Perez

1 / 20

Outline

  • Skill assessment with Atlantis "data"

    1. Quick Atlantis overview
    2. R package atlantisom
    3. R package mskeyrun dataset
    4. Multispecies assessment
    5. Food web models: WIP

atlantis icon

  • Discussion: what's next

    • Progress from modeling groups
    • Discuss plans for
      • common input parameters
      • common outputs
      • skill comparisons

poseidon

2 / 20

Multispecies model skill assessment

Stow et al. 2009 Stow et al 2009 Fig 1

Olsen et al. 2016 Olsen et al Fig 1

3 / 20

"Both our model predictions and the observations reside in a halo of uncertainty and the true state of the system is assumed to be unknown, but lie within the observational uncertainty (Fig. 1a). A model starts to have skill when the observational and predictive uncertainty halos overlap, in the ideal case the halos overlap completely (Fig. 1b). Thus, skill assessment requires a set of quantitative metrics and procedures for comparing model output with observational data in a manner appropriate to the particular application."

Virtual worlds with adequate complexity: end-to-end ecosystem models

Atlantis modeling framework: Fulton et al. 2011, Fulton and Smith 2004

Norwegian-Barents Sea

Hansen et al. 2016, 2018

NOBA scale 70%

California Current

Marshall et al. 2017, Kaplan et al. 2017

CCAspatial scale 80%

Building on global change projections: Hodgson et al. 2018, Olsen et al. 2018

4 / 20

Atlantis Basics: Audzijonyte et al. 2019, Atlantis Ecosystem Model GitHub

3D ocean model

Model timestep: 12-24 hours
Model output timestep: Set by modeler

  • Here, 73 days (5x/year)

Living groups:

  • Biomass pools
  • Age structured

Model tracers:

  • Nitrogen (reserve + structural)
  • Numbers at age class

Audzijonyte et al. 2019 Fig 2

Submodels for processes

Deterministic physics and ecology
Here, harvest forced F by fleet with age selectivity; not using assessment or economics submodels

Audzijonyte et al. 2019 Fig 1

5 / 20

Design: Ecosystem model scenario (climate and fishing)

6 / 20

  • Recruitment variability in the operating model

  • Specify uncertainty in assessment inputs using atlantisom

sardinerec scale 100%

Overview of atlantisom R package: link

Started at the 2015 Atlantis Summit atlantisom intro

atlantisom get started

7 / 20

atlantisom workflow: get "truth"

  • locate files
  • run om_init
  • select species
  • run om_species

additions!

  • consumption
  • subannual catch
  • biomass pools
8 / 20

atlantisom workflow: get "data"

  • specify surveys
    • (can now have many, file for each)
    • area/time subsetting
    • efficiency (q) by species
    • selectivity by species
    • biological sample size
    • index cv
    • length at age cv
    • max size bin
  • specify survey diet sampling (new)
    • requires detaileddietcheck.txt
    • diet sampling parameters
  • specify fishery
    • area/time subsetting
    • biological sample size
    • catch cv
  • run om_index
  • run om_comps
  • run om_diet
  • environmental data functions too
9 / 20

atlantisom outputs, survey biomass index, link

Perfect information (one Season) NOBA fall survey 1

Survey with catchability and selectivity NOBA fall survey 2

10 / 20

atlantisom outputs, age and length compositions, link

capelin lengths halibut lengths

capelin ages halibut ages

11 / 20

Testing a simple "sardine" assessment, CC Atlantis Kaplan et al. 2021

Kaplan et al Fig 2

Will revisit with newer CC model; issues with different growth than assumed in SS setup?

workinprogress

12 / 20

atlantisom outputs, diet compositions, link

true diets

seasonal survey diets

13 / 20

ms-keyrun simulated data

14 / 20

Dataset additions

Subannual fishery data: catch, length, age, weight at age example 5x year catch outputs Biological parameters:

  • Age class to age
  • Recruitment timing
  • Maturity ogives
  • Gape width (pred/prey wt ratio)

Food web model group biomass, catch, diet example food web group B

15 / 20

Example initial results for Hydra: Biomass Skill across model structures

FROM LAST YEAR: MODELS SHOWN ARE EXAMPLE TRIAL FITS, NOT FINISHED OR GOOD MODELS

16 / 20

Base run completed for 5 and 10 bins, now the work begins.

Initial results for Hydra: example skill summary statistics, 5 bin model structures

FROM LAST YEAR: MODELS SHOWN ARE EXAMPLE TRIAL FITS, NOT FINISHED OR GOOD MODELS

17 / 20

Multispecies model skill comparison, ICES WGSAM 2023-2024

Do the same thing with Gadget, CEATTLE, Mizer, LeMans, Rpath... more participants welcome!

Simulated datasets have been added to mskeyrun

Discussion ongoing for planned comparisons

18 / 20

P.S. What else could we test?

xkcd_ensemble_model_2x

https://xkcd.com/1885/

19 / 20

Discussion and plans for this week

Atlantis related

  • Calculations correct? attempted M, per capita consumption

Skill assessment related

  • Running stock assessment models is difficult to automate. A lot of decisions are made by iterative running and diagnostic checks.
  • Do we want to automate to have more repetitions? Or tailor a small set of fits to a single dataset?
  • Generating input parameters for models that are consistent with Atlantis can be time consuming (Atlantis is a little too realistic...)
    • Fit vonB length at age models to atlantisom output for input to length based model, not all converge (!)
    • What is M (see above)

Slides available at https://noaa-edab.github.io/presentations

Contact: Sarah.Gaichas@noaa.gov

https://xkcd.com/2289/

20 / 20

  • atlantisom is using outputs not often used in other applications

    • I don't run Atlantis so putting print statements in code not an option

    • could be more efficient with targeted group work

    • should we expect numbers in one output to match those in others?

    • diet comp from detailed file matches diet comp in simpler output
    • catch in numbers not always matching between standard and annual age outputs
    • YOY output

    • ... others that have been encountered

    • estimating per capita consumption from detaileddiet.txt results in lower numbers than expected

    • still can't get reasonable mortality estimates from outputs--understand this is an issue

Outline

  • Skill assessment with Atlantis "data"

    1. Quick Atlantis overview
    2. R package atlantisom
    3. R package mskeyrun dataset
    4. Multispecies assessment
    5. Food web models: WIP

atlantis icon

  • Discussion: what's next

    • Progress from modeling groups
    • Discuss plans for
      • common input parameters
      • common outputs
      • skill comparisons

poseidon

2 / 20
Paused

Help

Keyboard shortcuts

↑, ←, Pg Up, k Go to previous slide
↓, →, Pg Dn, Space, j Go to next slide
Home Go to first slide
End Go to last slide
Number + Return Go to specific slide
b / m / f Toggle blackout / mirrored / fullscreen mode
c Clone slideshow
p Toggle presenter mode
t Restart the presentation timer
?, h Toggle this help
Esc Back to slideshow