class: right, middle, my-title, title-slide # Generating datasets for <br> model performance testing: ## How challenging can it be? ### Sarah Gaichas, Isaac Kaplan, Patrick Lynch, and Christine Stawitz <br /> Northeast and Northwest Fisheries Science Centers, <br /> and Office of Science & Technology <br /> <br /> <br />Supported by REDUS project, IMR, and NOAA NMFS International Fellowship --- class: top, left ## Fragile ecosystems, robust assessments? Performance testing stock assessments for the California Current and Nordic and Barents Seas under climate change .pull-left[ - Scientist exchange central to this project - NOAA internal "international fellowship" and REDUS funding: - Gaichas, April 15 - June 15 - Lynch, April 18 - June 7 - Kaplan, May 11 - June 21 - Christine Stawitz, May 19-25 ] .pull-right[  *¹* ] - Additional NMFS and IMR collaborators with diverse skills and experiences - Emma Hodgson, other stock assessment and modeling folks - Cecilie Hansen, Daniel Howell, Erik Olsen, and now **YOU** .footnote[ [1] https://www.meganstarr.com/30-things-you-should-know-before-moving-to-norway/ ] --- # Project motivation .pull-left[ - Changing climate and ocean conditions &rarr; Shifting distributions, changing productivity - Needs: - Improve our ability to project global change impacts in the California Current and Nordic/Barents Seas (and elsewhere) - Test the performance of stock assessments to these impacts ] .pull-right[ *Climate-Ready Management¹*  ] .footnote[ [1] Karp, Melissa A. et al. 2019. Accounting for shifting distributionsand changing productivity in the development of scientific advice for fishery management. – ICES Journal of Marine Science, doi:10.1093/icesjms/fsz048. ] ??? --- ## End-to-end ecosystem operating models Atlantis modeling framework: [Fulton et al. 2011](https://onlinelibrary.wiley.com/doi/full/10.1111/j.1467-2979.2011.00412.x), [Fulton and Smith 2004](https://www.ajol.info/index.php/ajms/article/view/33182) .pull-left[ **Norwegian-Barents Sea** [Hansen et al. 2016](https://www.imr.no/filarkiv/2016/04/fh-2-2016_noba_atlantis_model_til_web.pdf/nn-no), [2018](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0210419)  ] .pull-right[ **California Current** [Marshall et al. 2017](https://onlinelibrary.wiley.com/doi/full/10.1111/gcb.13594), [Kaplan et al. 2017](https://www.sciencedirect.com/science/article/pii/S0304380016308262?via%3Dihub)  ] Building on global change projections: [Hodgson et al. 2018](https://www.sciencedirect.com/science/article/pii/S0304380018301856?via%3Dihub), [Olsen et al. 2018](https://www.frontiersin.org/articles/10.3389/fmars.2018.00064/full) ??? --- ## Project overview .pull-left[ 1. Scenarios for effects of temperature on growth, natural mortality 1. Focus on key stocks (Northeast Arctic cod, Norwegian spring spawning herring, California Current sardine, Pacific hake or a Pacific rockfish) 1. Atlantis output &rarr; dataset generator (`atlantisom`) &rarr; Stock Synthesis assessment 1. Compare performance of different model settings, multiple models, model ensembles ] .pull-right[ <img src="EDAB_images/CCAclimatescenario.png" width="921" style="display: block; margin: auto;" /> ] Bonus: `atlantisom` &rarr; multispecies, other single species models for testing ??? --- ## Test both estimation and (simple) MSE capability <img src="EDAB_images/PoseidonDesign.png" width="1709" style="display: block; margin: auto;" /> --- ## Design: Operating model scenario (climate and fishing) <img src="EDAB_images/projectionOMsetup.png" width="85%" style="display: block; margin: auto;" /> -- .pull-left[ * Recruitment variability in the operating model * Specify uncertainty in assessment inputs using `atlantisom` ] .pull-right[  ] --- ## Stock assessment uncertainty * Focus on climate impacts to growth and natural mortality (productivity) * Full factorial design over common stock assessment assumptions * Growth: 1. constant growth parameters 1. regime shifts in growth parameters ([Stawitz et al. 2019](https://www.sciencedirect.com/science/article/abs/pii/S0165783619300049?via%3Dihub)) 1. empirical weight-at-age ([Kuriyama et al. 2016](https://www.sciencedirect.com/science/article/abs/pii/S0165783615300837)) * Natural mortality: 1. fixed at an “uninformed” constant value of 0.2 1. fixed at a constant value that reflects the true average value (over time) from the OM 1. establishing regimes by fixing at the true average values from the OM over specified time blocks * Also, an equally weighted ensemble of the 9 estimation models * Stock synthesis framework initially, but extend to other assessment models, generalize, and use components of the REDUS framework --- ## Experimental design (figure added 7 June 2019) <img src="EDAB_images/expdesignfig.png" width="80%" style="display: block; margin: auto;" /> --- ## Make atlantis output into assessment model input Example workflow: 1. Get true biomass, abundance, age composition, length composition, weight at age, fishery catch, fishery catch at age, fishery length composition, and fishery weight age age for a "sardine-like species": https://sgaichas.github.io/poseidon-dev/FullSardineTruthEx.html 1. Format these outputs and get other life history parameters for input into a stock assessment model ([Stock Synthesis](https://www.sciencedirect.com/science/article/abs/pii/S0165783612003293), using [`r4ss`](https://github.com/r4ss)): https://sgaichas.github.io/poseidon-dev/CreateStockSynthesis.html 1. Run the assessment model --- ## What can we do so far? .pull-left[ Survey census test NOBA <img src="EDAB_images/NOBAcensus.png" width="1792" style="display: block; margin: auto;" /> True length composition NOBA <img src="EDAB_images/NOBAherringtruelf.png" width="1792" style="display: block; margin: auto;" /> ] .pull-right[ Standard survey test CCA <img src="EDAB_images/CCsurveyex.png" width="1792" style="display: block; margin: auto;" /> Survey length composition CCA <img src="EDAB_images/CCV3herringlfcomp.png" width="1792" style="display: block; margin: auto;" /> ] --- ## Full disclosure! .pull-left[ Still working on: 1. Estimating total mortality to back out M 1. Splitting aggregate age groups into true ages (not needed for NOBA) 1. Interpolating aggregate age groups weight at age for true ages 1. Fishery catch weight by area 1. Generalizing for different Atlantis codebase vintages ] .pull-right[  ] --- ## Specify uncertainty in assessment "data": REDUS to the rescue! What level of uncertainty is appropriate to carry through these analyses? .pull-left[ 1. Biological: 1. appropriate sigma-R? 1. Survey specification: 1. timing and spatial coverage? 1. which species are captured? 1. species-specific survey efficiency ("q")? 1. selectivity at age for each species? ] .pull-right[ 1. Survey uncertainty: 1. additional observation error (survey cv for index)? 1. effective sample size for biological samples? 1. Fishery uncertainty: 1. additional observation error (catch cv for total)? 1. catch sampled for length/age in all areas? 1. effective sample size for biological samples? ] --- ## External Resources * [Atlantis Model Documentation](https://github.com/Atlantis-Ecosystem-Model/Atlantis_example_and_instructions) * [atlantisom R package](https://github.com/r4atlantis/atlantisom) * [Testing atlantisom](https://github.com/sgaichas/poseidon-dev) * Slides available at https://noaa-edab.github.io/presentations --- background-image: url("EDAB_images/IMG_2733.jpg") background-size: cover ## Questions? # Tusen Takk til Havforskningsinstituttet!