Data Cleaning

library(atlantiscas)

The fishing data is a pulled from CAMS (1996-2021) data.

Field Descriptions

  • Year -
  • IDNUM
  • Area
  • Inside
  • PORTLANDED
  • PORTLND1
  • STATE1
  • MONTH
  • DATESAIL
  • GEARCODE
  • GEARCAT
  • NESPP3
  • SPPNM
  • QTYKEPT
  • LIVE
  • REVENUE
  • FMP
  • DOCID
  • TRIPID
  • PERMIT
  • SOURCE
  • DEALNUM
  • DAS
  • InsideREV
  • InsideLANDED
  • InsideDAS
  • nominal_revenue
  • YEAR
  • CAMS_PORT
  • CFG_PORT
  • PORT_NAME
  • STATE_ABB

We also need mesh size (VTR_MESH), target species (FISHERY_GROUP), and vessel size (VES_LEN, VES_GRTONS) information for each trip. We would like these fields to aid in data cleaning (eg. distinguish squid trips from groundfish) To get this we will need to join this data with CAMS_GARFO.CAMS_SUBTRIP table by IDNUM=VTR_IMGID. Then check CAMS_VTR_ORPHANS_SUBTRIP for absence of matches. This unfortunately doen’t work since VTR_IMGID which is defined as an ID at the subtrip level, is not! There are multiple cases where this definition is violated and thus results in a many-to-many relationship and hence duplicated data. This can not be reolved in a timely manner.

Ports

  • Port look up tables lack latitude and longitude
    • Use port, county, state to geocode lat and lon using open street maps
  • Ports have duplicate entries with spelling typos
    • eg. harwichport, harwich port. These were hand fixed

All PORTID within the Atlantis domain were identified. Other PORTIDs were grouped into either Canadian ports (north of the atlantis domain) or southern ports (south of atlantis domain)

PORTID was used to match PORTNM,COUNTY,STATE from the CAMS_GARFO.CFG_PORT table.Port names were then used to find lat and lon values using tidygeocoder using open street maps. Many port names and/or counties had to be edited due to spelling errors, incorrect information, inconsistent formatting, or missing information. For example:

  • SALISBURY COVE, ME -> SALSBURY COVE, ME (spelling)
  • (TOWN OF) -> (inconsistent formatting -removed text “(TOWN OF)” from port name)
  • PERKINS COVE, CUMBERLAND, ME -> OGUNQUIT, YORK, ME (incorrect)
  • OTHER FAIRFIELD, FAIRFIELD, CT -> FAIRFIELD, FAIRFIELD, CT (missing port info)
  • COMBS CREEK, ST. MARYS, MD -> NA, ST.MARY’S, MD (not found on maps)
  • DUKES, DUKES, MA -> NA, DUKES, MA (no such port)
  • ORRS ISLAND, ME <- ORR’S ISLAND, ME
  • LONG ISLAND CUMBERLAND, ME <- LONG ISLAND, ME
  • STUEBEN, ME <- STEUBEN, ME
  • WESTERLEY, RI <- WESTERLY, RI
  • BARINGTON, RI <- BARRINGTON, RI
  • DYERS BAY, ME <- STEUBEN, ME
  • MATHAIS POINT, VA <- DAHLGREN, VA
  • HAMPTON/SEABROOK, NH <- SEABROOK, NH
  • HARRIMANS POINT, ME <- BROOKLIN, ME
  • BROOKLIN, ME <- HANCOCK (COUNTY), ME
  • KINGS (COUNTY), NY <- BROOKLYN (COUNTY)

The following phrases were removed from PORTNM:

  • OTHER
  • COUNTY
  • TOWN OF
  • LYNNHAVEN
  • CAPTREE
  • PORT
  • AQUINNAH

The following were removed from COUNTY

  • CITY OF
  • COUNTY (except from NEW LONDON COUNTY, NEW HAVEN COUNTY)
  • NOT-SPECIFIED

Lat and lons were found by matching the port data in the folloiing order

  • PORTNM, COUNTY, STATE
  • COUNTY, STATE
  • PORTNM, STATE

The remaining records in the data with ports that did not have a PORTID were then matched to existing/known ports (with PORTIDs) based on the recorded PORTLANDED, STATE values.

Some records (negligible) were discarded since there was no identifying port, or state associated with the catch)

Species

  • Species have duplicate names with different formats
    • All species with NK or (NK) (Not known) in their name were formatted to NK SPECIESNAME
    • All species separated by commas (eg DRUM, BLACK) were formatted as BLACK DRUM
    • LOBSTER was replaced by AMERICAN LOBSTER
    • LOLIGO SQUID was replaced by LONGFIN SQUID
    • OTHER FISH have 2 codes (526,529)
  • Several NESPP3 codes are catch alls. For example:
    • NS SQUIDS
      • 803: MKTNM = UNCLASSIFIED
      • Coded as 802 (ILLEX SQUID)
    • SKATES
      • 365: MKTNM = UNCLASSIFIED, SMALL (WINGS), LARGE (WINGS)
      • Assigned to Atlantis group SK (Other skates)
    • OTHER FISH
      • 526: MKTNM = HEADS, FOOD, SPAWN
      • 529: MKTNM = INDUSTRIAL
      • Coded a either BPF (Other benthopelagic fish) or FDF (Misc. Demersal fish) based on gear that catched them (“Bottom Trawl”,“Sink Gillnet”,“Lobster Pot”,“Other Pot”,“Bottom Longline”,“Scallop Gear”) and (“Drift Gillnet”,“Midwater Trawl”,“Hand Gear”,“Pelagic Longline”,“Other Gear”,“Shrimp Trawl”,“Separator & Ruhle Trawl”) respectively
    • NK DOGFISH
      • 350: MKTNM: ROUND, TAILS, DRESSED, FINS. Codes as 352 (SPINY DOGFISH)
    • NK SHARK
      • 359: UNCLASSIFIED DRESSED/ROUND, FINS DRIED/FRESH/FROZEN,GUTTED,CHUNKS
      • Left as is
    • NK CLAM
      • 764:
      • Coded as 769 (SURF CLAM)
    • HAKE MIX RED & WHITE
      • 155:
    • SILVER&OFFSHIREHAKE MIX
      • 507:

GEARS

Squid fleet is not designated but both squid fishers and groundfish fishers use the same GEARCAT. To distinguish squid trip were teased apart from groundfish trips based on the composition of the catch

Things to Do

  1. Landings by Canadian fleet - Look at NAFO data. May be too large of a statistical area to do much with since we’d need to map it to NEUS atlantis polygons.

  2. Vessel size, mesh size

  3. Could use permits to help differentiate squid/groundfish