SHINE/ISWAT/ESWW Community Effort towards SEP Model Validation

Organizers: Katie Whitman (NASA JSC SRAG/University of Houston), Hazel Bain (NOAA SWPC/CU Boulder CIRES), Leila Mays (NASA Goddard), Phil Quinn (NASA JSC SRAG), Ian Richardson (NASA Goddard/University of Maryland), Mark Dierckxsens (BIRA-IASB)

Site updated 2021-10-22

Introduction

This website was created to provide SEP modelers with all the information needed to participate in the SHINE/ISWAT/ESWW SEP Model Validation Community Effort. All SEP modelers are invited to send their model output to Kathryn.Whitman@nasa.gov for this ongoing effort.

This effort supports the goals of ISWAT H3-01 SEP Model Validation section and the Integrated Solar Energetic Proton Alert/Warning System (ISEP) project. ISEP is a collaboration between the Space Radiation Analysis Group (SRAG) at NASA JSC, the Community Coordinated Modeling Center (CCMC) and NASA GSFC, and the Moon to Mars Space Weather Analysis Office (M2M) at NASA GSFC. The ISEP project was established to transition mature SEP models into operations and improve SRAG's SEP forecasting capability for human exploration missions beyond Low Earth Orbit.

GOALS:

• Collaborate directly with modelers to collect SEP model predictions for as many SEP models as possible that exist in the research and operations communities
• Develop a generalized coding infrastructure to validate all types of SEP models for quantities important for operations and science.
• Create a validation pipeline that uses the same inputs required by CCMC's SEP Scoreboard
• Determine the best-uses of the available data sets for validation; Derive validation data sets
• Provide modelers with measurements that may be interpreted as the suprathermal seed particle spectrum prior to each event

CURRENT STATUS (updated 2021-10-21):

• An end-to-end validation framework has been demonstrated
• Values are derived from observations and flux time profile models with the publicly available code created for this effort (https://github.com/ktindiana/operational-sep)
• A first draft validation code that is capable of comparing observations and predictions and calculating a variety of metrics and skill scores has been developed (see status update from ISWAT 2021)
• All modelers with models that can run in real-time are encouraged to contact CCMC about integration of their models into the SEP Scoreboard. (contact Leila Mays)
• For models that have flexibility in forecasting energy ranges, we are requesting model output in the following energy channels (see more at the bottom of the page):
• >10, >100 and any other GOES-like channels
• Energies corresponding to the SOHO/EPHIN energy channels, particularly for the validation of SEP event onsets
  

Approach for Validation Effort

A set of SEP events were chosen for this challenge and forecasting quality flare and CME parameters are provided. Modelers may choose whether to use these parameters.

Model results submitted to the challenge will be used as templates to build coding infrastructure that is capable of performing validation in a generic way that may be applied to all model types. The goal is a validation pipeline that is available to the community for validation of their models.

Model results will be compared with spacecraft measurements and validation metrics will be generated, however we acknowledge that this is a "partial" or "psuedo" validation due to the limited number of events. In the long-term, we intend to pursue creating long-baseline observational event lists (and negative event lists) for validation and work with modelers to submit results for a large number of historical cases. With enough test cases, meaningful validation metrics will be produced.

Additionally, compiling predictions from many models together in one place demonstrates the forecasting coverage that already exists (or could exist through R2O2R efforts) in the modeling community. We will also be able to highlight what information is missing.

The results of this effort will be compiled and presented at SHINE, ISWAT, and European Space Weather week and made available on this website for community review and feedback.

Participating Models

We are excited that many modelers in the community have shown interest in participating in this challenge.

ADEPT (White, Kahler, & Ling), AFRL PPS (White & Kahler), COMESEP (Dierckxsens et al.), FORSPEF (Anastasiadis et al.), HESPERIA/REleASE (Posner, Malandraki, Kuehl, Karavalos), iPATH (Li, Hu), M-FLAMPA (Sokolov, Borovikov, Zhao), PCA model (Papaioannou), PROTONS (Balch), SAWS-ASPECS (Anastasiadis, Papaioannou, Vasalos), SEPMOD (Luhmann, Lee), SEPCaster (Li, Jin), SEPSTER (Richardson), SEPSTER2D (Bruno), SPARX (Marsh, Dalla), STAT (Linker, Schwadron et al.), UMASEP (Nunez), Zhang model.

Contact Katie Whitman (kathryn.whitman@nasa.gov) to participate.

Challenge Events

Information about each SEP event can be found under SEP Events.

SHINE 2020 Challenge Events:
12 July 2012
6 and 7 January 2014
11 April 2013

SHINE 2019 Challenge Events:
10 September 2017 (again)
17 May 2012
7 March 2012

SHINE 2018 Challenge Events:
4 and 6 September 2017
10 September 2017
14 July 2017

Standard Inputs

At ISWAT 2020, the SEP modeling community expressed interest in using standard inputs for this challenge. We have compiled a list of flare and CME parameters for each SEP event. The CME parameters are forecaster quality, made at the time of event onset without any revision. The parameters are either from NOAA or CCMC forecasters. The intent is to understand how well models perform using real-time data to get a sense of their capability in an operational environment.

Our colleague Maher Dayeh (SWRI) has compiled suprathermal seed spectra for each SEP event for input into physics-based models.

All of these values can be found on the Model Input Parameters page. It is not required to use these parameters to participate, however please specify your inputs when you provide your results.

Requested Model Output

We request that modelers provide forecasts in GOES-like energy channels, particularly >10 and >100 MeV since these are used in operational decision-making by SRAG.

We also request, if possible, for models that forecast event start times to provide forecasts in SOHO/EPHIN energy channels. Comparison to SOHO/EPHIN data will allow for a more accurate assessment of model performance for SEP onsets. We expect that this request is most pertinent to physics-based models that produce flux time profiles.

SOHO/EPHIN Channel	Geometric Mean [MeV]	Minimum Energy [MeV]	Maximum Energy [MeV]
P4	5.79	4.3	7.8
P8	13.51	7.8	25
P25	31.98	25	40.9
P41	46.56	40.9	53

 

We understand that many models are developed to forecast for fixed energy values that may not correspond to the requested energy channels. That is fine and all model predictions are welcome for this effort. We will work one-on-one with modelers to find the best approach to validate each model.

Model Output Format

Modelers are encouraged to submit their predictions to this challenge using the CCMC json format, however any formats are accepted at this time. For models that produce flux time profiles, you may submit the time profiles directly and Katie will use operational_sep_quantities.py to create the json files for validation.