SHINE/ISWAT/ESWW SEP Model Validation Challenge
Organizers: Katie Whitman (NASA JSC SRAG/University of Houston), Hazel Bain (NOAA SWPC/CU Boulder CIRES), Leila Mays (NASA Goddard), Phil Quinn (NASA JSC SRAG), Ian Richardson (NASA Goddard/University of Maryland), Mark Dierckxsens (BIRA-IASB)
This website was created to provide SEP modelers with all the information needed to participate in the SHINE/ISWAT/ESWW SEP Model Validation Community Effort. All SEP modelers are invited to send their model output to Kathryn.Whitman@nasa.gov for this ongoing effort.
This effort supports the goals of ISWAT H3-01 SEP Model Validation section and the Integrated Solar Energetic Proton Alert/Warning System (ISEP) project. ISEP is a collaboration between the Space Radiation Analysis Group (SRAG) at NASA JSC, the Community Coordinated Modeling Center (CCMC) and NASA GSFC, and the Moon to Mars Space Weather Analysis Office (M2M) at NASA GSFC. The ISEP project was established to transition mature SEP models into operations and improve SRAG's SEP forecasting capability for human exploration missions beyond Low Earth Orbit.
- Collaborate directly with modelers to collect SEP model predictions for as many SEP models as possible that exist in the research and operations communities
- Develop a generalized coding infrastructure to validate all types of SEP models for quantities important for operations and science.
- Create a validation pipeline that uses the same inputs required by CCMC's SEP Scoreboard
- Determine the best-uses of the available data sets for validation; Derive validation data sets
- Provide modelers with measurements that may be interpreted as the suprathermal seed particle spectrum prior to each event
CURRENT STATUS (updated 2021-10-21):
- An end-to-end validation framework has been demonstrated
- Values are derived from observations and flux time profile models with the publicly available code created for this effort (https://github.com/ktindiana/operational-sep
- A first draft validation code that is capable of comparing observations and predictions and calculating a variety of metrics and skill scores has been developed (see status update from ISWAT 2021)
- All modelers with models that can run in real-time are encouraged to contact CCMC about integration of their models into the SEP Scoreboard. (contact Leila Mays)
- For models that have flexibility in forecasting energy ranges, we are requesting model output in the following energy channels (see more at the bottom of the page):
- > 10, >100 and any other GOES-like channels
- Energies corresponding to the SOHO/EPHIN energy channels, particularly for the validation of SEP event onsets
Progress and Status
A method to calculate observational quantities for validation purposes has been created (https://github.com/ktindiana/operational-sep).
Following the discussions and feedback at ISWAT 2021, we have identified these next steps in the validation effort.
The campaign organizers will:
- Create a balanced list of SEP events and non-events that will allow a statistically significant validation
- Focus on events in cycles 24 and possibly 23 to allow as many models as possible to participate
- Provide a standard set of flare and CME inputs, including uncertainties on the CME parameters
- Provide suprathermal seed spectra
- Create a pre-event definition for a forecast - i.e. specify a time stamp for each event after which no data may be used to make the forecast
- Models that make predictions using data prior to that time will be validated as operational-type forecasts
- Models that make predictions using data after that time may be validated from a scientific perspective, but the results would not inform operational usability
- Investigate the feasibility to provide multi-point observations for the SEP event list (e.g. STEREO A & B) for validation of 2D and 3D physics-based models
- Encourage all modelers with models that have the ability to operate in real-time to work with CCMC to be integrated into the SEP Scoreboard (contact Leila Mays, firstname.lastname@example.org) - this would allow a true validation of forecasting capabilities using events going forward
- Work with the Moon to Mars Space Weather Analysis Office to develop an automated method to validate models running and sending forecasts in real-time to the SEP Scoreboard
- Add validation of the event-integrated differential fluence spectrum
- Improve formatting of the reports produced by the validation code
- Revisit the "onset peak" algorithm in the OpSEP code
Approach for Validation Effort
A set of SEP events were chosen for this challenge and forecasting quality flare and CME parameters are provided. Modelers may choose whether to use these parameters.
Model results submitted to the challenge will be used as templates to build coding infrastructure that is capable of performing validation in a generic way that may be applied to all model types. The goal is a validation pipeline that is available to the community for validation of their models.
Model results will be compared with spacecraft measurements and validation metrics will be generated, however we acknowledge that this is a "partial" or "psuedo" validation due to the limited number of events. In the long-term, we intend to pursue creating long-baseline observational event lists (and negative event lists) for validation and work with modelers to submit results for a large number of historical cases. With enough test cases, meaningful validation metrics will be produced.
Additionally, compiling predictions from many models together in one place demonstrates the forecasting coverage that already exists (or could exist through R2O2R efforts) in the modeling community. We will also be able to highlight what information is missing.
The results of this effort will be compiled and presented at SHINE, ISWAT, and European Space Weather week and made available on this website for community review and feedback.
We are excited that many modelers in the community have shown interest in participating in this challenge.
ADEPT (White, Kahler, & Ling), AFRL PPS (White & Kahler), COMESEP (Dierckxsens et al.), FORSPEF (Anastasiadis et al.), HESPERIA/REleASE (Posner, Malandraki, Kuehl, Karavalos), iPATH (Li, Hu), M-FLAMPA (Sokolov, Borovikov, Zhao), PCA model (Papaioannou), PROTONS (Balch), SAWS-ASPECS (Anastasiadis, Papaioannou, Vasalos), SEPMOD (Luhmann, Lee), SEPCaster (Li, Jin), SEPSTER (Richardson), SEPSTER2D (Bruno), SPARX (Marsh, Dalla), STAT (Linker, Schwadron et al.), UMASEP (Nunez), Zhang model.
Contact Katie Whitman (email@example.com) to participate.
At ISWAT 2020, the SEP modeling community expressed interest in using standard inputs for this challenge. We have compiled a list of flare and CME parameters for each SEP event. The CME parameters are forecaster quality, made at the time of event onset without any revision. The parameters are either from NOAA or CCMC forecasters. The intent is to understand how well models perform using real-time data to get a sense of their capability in an operational environment.
Our colleague Maher Dayeh (SWRI) has compiled suprathermal seed spectra for each SEP event for input into physics-based models.
All of these values can be found on the Model Input Parameters page. It is not required to use these parameters to participate, however please specify your inputs when you provide your results.
Requested Model Output
We request that modelers provide forecasts in GOES-like energy channels, particularly >10 and >100 MeV since these are used in operational decision-making by SRAG.
We also request, if possible, for models that forecast event start times to provide forecasts in SOHO/EPHIN energy channels. Comparison to SOHO/EPHIN data will allow for a more accurate assessment of model performance for SEP onsets. We expect that this request is most pertinent to physics-based models that produce flux time profiles.
|SOHO/EPHIN Channel||Geometric Mean [MeV]||Minimum Energy [MeV]||Maximum Energy [MeV]|
We understand that many models are developed to forecast for fixed energy values that may not correspond to the requested energy channels. That is fine and all model predictions are welcome for this effort. We will work one-on-one with modelers to find the best approach to validate each model.
Model Output Format
Modelers are encouraged to submit their predictions to this challenge using the CCMC json format, however any formats are accepted at this time. For models that produce flux time profiles, you may submit the time profiles directly and Katie will use operational_sep_quantities.py to create the json files for validation.