SHINE/ISWAT/ESWW SEP Model Validation Challenge
Organizers
Organizers: Katie Whitman (NASA JSC SRAG/University of Houston), Hazel Bain (NOAA SWPC/CU Boulder CIRES), Leila Mays (NASA Goddard), Phil Quinn (NASA JSC SRAG), Ian Richardson (NASA Goddard/University of Maryland), Mark Dierckxsens (BIRA-IASB)
Introduction
This website was created to provide SEP modelers with all the information needed to participate in the SHINE/ISWAT/ESWW SEP Model Validation Community Effort. All SEP modelers are invited to send their model output to Kathryn.Whitman@nasa.gov for this ongoing effort.
The current challenge for 2023 is centered around the SEPVAL working meetings in September and November 2023. For updated challenge event lists and rules of participation, please visit the SEPVAL webpage.
This effort supports the goals of ISWAT H3-01 SEP Model Validation section and the Integrated Solar Energetic Proton Alert/Warning System (ISEP) project. ISEP is a collaboration between the Space Radiation Analysis Group (SRAG) at NASA JSC, the Community Coordinated Modeling Center (CCMC) and NASA GSFC, and the Moon to Mars Space Weather Analysis Office (M2M) at NASA GSFC. The ISEP project was established to transition mature SEP models into operations and improve SRAG's SEP forecasting capability for human exploration missions beyond Low Earth Orbit.
GOALS
- Collaborate directly with modelers to collect SEP model predictions for as many SEP models as possible that exist in the research and operations communities
- Develop a generalized coding infrastructure to validate all types of SEP models for quantities important for operations and science.
- Create a validation pipeline that uses the same inputs required by CCMC's SEP Scoreboard
- Determine the best-uses of the available data sets for validation; Derive validation data sets
- Provide modelers with measurements that may be interpreted as the suprathermal seed particle spectrum prior to each event
CURRENT STATUS (updated 2023-06-06):
- A new set of SEP event and non-event lists and supporting information have been curated and released, along with Rules of Participation, for our 2023 Challenge. The SEPVAL working meetings (Sept in San Antonio, Texas, Nov in Toulouse, France) will focus on this challenge.
- An end-to-end validation framework has been demonstrated, name SPHINX
- An interactive interface, called SPHINX-Web, is being developed to allow users to explore validation results in an intuitive and flexible manner
- Values are derived from observations and flux time profile models with the publicly available code created for this effort (https://github.com/ktindiana/fetch-sep)
- A validation code that is capable of comparing observations and predictions and calculating a variety of metrics and skill scores has been developed (see status update from ISWAT 2021) and is currently being revised and optimized
- All modelers with models that can run in real-time are encouraged to contact CCMC about integration of their models into the SEP Scoreboard. (contact Leila Mays)
- For models that have flexibility in forecasting energy ranges, we are requesting model output in the following energy channels (see more at the bottom of the page):
- > 10, >100 and any other GOES-like channels
- Energies corresponding to the SOHO/EPHIN energy channels, particularly for the validation of SEP event onsets
Progress and Status
Current Status
A method to calculate observational quantities for validation purposes has been created (https://github.com/ktindiana/fetch-sep).
A draft code has been written which validates a variety of predicted values - see the Validation Framework page and talks describing the progress on the Conferences page.
Next Steps
Following the discussions and feedback at ISWAT 2021, we have identified these next steps in the validation effort.
A lot of progress has been made as of June 2023 in preparation for the next phase of the challenge, which will be supported by two SEPVAL working meetings. Progress on our action items is provided below.
The campaign organizers will:
- Create a balanced list of SEP events and non-events that will allow a statistically significant validation - Lists have been created for the 2023 SEPVAL workshops and are available in this Google drive.
- Focus on events in cycles 24 and possibly 23 to allow as many models as possible to participate - Lists focus on SC24 and SC25 for the 2023 challenge
- Provide a standard set of flare and CME inputs, including uncertainties on the CME parameters - all flare and CME values have been provided and CME parameters from DONKI have been double checked by the M2M office and updated if needed to ensure quality across the event lists
- Provide suprathermal seed spectra
- Create a pre-event definition for a forecast - i.e. specify a time stamp for each event after which no data may be used to make the forecast - Robust Rules of Participation have been developed for the 2023 challenge
- Models that make predictions using data prior to that time will be validated as operational-type forecasts - Addressed in Question 1 in the Rules
- Models that make predictions using data after that time may be validated from a scientific perspective, but the results would not inform operational usability - Addressed in Question 2 in the Rules
- Investigate the feasibility to provide multi-point observations for the SEP event list (e.g. STEREO A & B) for validation of 2D and 3D physics-based models - **STEREO A & B measurements have been added to the tools that prepare the observations for validation **
- Encourage all modelers with models that have the ability to operate in real-time to work with CCMC to be integrated into the SEP Scoreboard (contact Leila Mays, m.leila.mays@nasa.gov) - this would allow a true validation of forecasting capabilities using events going forward
- Work with the Moon to Mars Space Weather Analysis Office to develop an automated method to validate models running and sending forecasts in real-time to the SEP Scoreboard - Ongoing
- Add validation of the event-integrated differential fluence spectrum
- Improve formatting of the reports produced by the validation code - an interactive interface called SPINX-Web is being developed by Phil Quinn (SRAG) to view validation results in a clearer and more intuitive manner
- Revisit the "onset peak" algorithm in the OpSEP code
Approach for Validation Effort
A set of SEP events and non-events were chosen for this challenge along with flare, CME and other supporting information. The 2023 Challenge info is available in this Google drive.
Model results submitted to the challenge will be used as templates to build coding infrastructure that is capable of performing validation in a generic way that may be applied to all model types. The goal is a validation pipeline that is available to the community for validation of their models.
Model results will be compared with spacecraft measurements and validation metrics will be generated, however we acknowledge that this is a "partial" or "psuedo" validation due to the limited number of events. In the long-term, we intend to pursue creating long-baseline observational event lists (and negative event lists) for validation and work with modelers to submit results for a large number of historical cases. With enough test cases, meaningful validation metrics will be produced.
Additionally, compiling predictions from many models together in one place demonstrates the forecasting coverage that already exists (or could exist through R2O2R efforts) in the modeling community. We will also be able to highlight what information is missing.
The results of this effort will be compiled and presented at SHINE, ISWAT, and European Space Weather week and made available on this website for community review and feedback.
Participating Models
We are excited that many modelers in the community have shown interest in participating in this challenge.
ADEPT (White, Kahler, & Ling), AFRL PPS (White), COMESEP (Dierckxsens et al.), FORSPEF (Anastasiadis et al.), HESPERIA/REleASE (Posner, Malandraki, Kuehl, Karavalos), iPATH (Li, Hu), M-FLAMPA (Sokolov, Borovikov, Zhao), PCA model (Papaioannou), PROTONS (Balch), SAWS-ASPECS (Anastasiadis, Papaioannou, Vasalos), SEPMOD (Luhmann, Lee), SEPCaster (Li, Jin), SEPSTER (Richardson), SEPSTER2D (Bruno), SPARX (Marsh, Dalla), STAT (Linker, Schwadron et al.), UMASEP (Nunez), Zhang model.
Contact Katie Whitman (kathryn.whitman@nasa.gov) to participate.
Standard Inputs
At ISWAT 2020, the SEP modeling community expressed interest in using standard inputs for this challenge. We have compiled a list of flare and CME parameters for each SEP event. The CME parameters use only imagery of the CME eruption and do not benefit from arrival time information or further modeling. The parameters are either from M2M, NOAA, or CCMC forecasters.
The most up-to-date parameters are available for the 2023 challenge in this Google drive. - Use these! Please follow the 2023 Rules of Participation.
Our colleague Maher Dayeh (SWRI) has compiled suprathermal seed spectra for each SEP event for input into physics-based models. If it is useful to you, you may use the suprathermal spectrum information provided on the Seed Spectrum page. Hopefully spectra for all 2023 challenge event periods will be made available prior to the SEPVAL workshops.
All of these values can be found on the Model Input Parameters (2019 challenge) page. -** The 2023 challenge list replaces this list. Do not use these parameters, except for the suprathermal spectra.**
Requested Model Output
We request that modelers provide forecasts in GOES-like energy channels, particularly >10 and >100 MeV since these are used in operational decision-making by SRAG.
We also request, if possible, for models that forecast event start times to provide forecasts in SOHO/EPHIN energy channels. Comparison to SOHO/EPHIN data will allow for a more accurate assessment of model performance for SEP onsets. We expect that this request is most pertinent to physics-based models that produce flux time profiles.
| SOHO/EPHIN Channel | Geometric Mean [MeV] | Minimum Energy [MeV] | Maximum Energy [MeV] |
|---|---|---|---|
| P4 | 5.79 | 4.3 | 7.8 |
| P8 | 13.51 | 7.8 | 25 |
| P25 | 31.98 | 25 | 40.9 |
| P41 | 46.56 | 40.9 | 53 |
We understand that many models are developed to forecast for fixed energy values that may not correspond to the requested energy channels. That is fine and all model predictions are welcome for this effort. We will work one-on-one with modelers to find the best approach to validate each model.
Model Output Format
Modelers are encouraged to submit their predictions to this challenge using the CCMC json format. For models that produce flux time profiles, you may submit the time profiles directly and Katie will create the json files for validation.
Challenge Events
The current 2023 challenge information is stored in this Google drive and description of how to participate in the challenge can be found on the SEPVAL webpage. Please use these lists and parameters as they have been supplemented and updated compared to the 2019 and 2021/2022 lists.
See Events Page for 2019 challenge SEP event lists and information.
The SHINE 2022 Non-events are hosted on the SHINE website. The overall list is here and a detailed look at each time period is here.