SEPVAL 2023
SEP Model Validation Working Meeting
The SEP Model Validation Working Meeting (SEPVAL) will focus on the validation of SEP models drawing upon a multi-year validation effort started in 2018 through the SHINE, ISWAT, and ESWW workshops. During the SEPVAL 2023 workshops, one in the U.S. and one in Europe, developers of solar energetic particle (SEP) prediction models will work with space agency end users to assess SEP model performance, establish standards, and develop a framework for SEP model validation.
SEPVAL is organized by CCMC, SWRI, and NASA SRAG. Contact Katie Whitman (Kathryn.whitman@nasa.gov) for questions and further information.
United States
The United States SEPVAL was held at the Southwest Research Institute in San Antonio, TX from September 5 – 7, 2023.
Europe
The European SEPVAL will be held at the European Space Weather Workshop (ESWW) location in Toulouse, France on Saturday November 18 (full day) & 19 (half day), 2023 prior to ESWW. SEPVAL will be part of the mini-ISWAT working meeting being organized in conjunction with ESWW. We will have hybrid capabilities. The link will be posted in the Agenda when it becomes available.
Logistics
United States
- Workshop Dates: September 5 – 7, 2023 (Tuesday – Thursday), note that September 4 is the U.S. holiday Labor Day
- Location: Southwest Research Institute, 6220 Culebra Rd, San Antonio, TX 78238
- Deadline to Submit Forecasts: All forecasts for the US challenge must be submitted in the appropriate format to Katie Whitman by the end of SHINE, August 11, 2023. Event lists linked below.
- Thank you to all who participated in the SEPVAL US working meeting! Twenty people attended in person and 20+ more joined online each day. It was extremely informative regarding the perspective of NASA and NOAA on validation and forecasting with SEP models. The hands-on sessions resulted in a lot valuable interaction with and feedback about the validation infrastructure being developed through this effort.
Europe
- Workshop Dates: Saturday (all day) and Sunday (morning), November 18 - 19, 2023
- Location: Pierre-Baudis Congress Center (ESWW venue)
- Hotel: Please arrange your ESWW hotel to arrive a few days early to attend our meeting
- Registration: We will be part of the mini-ISWAT side meeting associated with ESWW. Please register for mini-ISWAT when you register for ESWW. Find more info about mini-ISWAT on the ISWAT website.
- Registration Deadline: With ESWW - You do not need to register to join virtually.
- Deadline to Submit Forecasts: All forecasts for the European challenge must be submitted in the appropriate format to Katie Whitman by October 15, 2023. Event lists linked below.
Attendance
SEPVAL will be a primarily in-person working meeting with some sessions available virtually. Sessions will include presentations, discussion sessions, and time built in to work directly with colleagues. The plan for the working meetings is to make sessions as interactive as possible and to make progress on specific tasks as a community. The interactive SPHINX-Web tool will be available for people to use during the working meeting to investigate validation of participating models.
Agenda
Half of the workshop will focus on operational use of SEP models and the new infrastructure being developed for the R2O transitioning of models to ops and the other half will focus on validation.
Topics include:
- Approach to SEP model validation and status of validation tool
- SEP Scoreboard (real time model forecasting)
- How are SEP models used in real time ops by M2M and SRAG? (Met Office, ESA)
- CCMC-SWPC Proving Ground (ACE - Architecture for Evaluation)
- SWPC's Validation Test Bed
- ESA's R2O and forecasting infrastructure
- SEP model developer experience in the transitioning to realtime/ops
- Interactive discussion about validation results from challenge submissions (see below) e.g. What are we doing wrong? What are we doing right? Most informative metrics?
- Presentations by modelers about challenges, needs, insights
- Cross-model comparisons
- Dedicated time for collaboration, analysis, and code development
- Next steps
Rules and Participation in the Challenge
SEP model developers are invited to participate in the validation challenge by submitting predictions for a specific set of events following a specific set of rules. Detailed Rules of Participation are available along with event lists, proton observations, and movies of each event period.
Submitted forecasts will be validated by the Space Radiation Analysis Group (poc Katie Whitman, Kathryn.whitman@nasa.gov) using the validation code being developed there.
The challenge will aim to answer the questions:
1. How well do models perform following their default workflows? This question aims to assess how models would perform if running in a real time environment such as the SEP Scoreboard with specific inputs, without knowledge of the future and no human intervention to tune parameters.
2. What is a model’s best performance? Produce predictions with no restrictions. Turning all the knobs, is it possible to get it right? What can we learn about SEP events from our best-matching predictions?
3. How do models perform with respect to physical parameters, such as eruption location, CME speed, etc? Submissions to 1. and 2. will be filtered to determine the strengths and weaknesses of our current predictive capabilities.
Model developers are asked to submit their forecasts in CCMC’s JSON format for the SEP Scoreboard as this is the format required by the validation code. It is suggested that participants submit a representative forecast to Katie (Kathryn.whitman@nasa.gov) in any format and she will create an example JSON file that you can use as a template for all submitted forecasts.
More information about the Challenge can be viewed at: https://ccmc.gsfc.nasa.gov/challenges/sep/
Challenge Event Lists
Two lists have been curated for this challenge:
- 33 SEP Events
- 30 Non-Event periods
Please see the FAQ at the end of the Rules of Participation presentation to see reasoning behind the choices of the SEP event and non-event periods.
The curated SEP lists are located in the google drive in the EventLists folder. This lists contain proton, flare, and CME information. There are 3D CME parameters from DONKI and M2M along with 2D CME parameters from CDAW and CACTus.
The M2M Office put in a lot of work to double check all 3D CME parameters in the lists and updated many of them, so note that some CME parameters for the original SHINE events (from 2019 challenge) and non-events (from 2022 challenge) have changed. They also provided information the last image time used in the fit, which we will use in this challenge to estimate forecast issue time.
SEPVAL Working Meeting Goals
- Enhance communications between the operations and research communities
- Create a benchmark data set of SEP event and non-event time periods. Provide flare, CME, proton intensity and other supporting information to the community.
- Collect SEP model predictions following a specific set of rules to enable meaningful assessment of performance and the possibility for cross-model comparisons
- Create a validation code and infrastructure that will be made available for public use
- Publish a journal paper(s) reporting on the outcomes of the workshops
The SPHINX Validation Code
The generalized validation code, Solar Particles in the Heliosphere validation INfrastructure for SpWx (SPHINX), is being developed at NASA SRAG in collaboration with M2M and CCMC to validate forecasts made by all models in the SEP community. Additionally, SPHINX will be used to validate forecasts made in real time to the SEP Scoreboard.
The SPHINX workflow includes the preparation of observational data, ingestion of forecasts and matching to observations, comparison of observed and forecasted values, calculation of metrics, and the production of a validation report in both a pdf format and an interactive web interface, SPHINX-Web.
SPHINX: A gatekeeper that devours all who are not able to answer her riddle.
Image credit: DALLE 2