SEPVAL 2023

SEP Model Validation Working Meeting

The SEP Model Validation Working Meeting (SEPVAL) will focus on the validation of SEP models drawing upon a multi-year validation effort started in 2018 through the SHINE, ISWAT, and ESWW workshops. During the SEPVAL 2023 workshops, one in the U.S. and one in Europe, developers of solar energetic particle (SEP) prediction models will work with space agency end users to assess SEP model performance, establish standards, and develop a framework for SEP model validation.

SEPVAL is organized by CCMC, SWRI, and NASA SRAG. Contact Katie Whitman (Kathryn.whitman@nasa.gov) for questions and further information.

United States

The United States SEPVAL will be held at the Southwest Research Institute in San Antonio, TX from September 5 – 7, 2023.

Europe

The European SEPVAL will be held at the European Space Weather Workshop (ESWW) location in Toulouse, France on Saturday November 18 (full day) & 19 (half day), 2023 prior to ESWW. SEPVAL will be part of the mini-ISWAT working meeting being organized in conjunction with ESWW.

Logistics

United States

Workshop Dates: September 5 – 7, 2023 (Tuesday – Thursday), note that September 4 is the U.S. holiday Labor Day
Location: Southwest Research Institute, 6220 Culebra Rd, San Antonio, TX 78238
Hotel: There is not a specific workshop hotel, but there are a number of nearby, reasonably priced hotels along I-410. Specific recommendations will be added as the workshop approaches.
Registration: deadlines and details TBD
Deadline to Submit Forecasts: All forecasts for the US challenge must be submitted in the appropriate format to Katie Whitman by the end of SHINE, August 11, 2023. Event linked below.

Europe

Workshop Dates: Saturday November 18 - 19, 2023
Location: Pierre-Baudis Congress Center (ESWW venue)
Hotel: Please arrange your ESWW hotel to arrive a few days early to attend our meeting
Registration: We will be part of the mini-ISWAT side meeting associated with ESWW and there may be an option to pay a registration fee as part of your registration for ESWW. Details will be posted here and on the ISWAT website.
Deadline to Submit Forecasts: All forecasts for the European challenge must be submitted in the appropriate format to Katie Whitman by October 15, 2023. Event lists linked below.

Attendance

SEPVAL will be a primarily in-person working meeting with presentations, discussion sessions, and time built in to work directly with colleagues. We will aim to make at least part of the workshop available virtually. The plan for the working meetings is to make sessions as interactive as possible and to make progress on specific tasks as a community.

Agenda

Detailed agendas will be posted here as the working meeting approaches. Potential discussion topics include:

Approach to SEP model validation and status of validation tool developed at SRAG
SEP Scoreboard (real time model forecasting)
How are SEP models used in real time ops by M2M and SRAG? (Met Office, ESA, etc?)
CCMC-SWPC Proving Ground (ACE - Architecture for Evaluation)
SWPC's Validation Test Bed
Interactive discussion about validation results from challenge submissions (see below) e.g. What are we doing wrong? What are we doing right? Most informative metrics?
Presentations by modelers about challenges, needs, insights
Cross-model comparisons
Dedicated time for collaboration, analysis, and code development
Next steps

Rules and Participation in the Challenge

SEP model developers are invited to participate in the validation challenge by submitting predictions for a specific set of events following a specific set of rules. A detailed description of the rules of participation are available in this google drive along with event lists, proton observations, and movies of each event period.

Submitted forecasts will be validated by the Space Radiation Analysis Group (poc Katie Whitman, Kathryn.whitman@nasa.gov) using the validation code being developed there.

The challenge will aim to answer the questions:

1. How well do models perform following their default workflows? This question aims to assess how models would perform if running in a real time environment such as the SEP Scoreboard with specific inputs, without knowledge of the future and no human intervention to tune parameters.

2. What is a model’s best performance? Produce predictions with no restrictions. Turning all the knobs, is it possible to get it right? What can we learn about SEP events from our best-matching predictions?

3. How do models perform with respect to physical parameters, such as eruption location, CME speed, etc? Submissions to 1. and 2. will be filtered to determine the strengths and weaknesses of our current predictive capabilities.

Model developers are asked to submit their forecasts in CCMC’s JSON format for the SEP Scoreboard (https://ccmc.gsfc.nasa.gov/scoreboards/sep/#submission-file-format-information) as this is the format required by the validation code. It is suggested that participants submit a representative forecast to Katie (Kathryn.whitman@nasa.gov) in any format and she will create an example JSON file that you can use as a template for all submitted forecasts.

More information about the Challenge can be viewed at: https://ccmc.gsfc.nasa.gov/challenges/sep/

Challenge Event Lists

Two lists have been curated for this challenge:

30 SEP Events
33 Non-Event periods

Please see the FAQ at the end of the Rules of Participation presentation available on the google drive to see reasoning behind the choices of the SEP event and non-event periods.

The curated SEP lists are located in the google drive in the EventLists folder. This lists contain proton, flare, and CME information. There are 3D CME parameters from DONKI and M2M along with 2D CME parameters from CDAW and CACTus.

The M2M Office put in a lot of work to double check all 3D CME parameters in the lists and updated many of them, so note that some CME parameters for the original SHINE events (from 2019 challenge) and non-events (from 2022 challenge) have changed. They also provided information the last image time used in the fit, which we will use in this challenge to estimate forecast issue time.

SEPVAL Working Meeting Goals

Create a benchmark data set of SEP event and non-event time periods. Provide flare, CME, proton intensity and other supporting information to the community.
Collect SEP model predictions following a specific set of rules to enable meaningful assessment of performance and the possibility for cross-model comparisons
Create a validation code and infrastructure that will be made available for public use
Publish a journal paper(s) reporting on the outcomes of the workshops

The SPHINX Validation Code

The generalized validation code, Solar Particles in the Heliosphere validation INfrastructure for SpWx (SPHINX), is being developed at NASA SRAG in collaboration with M2M and CCMC to validate forecasts made by all models in the SEP community. Additionally, SPHINX will be used to validate forecasts made in real time to the SEP Scoreboard.

The SPHINX workflow includes the preparation of observational data, ingestion of forecasts and matching to observations, comparison of observed and forecasted values, calculation of metrics, and the production of a validation report in both a pdf format and an interactive web interface, SPHINX-Web.

SPHINX: A gatekeeper that devours all who are not able to answer her riddle.