Related Links | Frequently Asked Questions | Community Feedback | Downloads | Sitemap
CCMC Home | CCMC Stakeholders | Our Team | Publications | Meetings and Workshops | Concept of Operations
Models At A Glance | ModelWeb Catalog and Archive
Request Procedures | Generate Input Data Files & Parameters | Movies on Request | About the Run Process | Publications Policy
Search run database | Request run output | Special events | Kameleon Software | Space Weather Explorer | Publications policy
Instant Model Run
Forecasting Support tools | iSWA | DONKI | Mission Support | Experimental Real Time Simulations | Operational Geospace Model Validations
Intl Forum | GEM Challenge | CEDAR ETI Challenge | GEM-CEDAR Challenge | SHINE Challenge | CME Arrival Time Scoreboard | Flare Scoreboard | SEP Scoreboard | IMF Bz Scoreboard
Educational materials &activities | Space Weather REDI Initiative | SW REDI Bootcamp | Student Research Contest | Tutorial at CEDAR | Forecaster Tools
Missions near Earth/in Earth-orbit | MMS | Van Allen Probes | THEMIS | MESSENGER | STEREO | Spitzer | MAVEN | MSL | Dawn | Kepler | EPOXI | Juno | CASSINI | Voyager | New Horizons | Sounding Rockets | International
Research Community Support | CCMC Workshops | NASA Robotic Mission Operator Workshops | LWS Support | Exo-CCMC | DREAM2 Support | HELCATS Support
iSWA | DONKI | Kameleon | StereoCat | EEGGL | CME Scoreboard | SEP Scoreboard | FLR Scoreboard | SEA5
Real-time Forecasting Methods Validation: Flare Scoreboard

CCMC is in the implementation phase of the "Flare Scoreboard" together with Sophie Murray of the UK Met Office and the international research community. The flare scoreboard is an automated system such that model/method developers upload their predictions automatically uploaded to an anonymous ftp which will be parsed by the system. The forecasts are shown on an interactive display of SDO/AIA or HMI images, and will also be displayed together on a graph of probability vs. time.

Quick Links:
Latest News:
Active region forecasts are available in the Flare Scoreboard cygnet.
Forecast probability vs time plots are available in the Flare Scoreboard cygnet.
See the agenda/materials of the ESWW15 working meeting: Flare Forecasting: where are we and where should we be going?
➡ Flare scoreboard team members are involved in the ISEE/PSTEP "BenchMarks for Operational Flare Forecasts Workshop"
➡ The Flare scoreboard is part of the the Solar Flare Working Team in the Community-wide International Forum for Space Weather Modeling Capabilities Assessment.
See the agenda/materials of the ESWW13 working meeting: Community-wide space weather Scoreboards: Research assessment of real-time forecasting models and techniques.

Flare scoreboard planning group:
Leads: Sophie Murray (TCD), Jesse Adries, Veronique Delouille (SIDC)
Please email Sophie Murray, Masha Kuznetsova, and Leila Mays with your feedback which will be shared with the flare scoreboard planning group.

Mike Terkildsen, Graham Steward (Australia Bureau of Meteorology, Space Weather Services), K.D. Leka (NWRA), Jordan Guerra, Shaun Bloomfield (Northumbria University), Masha Kuznetsova, M. Leila Mays (CUA/GSFC)

Currently registered models and particpating partners:
Automatic McIntosh-based Occurrence probability of Solar activity
Automated Solar Activity Prediction
Automatic Solar Synoptic Analyzer
Data-driven probabilistic flare forecast model
MAG4 LOS and Vector Magnetogram Forecasts (four products)
Met Office  
Space Weather Forecast (full disk) and Sunspot Region Summary
SIDC human operator moderated
Flare Prediction System
Universal Forecast Constructor by Optimized Regression of INputs

Flare Scoreboard Probability Timeseries Screenshot

First version. Feedback welcome.

Proposed forecast submission file format and Registration Questions:

Flare scoreboard functions and workflow:
     Participants respond the questionnaire below to get started (see below).
     Participants produce probabilistic flare forecasts conforming to the file format (XML or plain text; see below). Additionally, occasional participants can provide forecasts by filling out a form, similar to the CME scoreboard. (Please contact Leila Mays if you would like to use this option).
     Flare forecast files are accepted by anonymous ftp, within the relevant subdirectory. Forecast files will be processed at least every hour.      The minimum requirement for participation is a probabilistic Full Disk Forecast for M and X bins. Active Region forecasts and C bins are also accepted.
     Required forecast information includes: Forecasting Method, Issue Time, Prediction Window Start/End Time, M and X flare Probability. For Active Region Forecasts, Region number, Location, and Time is also required. See below for details.
     All information is stored in a database and is made accessible to users. Click here to learn how to download flare forecast files from the database.
      CCMC Rules of the Road apply: CCMC requests that users notify the CCMC, flare model/technique developers and submitting forecasters before performing validation studies with the Flare Scoreboard database. It is recommended that such validation studies be performed with the knowledge and collaboration of developers and submitting forecasters/researchers.
     Probabilistic forecasts are displayed side by side on the "Flare Scoreboard" website.

Proposed forecast submission file formats:
Flare forecasts will be accepted via three avenues (click each for proposed template):
    (1) XML: XML schema, , template, full example
    (2) Plain text
    (3) manual form input

Example plain text templates and description:
Full Disk Forecast: (red=required fields, blue=optional, use four dashes if empty: ----)
Forecasting method: MAG4
Issue Time: 2013-10-23T12:00Z
Prediction Window Start Time: 2013-10-23T12:00Z
Prediction Window End Time: 2013-10-24T12:00Z
Probability Bins: M+
Input data: ----

#Full Disk Forecast
#X_prob   X_CI_Lower  Upper  X_Level     M_prob   M_CI_Lower  Upper  M_Level     C_prob   C_CI_Lower  Upper  C_Level
0.4000          ----   ----     ----     0.6800         ----   ----     ----       ----         ----   ----     ----

Full Disk and Active Region Forecast with optional fields filled in:
Forecasting method: MAG4
Issue Time: 2013-10-23T12:00Z
Prediction Window Start Time: 2013-10-23T12:00Z
Prediction Window End Time: 2013-10-24T12:00Z
Probability Bins: M+
Input data: SDO/HMI LOS_Magnetogram

#Full Disk Forecast
#X_prob   X_CI_Lower  Upper  X_Level     M_prob   M_CI_Lower  Upper  M_Level     C_prob   C_CI_Lower  Upper  C_Level
0.4000        0.3000 0.4800        3     0.6800       0.6500 0.7000        3     0.7500       0.7100 0.7700        3

#Region Forecast
#Model_Region_No.  Time              Location  NOAA_AR_No.  Time              Location   X_prob   X_CI_Lower  Upper  X_Level     M_prob   M_CI_Lower  Upper  M_Level     C_prob   C_CI_Lower  Upper  C_Level
06                 2013-10-23T00:30Z   S10E00        12192  2013-10-23T00:30Z   S12E00   0.3900       0.3100 0.4500        3      0.7000      0.6500 0.7500        3      0.7200      0.7000 0.7400        3
08                 2013-10-23T00:30Z   N05W35        12193  2013-10-23T00:30Z   N05W37   0.0050       0.0000 0.0500        1      0.0200      0.0000 0.1000        2      0.0500      0.0100 0.0800        2
File notes:
     Color coding: blue=optional, use four dashes if empty: ----.
     "Probability Bins" should be specified as "M" or "M+" depending on whether the method forecasts "M1.0-9.9" or "M and above"
     Probabilities and uncertainties should be displayed as decimal, not percent, up to four decimal places.
     Forecast files will be processed every six hours based on the forecast Issue Time of 0, 6, 12, and 18 UT. The files will be accepted within a file submission time cushion (±0.5h), such that event if the file is delayed, it is the forecast issuance time that is used.

Model/Technique Registration (*=required):
Information about your group/project:
    Name(s) and e-mail (s) (please list primary contact first)*:
    Associated Institution/Project name/Group name*:
    Website url(s):

Information about your method:
    Forecasting method name*:
    Shorthand unique identifier for your method (methodname_version, e.g. ASSA_1, ASAP_201201):
    Short description*:

Further Model Details:
    (1)* Please specify if the forecast is human generated, human generated but model-based, model-based, or other.

    (2)* Does your flare prediction method forecast “M1.0-9.9” or “M and above”? Would it be difficult for you to adapt your method from one to another? Which forecast binning method do you prefer?

    (3)* How do you specify active regions in your model? Do you relate their location to other schemes such as NOAA or Catania? If so, what is your criteria to relate them?

    (4)* Uncertainties given as an upper and lower bound is an optional field. Does your model provide uncertainties for the forecasted probability?
If yes, what percentiles do you use to determine your upper and lower bound? If you are using the XML format, multiple percentiles of the probability density distribution can be specified.

    (5)* For each forecast, what prediction window(s) does your method use? E.g. Does your method predict for the next 24 hours, 48, and 72 hours or the next 24 hours, 24-48 hours, 48-72 hours? (This information is useful for displaying only comparable methods together).

    (6)* For how many degrees from disk center is your full disk forecast valid for? E.g. 60 degrees; or 90 degrees (valid for entire disk).

    (7)* Calibration levels are optional fields. Do you have calibration for the probabilities from your model? E.g. is a 40% forecast a “high” probability for your method?
If yes, please provide the probability ranges for what is considered a “low”, “medium”, or “high” level. This can be provided in a separate file, or as part of the main template, labeled “X_Level”, “M_Level”, and “C_Level” fields which can take the values 1, 2, or 3. These levels represent probability calibration for a model for each flare class. With 1 meaning “low” probability, 2 is “medium” probability, and 3 is “high” probability.

        For example in the MAG4 model:
        Probabilities in range → Level
                   Prob < 0.02 → 1
        0.02 ≤ Prob < 0.18 → 2
                   Prob ≥ 0.18 → 3

National Aeronautics and Space Administration Air Force Materiel Command Air Force Office of Scientific Research Air Force Research Laboratory Air Force Weather Agency NOAA Space Environment Center National Science Foundation Office of Naval Research

| | Privacy, Security Notices | CCMC DATA Collection Consent Agreement

CCMC logo designed by artist Nana Bagdavadze