Related Links | Frequently Asked Questions | Community Feedback | Downloads | Sitemap
CCMC Home | CCMC Stakeholders | Our Team | Publications | Meetings and Workshops | Concept of Operations
Models At A Glance | ModelWeb Catalog and Archive
Request Procedures | Generate Input Data Files & Parameters | Movies on Request | About the Run Process | Publications Policy
Search run database | Request run output | Special events | Kameleon Software | Space Weather Explorer | Publications policy
Instant Model Run
Forecasting Support tools | iSWA | DONKI | Mission Support | Experimental Real Time Simulations | Operational Geospace Model Validations
Intl Forum | GEM Challenge | CEDAR ETI Challenge | GEM-CEDAR Challenge | SHINE Challenge | CME Arrival Time Scoreboard | Flare Scoreboard | SEP Scoreboard | IMF Bz Scoreboard
Educational materials &activities | Space Weather REDI Initiative | SW REDI Bootcamp | Student Research Contest | Tutorial at CEDAR | Forecaster Tools
Missions near Earth/in Earth-orbit | MMS | Van Allen Probes | THEMIS | MESSENGER | STEREO | Spitzer | MAVEN | MSL | Dawn | Kepler | EPOXI | Juno | CASSINI | Voyager | New Horizons | Sounding Rockets | International
Research Community Support | CCMC Workshops | NASA Robotic Mission Operator Workshops | LWS Support | Exo-CCMC | DREAM2 Support | HELCATS Support
iSWA | DONKI | Kameleon | StereoCat | EEGGL | CME Scoreboard | SEP Scoreboard | FLR Scoreboard | SEA5

Metrics Studies

Metrics studies are standardized, repeatable comparisons of model output to a measurable parameter of interest. They allow us to objectively compare the performance of models with comparable output, to measure the improvement of model capabilities over time, or to measure model usefulness for operations in comparison with some simple standard model.

The result is usually expressed as a skill score, which measures the performance of a model against the standard model based on modeling of one particular parameter. This allows us to construct a simple, “one number” measure of model capabilities. When the model's skill score(M) is less than 0, the model is worse than standard model; M = 0 means the model is as good as standard one; 0 < M < 1: the model is better than the standard model; M = 1 signifies the perfect score.

Metrics Studies, Verification and Validation - overview of CCMC metrics activites (presented at the 2005 CCMC Workshop, October 2005, Clearwater Beach, FL)

Examples of CCMC Metrics Studies:
Model Name
Model Output Parameters
Measurements
Standard Models
Status/
Report/
Presentation
Velocity and density at ACE location

ACE velocity and density

mean,
persistence

Completed
Report

Connection to active regions
Energetic particles precipitation (event/no event)
persistence
Started
Started
Magnetopause position as boundary surface between open and closed filed lines. .

Magnetopause crossings by satellites

persistence
Started
Magnetic perturbations at geosynch. orbit GOES magnetic field data
no perturbations
Started
Boundary between open and closed field lines mapped to ionosphere
Polar cap boundary observed by POLAR and IMAGE
mean
Completed
Presentation
Polar cap from field-aligned current pattern
Particle fluxes at geosynchronous orbit. Particle fluxes observed by LANL geosynch. sat.
mean

persistence,
average,
27-day recurrence

Completed
Report
Ground magnetic perturbations calculated from ionospheric potential pattern Ground magnetometer data (H component of magnetic perturbations)
no perturbations
Completed
Report
In progress
In progress

Electron density parameters (vertical profiles, TEC, NmF2)

Incoherent scatter radars, GPS stations, ionosonds
mean
Started

First community metrics challenge evaluation

E x B drift in ionosphere Velocity measurements from DMSP F13
zero velocity

 

Presentations and Publications on Metrics

National Aeronautics and Space Administration Air Force Materiel Command Air Force Office of Scientific Research Air Force Research Laboratory Air Force Weather Agency NOAA Space Environment Center National Science Foundation Office of Naval Research

| | Privacy, Security Notices

CCMC logo designed by artist Nana Bagdavadze