Metrics Studies
Metrics studies are standardized, repeatable comparisons of model output to a measurable parameter of interest. They allow us to objectively compare the performance of models with comparable output, to measure the improvement of model capabilities over time, or to measure model usefulness for operations in comparison with some simple standard model.
The result is usually expressed as a skill score, which measures the performance of a model against the standard model based on modeling of one particular parameter. This allows us to construct a simple, “one number” measure of model capabilities. When the model's skill score(M) is less than 0, the model is worse than standard model; M = 0 means the model is as good as standard one; 0 < M < 1: the model is better than the standard model; M = 1 signifies the perfect score.
Metrics Studies, Verification and Validation - overview of CCMC metrics activites (presented at the 2005 CCMC Workshop, October 2005, Clearwater Beach, FL)
| Examples of CCMC Metrics Studies: |
Model Name |
Model Output Parameters |
Measurements |
Standard Models |
Status/ Report/ Presentation |
Velocity and density at ACE location |
ACE velocity and density |
mean, persistence |
Completed |
|
| In progress Sample ENLIL/Ace comparison |
||||
Connection to active regions |
Energetic particles precipitation (event/no event) |
persistence |
Started |
|
| Started |
||||
Magnetopause position as boundary surface between open and closed filed lines. . |
Magnetopause crossings by satellites |
persistence |
Started |
|
| Magnetic perturbations at geosynch. orbit | GOES magnetic field data | no perturbations |
Started |
|
Boundary between open and closed field lines mapped to ionosphere |
Polar cap boundary observed by POLAR and IMAGE |
mean |
Completed Presentation |
|
Polar cap from field-aligned current pattern |
||||
| Particle fluxes at geosynchronous orbit. | Particle fluxes observed by LANL geosynch. sat. | mean |
||
persistence, |
Completed Report |
|||
| Ground magnetic perturbations calculated from ionospheric potential pattern | Ground magnetometer data (H component of magnetic perturbations) | no perturbations |
Completed Report |
|
In progress |
||||
In progress |
||||
Electron density parameters (vertical profiles, TEC, NmF2) |
Incoherent scatter radars, GPS stations, ionosonds | mean |
Started |
|
First community metrics challenge evaluation |
E x B drift in ionosphere | Velocity measurements from DMSP F13 | zero velocity |
| Presentations and Publications on Metrics |
- Kuznetsova, M., "Metrics Studies, Verification and Validation", 2005 CCMC Workshop, Clearwater Beach, FL, October 11-14, 2005
- Rastätter, L., Hesse, M., Gombosi, T. I., Raeder, J., Weimer, D., "Polar cap size metrics study at CCMC", Eos Trans. AGU, 86(52), Fall Meet. Suppl., Abstract SA31A-0341, 2005.
- Rastaetter, L., Kuznetsova, M. M., Hesse, M., Gombosi, T. I., Raeder, J., Weimer, D., "Polar cap size metrics study at CCMC", American Geophysical Union, Fall Meeting 2005, abstract #SA31A-0341
- Narock, A., L. Rastaetter, M. Kuznetsova, M. Hesse, "Realtime magnetosphere simulation at CCMC," Eos Trans. AGU, 85(47), Fall Meet. Suppl., Abstract SA43B-06, 2004.
- Keller, K, M. Hesse, M. Kuznetsova, L. Rastaetter, and A. Falasca, "Assessing the Performance of Space Weather Models Using Metrics," Eos Trans. AGU, 85(17), Jt. Assem. Suppl., Abstract SM21A-12, 2004.
- Keller, K., , M. Hesse, M. Kuznetsova, L. Rastaetter, and A. Narock, "Validation of Heliosphere Models," SHINE 2004 Workshop, Big Sky, Montana, June 27 - July 2, 2004.
- Keller, K., Hesse, M., Rastaetter, L., Kuznetsova M., Review of Presently Employed Metrics at the CCMC, CCMC Workshop, 2003
- Keller, K., M. Hesse, L. Rastätter, M.M. Kuznetsova, T. Moretto, "Using metrics to measure the performance of space weather models," 2003 Space Weather Week, Boulder, CO, May 19-22, 2003
- Keller, K., M. Hesse, L. Rastätter, M.M. Kuznetsova, and T. Moretto, "Measuring the performance of space weather models, Effects of Space Weather on Technology Infrastructure (ESPRIT-2003)," Rhodes, Greece, March 25-29, 2003.
- Hesse M., "The First Magnetospheric Space Weather Metrics Challenge", GEM Workshop, Snowmass, Colorado, June 17-22, 2001
