Last Updated: 03/06/2024

Thermosphere Model Assessment and Improvement

About

ISWAT Team ID: G2A-01

Team Leads:

Communications: See the ISWAT G2A-01 team page for an up-to-date list of participants and latest news about the working team.

Validation Campaign in the CCMC CAMEL Framework

See Results on CAMEL
➞
See Available Data and Output Files On 'CAMEL-Data' Repo
➞

Introduction

Thermospheric density is the dominant source of uncertainty in the atmospheric drag. The diagram in Figure 1 shows how the data and model are involved in drag calculation. Thermosphere models estimate neutral density, composition, and temperature based on the solar and geomagnetic drivers. Physics-based models with the lower boundary located around the mesopause also need to specify the lower boundary condition representing the variability from the lower atmosphere. Biases from thermospheric models are amplified due to the satellite shape and aerodynamic model when calculating the drag force. This, in turn, introduces several error sources originating from the modeled thermospheric states in orbit computation. To make advances in orbit computation and determination, accurate specification and forecasting of thermosphere are required. Modelled neutral density must be validated against high-quality and high-spatial resolution neutral density datasets to identify strengths and weaknesses, establish error budgets, and improve the models after ingestion.

Figure 1: Diagram showing the data and model of a drag calculation.

However, there are still several challenges remaining in the validation of neutral density.

  1. Validation studies often involved only one or two events and a subset of models. This approach may not be robust or comprehensive.
  2. Staying updated with the growing number of models and their various versions remains challenging, especially with open-source models.
  3. Unified validation effort requires an online platform to keep track of the progress of model development.

To address these challenges, an assessment of thermosphere models under storm conditions was initiated within the COSPAR ISWAT framework, leveraging the international collaborative network. The neutral density validation project is built upon the CAMEL (Comprehensive Assessment of Models and Events Using Library Tools) framework developed and hosted by the NASA CCMC. This allows the community to systematically track the progress of thermosphere models over time.

This CAMEL campaign focuses on validating 1-D neutral density output from various models runs/solutions with observation data from GOCE, CHAMP, and/or GRACE_FO for different time periods. The thermosphere models are executed in-house using CCMC Runs-on-Request system and accessed. The model performance during the selected geomagnetically storm times from 2001 to 2022 are assessed for this study.

Methodology

An updated metric for thermospheric model assessment under geomagnetic storm conditions were proposed and implemented in the validation project (Sutton, 2018; Bruinsma et al., 2021). The metrics for comprehensive thermospheric model-data comparison are applied to establish the thermospheric model scorecard in the CAMEL framework.

Each storm is divided in four phases, two before and two after the minimum Dst value. The phases correspond to (1) pre-storm, (2) main, (3) recovery, and (4) post-storm. Figure 2 illustrates the four phases with respect to the time of minimum Dst. The pre-storm phase is used to de-bias the model with respect to the observations. A scaling factor is determined by computing the observed-to-computed (O/C) density ratio in the pre-storm phase, then applied to the model densities in all four phases. This de-biasing procedure is used to minimize the effect of non-storm related model errors on the assessment.

Figure 2: Dst value as a function of time during May 2015 storm. Red line (t0) indicates the time when Dst value reached its minimum during the time period.

After debiasing, the observed-to-computed (O/C) density ratio is re-computed for the main and recovery phases of each storm to express model’s skill to reproduce observations during the geomagnetically storm times. Density ratios of one indicate perfect duplication of the observations, i.e., an unbiased model that reproduces all features; deviation from unity points to under (larger than one) or overestimation (smaller than one). A model bias, i.e., the mean of the density ratios differs from unity, is most damaging to orbit extrapolation because it causes position errors that increase with time.

The standard deviation (Std. Dev.) of the density ratios, computed as percentage of the observation, represents a combination of the ability of the model to reproduce observed density variations, and the geophysical noise (e.g., waves, the short duration effect of large flares) and instrumental noise in the observations.

The mean and Std. Dev. of the O/C density ratios, due to their distribution, are computed in log space (Sutton, 2018; Bruinsma et al., 2021):

  • Average Observed-to-Compute Density (O/C) (= mean scaling factor of the model)

  • Average standard deviation (Std. Dev.) of Observed-to-Compute Density (O/C)

where N is the total number of observations.

List of solutions/runs settings

  • CTIPe 4.1 Model Run/Solution Setting 1
  • DTM 2013 Model Run/Solution Setting 1
  • DTM 2020 Model Run/Solution Setting 1
  • DTM 2020 Model Run/Solution Setting 2
  • GITM 21.11 Model Run/Solution Setting 1
  • JB2008 Model Run/Solution Setting 1
  • MSIS 2.0 Model Run/Solution Setting 1
  • TIEGCM 2.0 Model Run/Solution Setting 1
  • WAMCCM 2.2 Model Run/Solution Setting 1
  • WAM-IPE 1.0 Model Run/Solution Setting 1
  • Bruinsma, S., Sutton, E., Solomon, S. C., Fuller-Rowell, T., & Fedrizzi, M. (2018). Space weather modeling capabilities assessment: Neutral density for orbit determination at low Earth orbit. Space Weather, 16, 1806–1816. https://doi.org/10.1029/2018SW002027
  • Sutton EK. 2018. A new method of physics-based data assimilation for the quiet and disturbed thermosphere. Space Weather 16: 736–753. https://doi.org/10.1002/2017SW00178.
  • Bruinsma S, Boniface C, Sutton EK & Fedrizzi M 2021. Thermosphere modeling capabilities assessment: geomagnetic storms. J. Space Weather Space Clim. 11, 12. https://doi.org/10.1051/swsc/2021002.

Sections in this page