NASA/NOAA MOU Annex
Fall AGU 2018 Abstract
NOAA SWPC / CCMC Partnership to Validate Heliospheric Models
CCMC: M. Leila Mays, Peter Macneice, Aleksandre Taktakishvili,
NOAA/SWPC: Eric Adamson, Vic Pizzo, Doug Biesecker
In 2017 NOAA SWPC and CCMC started a new project under an annex to a memorandum of understanding between NASA and NOAA. The purpose of this project is to assess improvements in CME arrival time forecasts at Earth using the Air Force Data Assimilative Photospheric Flux Transport (ADAPT) model driven by data from the Global Oscillation Network Group (GONG) ground observatories. These outputs are then fed into the coupled Wang-Sheeley-Arge (WSA) - ENLIL model and then compared to the current operational version of WSA-ENLIL (without ADAPT). The project is performed in close collaboration with model developers.
SWPC has selected a set of 33 historical events over the period of three years from 2012-2014. The overall two-year project consists of multiple simulation validation studies for the entire event set: (a) benchmark (current operational version) (b) time-dependent sequence of GONG maps driving WSA-ENLIL (c) single test simulation of a time-dependent sequence of GONG maps driving ADAPT-WSA-ENLIL (d) single GONG map driving ADAPT-WSA-ENLIL (e) time-dependent sequence of GONG maps driving ADAPT-WSA-ENLIL. After each stage, the performance of the new simulation results will be compared to the benchmark (stage a) and to other previous stages. Here we present preliminary validation results for stages (a)-(c) that test the performance when using time-dependent backgrounds.
Download this presentation [PPTX]
Final Report [PDF] (delivered 2020-05-01)
Final Report Simulation Downloads
Full download=evo *dat, evo graphics, bnd graphics, and contour plot graphics for all runs. Does not include full data cube netCDF files but these available upon request.
Downloads for Annex Tasks (a) and (b)
⊳ Download plasma parameters time-series plots comparing simulation results for (a-b), 1(a-b), 2(a-b), 3(a-b), (d-e) at Earth:
⊳ Full plots | Full plots - speed only
⊳ Zoomed in plots near the CME arrival time | Zoomed plots - speed only
⊳ (a-b) Full download (9.3 GB) GONGb WSA 2.2 ENLIL 2.6.2 a3b2 (operational prior to May 2019)
⊳ 1(a-b) Full download (1.5 GB) GONGb WSA 2.2 ENLIL 2.9e a8b1 (operational after May 2019)
⊳ 2(a-b) Full download (1.5 GB) GONGz WSA 4.5 ENLIL 2.6.2 a3b2
⊳ 3(a-b) Full download (1.5 GB) GONGz WSA 4.5 ENLIL 2.9e a8b1
Downloads for Annex Tasks (c), (d) and (e)
GONGz ADAPT WSA 4.5 ENLIL 2.9e a8b1:
⊳ (d-e): Download time-series plots comparing simulation results for (d-e) at Earth:
⊳ Full plots | Full plots - speed only
⊳ Zoomed in plots near the CME arrival time | Zoomed plots - speed only
⊳ (d-e): Full download (24 GB)
⊳ Input file listing for (d) and (e)
⊳ (c): Full download of single event resolution test (2014-08-16), and bnd graphics download.
⊳ Download CME arrival time error statistics for all simulations.
⊳ Bonus plots (not included in report) of CME arrival time error vs Ambient Solar Wind Speed Error: unfiltered, filtered by CME input speed: 500 km/s, 800 km/s, 1100 km/s, 1400 km/s
The ambient solar wind speed error was calculated as the average observed (1 hour OMNI) solar wind speed subtracted from the simulated ambient solar wind speed for the first day of each simulation.
⊳ Logs of input map counts: (a) (b) 1(a) 1(b) 2(a) 2(b) 3(a) 3(b) (d) (e)
General project information:
- Results: Year 1 interim report [PDF]
- Latest project timeline (201902) [PDF] (old project timelines: v201807 [PDF]; v20170720 [PDF])
- Year 1 simulation variations planning spreadsheet
- Initial project storage and CPU estimate (to be revised for 2019-2020)
- Initial Status of Plans of CCMC Activities for the Annex (working document updated on 20170703) [PDF]
- Report discussion (https://ccmc.gsfc.nasa.gov/annex/)
- I added an additional plot to the project website that is not in the report. This shows the CME arrival time error vs the ambient solar wind prediction error for all runs (computed from the average of the first day of each run): https://ccmc.gsfc.nasa.gov/RoR_WWW/annex/error_vdiff_v1.png
- Plans for the paper based on the report. What analysis to add, what to remove
- Already planned: validation for the best ranking realization (ADAPT)a
- Already planned: rank histogram for ADAPT
- Already planned: single event WSA-ENLIL-ADAPT high (1 deg) resolution test
- Already planned: single event ADAPT test with wider range of realization uncertainties
- Carl's suggestion to validate the performance of the average of all run variations Additional suggestions on pruning the ensemble from Eric, Nick, Carl?
- Eric's suggestion: remove 2 events that were not well replicated? Or replicate them?
- Other suggestions
- Which journal to submit to
- Some folks are still reviewing the report and will provide more feedback later
- Two events were not replicated well for unknown reasons. We may leave these out of the validation presented in the paper. ACTION: Eric will check again on how these runs were initialized, with help from Doug.
- Dusan suggests calculating the performance for 2 different event subsets, glancing blow and direct impact. All agreed this was a good idea. Doug provided some NRL reports that started investigating this. ACTION: Doug and Eric to derive a analytical impact parameter calculation, apply it to our event list, and select a threshold for the parameter that divides the list into 2 groups. Please add this information to the first tab of this spreadsheet, or create a new tab: https://docs.google.com/spreadsheets/d/1mqzRePy1xo_cIMzeo1XjPqVJF3OmNvwzPLTl8AFHsLY/edit#gid=934388886 (you can remove the events Eric crossed out).
- We are wrapping up some of the ensemble analysis from this project in the form of the paper even though the report requirement has been fulfilled. Are there any ideas for another annex project? What is needed? Some activities can be worked within ISWAT teams, but some projects may benefit from a formal annex project with funding. Carl mentioned that the air force has an interest in Kp.
- Carl is now experimenting with a wider range of uncertainties in the ensemble (and maybe increasing the number of members). This approach is only ok if people using the maps have a method of getting the best members and not taking any one member on its own as a reasonable result. ACTION: Carl will provide experimental ADAPT maps for the our test event 2014-08-16 so we can compare to the current ADAPT maps.
- Carl had the idea of evaluating the performance for the median of all of the CME arrival times for each event (all variations). Some thought this was interesting as a way of having more members in the ensemble. Some thought it was less relevant to operations since it would not be like this in a real-time environments, some thought it may be interesting as long as the WSA 2.2 variations were left out. At this point it is unclear whether or not to proceed with this, but more exploration may be useful.
- There also was more interest in evaluating performance of more variations instead, such WSA with new tuning, or assessing which ADAPT uncertainties have a greater impact on arrival time error (e.g. source surface height or supergranulation). Also do the ADAPT maps with farside regions added perform better? Some of these ideas are outside the scope of the paper, but something this group or the community could pursue in the future.
ADAPT Ensemble pruning:
- Shaela provided the best realizations for each event using a skill score checking for both polarity and speed performance. (ACTION) Leila will send this list to everyone. Leila will evaluate performance for the best realizations. There was also an idea of evaluating the performance of the best 3 (and not just the best 1).
- Leila pointed out that while it may be useful to use the best realizations (also from a computational standpoint), the current performance of whether or not the observed arrival falls within the ensemble range is extremely poor (less than 15%), so more variations, more members, may actually be helpful.
- Other pruning options include:
- white light coronal structure (Shaela's work)
- coronal hole comparison
- CME comparison in heliospheric imagers
- ADAPT monopole signal
We could possibly show a proof of concept of all of these methodologies in a separate paper focused on just our test event 2014-08-16 (along with our resolution tests and expanded ADAPT uncertainties test of this event).
Ambient solar wind error:
- We discussed the plot of the arrival time error vs ambient solar wind speed error and how there isn't a correlation: https://ccmc.gsfc.nasa.gov/RoR_WWW/annex/error_vdiff_v1_filter0.png Nick suggested filtering based on CME input speed and these plots and more discussion were provided in a followup email (see below)
- Leila mentioned this is a topic that the CME arrival time working team is planning to address with numerical experiments as it is hard to assess this with real events.
- Doug and Nick mentioned that the effect of the background error is likely less important for multi-CME events and when there is interaction with the current sheet.
=========== Follow up Email:
Thanks for the helpful discussion on Monday! I added the telecon notes (with some action items identified) to the project website: https://ccmc.gsfc.nasa.gov/annex/files/20200518_telecon_notes.txt
I updated the arrival error vs solar wind speed error plots to change the plot symbols based on CME input speed threshold: unfiltered: https://ccmc.gsfc.nasa.gov/RoR_WWW/annex/error_vdiff_v1_filter0.png 500 km/s: https://ccmc.gsfc.nasa.gov/RoR_WWW/annex/error_vdiff_v1_filter500.png 800 km/s: https://ccmc.gsfc.nasa.gov/RoR_WWW/annex/error_vdiff_v1_filter800.png 1100 km/s: https://ccmc.gsfc.nasa.gov/RoR_WWW/annex/error_vdiff_v1_filter1100.png 1400 km/s: https://ccmc.gsfc.nasa.gov/RoR_WWW/annex/error_vdiff_v1_filter1400.png
Also it completely slipped my mind that Christina Kay recently did some work quantifying the background (and CME input parameters) effects on CME arrival with comprehensive numerical experiments with her drag based model, ANTEATR, and DBM. She finds that the weaker CMEs (slower, less massive, smaller) do show a substantial change in arrival time when the background speed is changed. See section 3.3 and figure 4: https://doi.org/10.1029/2019SW002382
So while the ambient solar wind error can make a difference, my plots linked above show this effect is overshadowed by other errors, including CME input parameters. Christina's work also looks at the CME input parameters and discusses speed and width as most important (depending on CME strength) which is in agreement with Vic's earlier findings--see the paper linked above for more details.
ADAPT realization ranking: I have forwarded Shaela's email to everyone listing the best realizations for our events and plots she used to assess them. Please reply to that email thread if you have any comments/suggestions on how we should approach using the ranking.
9 March 2020 Telecon Notes - telecon with model developers to review preliminary validation results (MAE for different runs and different years)
- review results with model developers Nick, Carl, Dusan
- review material from Dusan https://ccmc.gsfc.nasa.gov/annex/files/mar09-agong_FromDusan.pdf
- followup on previous telecon items --adding more CMEs to events, leaving some CMEs out of error statistics --file counts for time dependent runs --any other
- Carl discussed how ADAPT HMI does not agree with ADAPT GONG at the poles starting in 2018 and he sent a followup plot and description.
- WSA tuning: Nick performed tuning in the last few years by running ADAPT-GONG, ADAPT-KPVT, and ADAPT-VSM for select periods spread over a few rotations per years and spread over the solar cycle
- Eric would like to decrease the error bars so that results are more statistically significant. Suggests continuing with the simulation reruns with additional CMEs.
- Nick thinks that the ADAPT-WSA best realization will perform better than the ADAPT-WSA-ENLIL median arrival. Will check this in the followup paper based on the report.
- Leila mentioned that spread in arrivals from ADAPT ensemble is not more than a few hours, what does Carl think about whether or not the ensemble has enough spread at the Sun. --Carl said he recently expanded the supergranulation uncertainty spread. Also right now the mean is used for the differential rotation profile, instead of uncertainty. Reason he didn't have a larger ensemble spread was he wanted to reduce the "bad" maps in the ensemble in case they were used directly. He would like to periodically explode the ensemble and then constrain it. --For a followup paper we will get maps generated with a larger uncertainty spread for our test event from Carl to test. --Nick suggests not weighting the resulting CME arrival from the realizations equally, but instead weigh them based on past performance (Leila: for determining the median arrival? and/or the distribution?). A subset of the good performing realizations can be used to get the range in arrivals (instead of using all).
GONG data issues:
- ADAPT uses zero point corrected maps (heliographic remaps produced by NSO)
- Andy described all of the issues with gongb and gongz and promised a write up which he delivered and was included in the report. Basically gongz pre 2017 is not the same as post 2017 and that impacts our results.
- Carl currently trusts HMI LOS the most. When VSM is functioning, that helps with the calibration of GONG.
- ADAPT farside region insertions: this is Carl's ongoing research. So far their test period of May 2010 shows an improvement in results (HMI input was used). Regions were inserted manually. This should help with issues when active regions are not fully in view, causing the poles to be incorrect. Would be good to estimate how many days of "incorrect poles" there are.
- There is a desire from SWPC to rerun more simulation with multiple CMEs. It's not possible to both complete this task and do full ensemble validation (or validation of the best realization, since we haven't received that list yet), in time for the May 1st deadline. SWPC decided the event reruns were more important to complete in hopes that the overall errors may decrease, so that the error bars may decrease. For SWPC, understanding the performance of the best ranked ADAPT-WSA realization was less important for the purpose of the project.
20200128 telecon notes
Leila and Dusan worked on replicating 2.9e with 2.9f and 2.9g but did not have success. The only time the results were the same were if case files from 2.9e were used to 2.9f or 2.9g. Using 2.9e source code in 2.9f and 2.9g did not help.
the gongb and gongz products are different streams are therefore likely not to have the same timestamps. Only concerned about major time gaps for our events. Leila will provide a file count
Note that for the ADAPT driven runs, the full 2 hour input map frequency was used.
We examined the preliminary errors for the run variations, and ADAPT broken up by event period. There aren't any many clear conclusions at the moment and error bars overlap. There seems to be improvement in using gongz with ADAPT for the more recent years
Event list issues. Eric discovered that the spreadsheet does not match what was used as input to the operational runs. He will discuss with Doug and decide which runs we should redo. Leila does not anticipate a change in trends but the overall MAE may decrease and the error distribution/error bars.
Dusan suggests using different ambient settings for different periods. SWPC only interested if there is a clear improvement.
Leila will see about performing ADAPT error analysis using best realization in addition to the median arrival.
We will meet again in February once there are more results and plots, making sure Nick and Carl can attend.
Christina Lee is visiting in March or April to receive analytical version of ENLIL and Eric is invited.
20191120 Telecon notes
=======Summary of emails between last telecon and this one: =====
- Updated the simulation variations in consultation with Eric and Nick Arge: https://docs.google.com/spreadsheets/d/1wVWmeRoGmn0gXYDYwdx9OuTSIEpWSrFZsQ5nA3QRa1M/edit#gid=767735919
- Eric provided operational settings script so we can be sure we are using the exact same a8b1 parameters.
- Leila identified that the ADAPT-WSA-ENLIL resolution test in the year 1 report for R000 needs to be repeated for the high resolution case because WSA was run at medium resolution.
- Eric added 5 events to the list, and provided the relevant input files and parameters needed. Email excerpt below from Eric: "Hi Leila,
After chatting more regarding the timeline of GONG processing changes with Andy Marble, it is clear that prior to June 23, 2017, the zero point corrected synoptic maps (gongz) were substantially less accurate than the non-zero point corrected maps (gongb). To address this, NSO implemented frame exclusion into their acquisition software beginning September 1, 2016. Unfortunately, the nature of this change, i.e., ignoring the first number of frames, means that archived observations could not be reprocessed for consistency - effectively GONG was a new instrument beyond that date. However, this change was not rolled out across the entire network of sites until April 22, 2017. Due to the manner in which the synoptic maps are created, the first full synoptic map to include only steady-state frame exclusion is dated June 23, 2017.
To better constrain the impact of the "proper" implementation of the zero point correction, I'd suggest we add some events from subsequent period. If you are agreeable to this, I've gone over SWPC's event list and identified 5 additional runs which I believe will serve this purpose well. Let me know your thoughts and I'll send you the run details if this sounds good to you.
===== 20191120 telecon notes: ======
- Recapped progress so far and remaining items: simulation variations, and ADAPT simulations.
- Simulation variations are listed here: https://docs.google.com/spreadsheets/d/1wVWmeRoGmn0gXYDYwdx9OuTSIEpWSrFZsQ5nA3QRa1M/edit#gid=767735919
- ADAPT simulations to do will be single map driven and time dependent map driven.
- Peter completed the necessary WSA runs needed for the variations
- Leila did a test to successfully run 2.9f using WSA 2.2 with gongb and gongz, and WSA 4.5 with gongz and ADAPT WSA 4.5.
- Eric and Leila will followup and check that ADAPT reruns have finished and what is on the NSO server. Peter will run WSA and Leila will run ENLIL on the year 1 test event to compare.
- Leila is consulting with Dusan on issues replicating 2.9e and 2.9f.
- Decided as a group to run variations with 2.9e instead of 2.9f and maybe run adapt cases with 2.9f, or see about using 2.9e instead
- Concern about changes between 2.9e and 2.9f not being documented/clear
- Will run all time dependent simulations with the 6 hour magnetogram time cadence for now, later may examine a subset using the 2 hours time cadence that is available in more recent years.
- Dusan's paper has WSA-ADAPT runs with different cadences that we can examine--ask for preprint.
- Mid January for next telecon, and try to meet more often for better project tracking.
20190718 SWPC - CCMC MOU annex telecon
- Last telecon was Feb 2019 before Leila went on leave. Followup items from that telecon: --Leila updated y axis range of evolution timeline plots for all events and added them to the website --Peter uses gnu compilers, not intel, for WSA. Eric will continue to discuss the intel compiler issues with Nick next week. --Leila sent over information from Dusan about changes between ENLIL 2.9e and 2.9f. --It was decided to delay the selection of the events (by SWPC) for magnetogram time-cadence testing until year 1 simulation variations are complete.
- Annex extended to May 2020, funding received
- Peter completed most the required WSA simulations needed for the variation simulations. (WSA 2.2 complete, WSA 4.5 running now)
- Validation scripting was automated by Jan Merka and should be easy to apply to new simulations.
- Leila sent over spreadsheet of year 1 simulation variations that were proposed in the report to discuss if adjustments need to be made. SWPC will check and let us know next week and they will also consult with Nick Arge. https://docs.google.com/spreadsheets/d/1wVWmeRoGmn0gXYDYwdx9OuTSIEpWSrFZsQ5nA3QRa1M/edit?usp=sharing
- Carl Henney reported that the ADAPT maps on the NSO server need to be reprocessed due to a jitter issue. This will likely happen by October/November. We do not antcipate this greatly impacting the timeline of this project.
- SWPC ok with suppressing most of the 3D output for the ADAPT runs and focusing on Earth timelines. If graphics are needed specific simulations can be re-run.
- Side question: Eric having issues with run output producing the same binary results when changing the number of processors. He will send some evaluation commands to Leila to check if CCMC has the same issue.
- Discussed Owens 2018 paper on solar wind validation approaches: https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2018SW002059
20190206 SWPC - CCMC MOU telecon
Last telecon was July 2018, but discussions also happened during Leila's visit at end of August 2018. Since then we iterated on the year 1 report via email.
Leila reviewed the updated timeline: https://ccmc.gsfc.nasa.gov/annex/files/ProjectTimelineUpdated201902.pdf This timeline was ok with everyone but we may need to extend the length of the MOU annex to accomodate this. Everyone happy with progress so far, just need more time due to various delays. Also interested in establishing more annexes
WSA: is CCMC using gnu or intel compilers? SWPC requirement is to use intel and Eric has found discrepancies in velocity results near the equator.
Leila and Peter join WSA team meetings from time to time, and Eric can dial in for relevant topics
Leila will get new 2.9 delivery from Dusan this week. She will take notes on scripting/vis changes and share with everyone.
Leila will send list of events and explanations for large offsets between SWPC operational arrival times and CCMC benchmark times.
Leila to fix in future report: adjust ymin and ymax axis to represent all three time series (right now it adjusts to ymin and ymax of simulation time series, not data).
Eric to provide a few events for CCMC to do magnetogram time-cadence testing (using gongb).
Leila: problems had replicating 2.9e compared to 2.9d. Started from scratch with 2.9e scripts and got it working. Unknown as to the initial error Next steps: producing all benchmark runs and time-dependent runs, then performing validation.
Eric: will send questions about div/difb (posted below) Leila: will go over div/dif settings with Dusan when he is back from Dusan and develop documentation
Eric: comparing gongb and gongz for a specific time period got funny results. Will followup with Nick and Dusan about it
Leila: Work with Nick and Dusan to test WSA 4.5 with ENLIL 2.9d for July-August 2010, all tests going ok. Now working on ADAPT testing. Will cc everyone with status and plots
Eric: Aug 16, 2014 to use for year 1 ADAPT test. Also test effect of input cadence. Time-dependent runs: magnetogram time cadence, use 6 hrs for all years. ADAPT files to use are located here: ftp://gong2.nso.edu/adapt/maps/gong/
Leila: Aiming for Sep 1st for interim report.
====== Message from Eric: In 2.6 the following divB-related keywords are set:
difbb=nodifbb divb=difb divbt=difbtt
whereas the default values set in 2.9 are:
difbb=nodifbb divb=nodivb difb=difb
Assuming that the usage of divb is unchanged, it would appear that divb=difb should be used to replicate 2.6. However, it is not clear what value should be used for difb. In 2.9, difb=difb, but this parameter is not used in 2.6. Additionally, divbt has disappeared between 2.6 and 2.9. It is no clear whether divbt has been renamed as difb, or if the current implementation changed such that divbt is no longer used?
Leila: Still finishing up automated scripting process for 2.9d to run through all of the events in replication mode. Need to know model settings for next set of simulations
SWPC: No need to decide on new model settings for 2.9d. SWPC would like to use a3b2/current settings for the whole study.
Leila: what settings does SWPC plan to use for 2.9 on their operational system? Don't worry about this for now.
SWPC: Should we explore gong vs gong2 with better polar settings SWPC are not currently using gong2 (Called gongb and gongz in ENLIL)
SWPC: Year 1 task is to only test only time dependence aspect, keeping background parameters the same. Change one thing at a time, no need to change background parameters because they will not have a noticeable effect on the the CME arrival time. Gongb vs gongz on the other hand will likely make a big difference.
SWPC chose hourly updated magnetogram for the input frequency. NOTE AFTER TELECON: Hourly magnetograms are only available after November 2012. Prior to this they are available every 6 hours. Is it ok to us the highest magnetogram update frequency available? Or should they all be fixed at 6 hours?
Vic: Possibly test gongb and gongz for a subset of events, but stick to the plan first. If we are running ahead we can do these additional tests.
Leila will post a few time dependent examples
Leila: note that from year 1 to year 2 we are moving WSA 2.2 to WSA 4+ so we might want to rerun simulations. SWPC thinks we should probably rerun.
Eric has spoken to Carl Henney. Maps will be from 2016 onwards on their server. Do we grab the old version of ADAPT maps prior to 2016? Or would he be willing to give us some maps from prior to 2016? Eric asked Carl. After 2016 there was some "frame exclusion" and "zero point correction" was applied.
Howard would like: Block chart of all of the different runs, say what we are testing at which stage. Eric will put together?
Eric needs 2.9d delivery from Dusan soon for internal SWPC deadlines.
Eric: ADAPT is using gongz, so maybe we should understand how gongz compares to gongb for when we get to year 2.
Vic: A lot of moving parts with the new WSA version and settings, and magnetograms.
Leila: Will post some runs with gongb vs gongz that have already been performed.
Leila introduced Christine Verbeke who worked on EUHFORIA, and Neel Savani who will help validate the runs for this project
Begin forwarded message:
From: "Mays, M. Leila (GSFC-6740)" email@example.com Subject: Re: Resolution comparison Date: March 9, 2018 at 6:28:44 PM EST To: Vic Pizzo - NOAA Federal firstname.lastname@example.org Cc: Eric Adamson - NOAA Affiliate email@example.com, Dusan Odstrcil firstname.lastname@example.org
Thanks for the info Vic, that will work fine for us at this stage without ADAPT.
Here is a CPU and storage estimate I made early on during project planning. Eric, I also updated this with the CPU time numbers for the 4 events we ran at all three resolutions: https://docs.google.com/spreadsheets/d/1CnobkPRkBFnDdG7BvIDvV29lJVWVRHzOGzXxnvTaqQY/edit?usp=sharing Post processing/graphics time also should be added to the CPU time estimate and is not listed there.
Another telecon soon would be good. We have a visualization event in NYC March 21-24 and I have a workshop 28-30. So March 19, 20, 26, or 27 would work for me.
Before we have the telecon it would be good to have your inputs on what you want to use for:
- the new version ambient parameter settings and other model settings. I sent over the most recent 2.9 run script that shows the defaults, and we can consult with Dusan on this. We can do test runs to help decide the settings you need.
- time-dependent inner boundary runs: magnetogram input frequency to use (e.g. use all maps up until CME start time?)
- single block or two block runs?
On Mar 9, 2018, at 5:39 PM, Vic Pizzo - NOAA Federal email@example.com wrote:
After some discussion here, we would like to proceed with the GONG-driven runs of our full CME list using "medium" resolution, which is what we use in our Ops runs. Coarse grid affects the arrival time noticeably, and when you get interacting CMEs or a CIR-CME interaction, it tends to smear out the resultant structures excessively. On the other hand, you don't gain too much going to fine or super-fine grid, since the inputs are so crude to begin with - not to mention each factor of 2 in resolution chews up another order of magnitude compute time and storage.
So if that is ok with you, that's how we'd like to go.
We should have another telecon tag-up in a week or two, maybe Mar 21-22 timeframe
Begin forwarded message:
From: "Mays, M. Leila (GSFC-6740)" firstname.lastname@example.org Subject: Re: status update Re: CCMC event list Date: February 15, 2018 at 8:03:40 PM EST To: Vic Pizzo - NOAA Federal email@example.com, Eric Adamson - NOAA Affiliate firstname.lastname@example.org, Doug Biesecker email@example.com Cc: Dusan Odstrcil firstname.lastname@example.org, "Macneice, Peter J. (GSFC-6740)" email@example.com, "Onsager, Terrance G. (GSFC-4170)[NOAA]" firstname.lastname@example.org, Howard Singer email@example.com, "Kuznetsova, Maria M. (GSFC-6740)" firstname.lastname@example.org, "Taktakishvili, Aleksandre (GSFC-674.0)[CATHOLIC UNIV OF AMERICA]" email@example.com, "Savani, Neel (GSFC-672.0)[UNIVERSITY OF MARYLAND BALTIMORE COUNTY]" firstname.lastname@example.org
I completed resolution tests for 4 of the events in the subset and have posted all of the run output files, plots, and resolution comparison plots on the website: https://ccmc.gsfc.nasa.gov/annex/ See the links under â€œReplication of event subset.â€ Here are the direct links to the resolution plots in particular: https://ccmc.gsfc.nasa.gov/RoR_WWW/annex/ResolutionComparison_20111126_v2.9d.png https://ccmc.gsfc.nasa.gov/RoR_WWW/annex/ResolutionComparison_20120123_v2.9d.png https://ccmc.gsfc.nasa.gov/RoR_WWW/annex/ResolutionComparison_20120614_v2.9d.png https://ccmc.gsfc.nasa.gov/RoR_WWW/annex/ResolutionComparison_20120928_v2.9d.png
Overall the tests look good. There is some different resolution behavior for 20120123 probably due to the interaction with the stream, which is more well-defined/stronger in the medium and high resolution cases.
Iâ€™m not sure why with the other version (2.8f, linked directly above the event subset links https://ccmc.gsfc.nasa.gov/RoR_WWW/annex/ResolutionTestPlot_v2.8f.png) we didnâ€™t see the same resolution behavior, but it may come down to setting differences. Ultimately when you choose the â€œnumoâ€ method and other settings we should probably redo these tests to verify the resolution behavior for those settings. It would be good to get an idea of what 2.9 settings you would like to use (or test) by early March.
Begin forwarded message:
From: "Mays, M. Leila (GSFC-6740)" email@example.com Subject: Re: Resolution comparison Date: February 8, 2018 at 4:47:08 PM EST To: Eric Adamson - NOAA Affiliate firstname.lastname@example.org, Vic Pizzo - NOAA Federal email@example.com Cc: Dusan Odstrcil firstname.lastname@example.org
Hi Eric, Vic, Dusan,
Here is another low, medium and high resolution comparison for one of the events in the subset that had a clear arrival: https://ccmc.gsfc.nasa.gov/RoR_WWW/annex/ResolutionComparison_20120928_v2.9d.png (also linked from the main page of the website https://ccmc.gsfc.nasa.gov/annex/).
I also running high x2 but it will probably take a week for that to finish and I can add it to the plot.
This one behaves more like you would expect.
Here is the earlier comparison test plot you were looking at: https://ccmc.gsfc.nasa.gov/RoR_WWW/annex/ResolutionTestPlot_v2.8f.png The difference between this test and the previous one is two things. One is the model version (2.8f vs 2.9d). Note that I wasnâ€™t able to replicate 2.6 with 2.8f, so there is something different that is unaccounted for. Second is the numerical model settings that are almost the same except vc (Char. speed at cell interfaces).
By the way I did get all of the events in the subset replicated in the end. The one that wasnâ€™t working was because I overwrote the input file by accident, everything is updated on the website now.
20180207 telecon notes
Leila posted notes on event subset. Two events are not in larger event list. SWPC to decide whether to add them or not.
All events (7) replicated well wtih Dec 2017/Jan 2018 v2.9d except for "2012_01_23". Leila will investigate. All simulation files and comparison plots have ben posted to the wesbite. Update: 2012_01_23 is now correctly replicated
Once Leila completes runs on request scripting for Dec 2017/Jan 2018 v2.9d CCMC will replicate all simulations for the event list and they will be in the CCMC run database as the benchmark.
Next step will be to run v2.9d with a single map, and then time-dependenet sequence for all of the events. Leila sent run script to Vic and Eric who will check which defaults are appropriate. CCMC and/or SWPC can do test runs to decide on default. Leila will also ask Dusan on ambient setting defaults he recommends
SWPC: testing out jmap features but there are issues with scripting and system calls they are working on. Peter warned about stale NFS handles that can appear.
New WSA version and ADAPT: Peter has the new WSA version (delivered), Dusan still needs to do some testing with the new WSA files, and new WSA-ADAPT files. Then CCMC can do one event with ADAPT before the end of year 1. SWPC to choose which event.
Grid resolution: run at low resolution for now and do a subset at medium later, revisit this issue.
Metrics: ok to go with basic metrics outlined in SWPC discussion document and telecon notes, and then revisit this issue.
Side discussion on how to enter correct ENLIL simulation on SWPC's behalf on CME scoreboard (related to Pete Riley's paper). Doug will think about this and get back to CCMC. This could be time-consuming/tricky task.
Use original gong fits wsa.gong.fits, because gong maps are updated at a later date. Doug provided these in the tarball
5-6 runs, first cut coarse grid, no sense in doing this with medium grid yet significant work to be done just with the metrics, questions will pop up that might change things
Pick some select medium grid runs for exploration later.
What time averaging of the data should be used to compared with the model?
Vic has a plot of a 3 hour median filter on the observations. Model runs are still broader than the 3 hour average.
Figure out metric on time series of observations vs model
Plot output of runs similar to Vic's plots using Level 2 Wind or ACE with a 3 hour median filter.
may also just look at amplitudes at velocity vs denisty or temperature.
SWPC will give CCMC several run outputs for this first cut of 3-6 runs for which there is a full archive record, to make sure we are getting the right thing
SWPC will review resolution and time dependent run test on annex website
CCMC is waiting on ENLIL v2.9 update delivery
Following the meeting:
- Leila helped with v2.9b JMAP debugging of SWPC installation, everything seems to work now
- Vic and Eric identified a handful of events and provided the full input/output files
- Leila will test v2.6 replication these events (using 2.9d until 2.9x delivery update is received) and we will discuss at AGU.
Question for Dusan: major changes between 2.9d and earlier 2.9 versions? -Dusan: no major changes, just new options for azimuth and time shift at inner boundary to remove one step of interpolation.
CCMC will check replication on 2-3 of the events from Doug's list, SWPC will provide the full run input/output for comparison.
Leila question: what background/ambient settings should we use for the single map 2.9x runs? SWPC: not sure yet, they will need to discuss
Leila question: what time cadence of maps are desired for the time-dependent inner boundary 2.9x runs? And should the map updating stop on the CME date/time as we would expect in real-time? TBD
Grid resolution impacts were discussed, Leila showed sample runs at three resolutions on the annex website. Needs to be discussed further.
Discussed the metrics documents provided by SWPC. Agreed to basic metris to start with (as outlined in SWPC metrics discussion document). Agreed that arrival time can be derived automatically from the model, SWPC will approve the alogirithm.
Discussion on how to move beyond the standard metrics: --look at validating Bmax, Nmax, Vmax? Tmax probably doesn't make sense --validate pile-up ahead of CME? --validate MC duration (e.g. from RC catalog) using dp variable? beyond the scope of the annex project but could be a follow-on research project for someone --how to directly compare the time-series? how much do you average the observational data before comparing?
Summary of SWPC Enlil Metrics Discussion, Sept 25, 2017
Keep in mind research metrics vs NRT metrics
There are things you can readily test and verify in research (hindsight) mode that are
impractical in NRT forecasting mode. Hence metrics for the two logically differ.
Below we focus on metrics for research mode applications
Currently, CME arrival time is the only widely used metric for comparing predictions with
observed arrivals, which are well-defined for significant CMEs
• SWPC defines forecast arrival as “middle” of velocity rise in medium grid soln
• CCMC defines as 1st few points of velocity rise in coarse grid soln
• (In operations, forecasters typically adjust predicted arrival times to account for
other factors, such as WSA accuracy, confidence in CME fit parameters, etc.)
• Given all the uncertainties, differences in how arrival time is defined is not a big
Other simple potential metrics (sheath only):
• Density pile-up
• CME-front |B| (not Bz)
• TBD define measures for real data:
• Peak or use mean or median over some temporal window?
• Timing and magnitude? (can differ from that of velocity)
• “Driver gas” (related to head-on vs edge vs clear miss definitions TBD)
• Enlil defines via tracer particles (but not very precisely)
• How to identify in real SW (ref Cane and Richardson papers)
• Composition (charge state / species)
• Bi-directional electrons
• Identify data sources to be used
• SWPC keeps None
• 1 AU speed and polarity are only potential metrics
How and what would you quantify?
• The flow speed in front of a CME has a significant effect on its propagation time
• So, narrow consideration to the ambient flow just in front of CMEs
• Ignore CIRS, non-CME flows, stats on those are separate issue
• Same for B polarity
• What averaging range? days? hrs?
• Distinguish between flows perturbed by known prior transient vs simple WSA
201707 Doug, Eric, Leila, Peter meeting at SHINE
Grid resolution: TBD, SWPC would like to see some examples. *Leila to post some (DONE)
Arrival time extraction from the model:
- SWPC rounds to the nearest hour, and generally use the midpoint to rise or 1st data point after rise.
- *Need to decide if an automated procedure should be used
Observed arrival time:
- decision was made to use the list provided by Doug
Ambient setting for runs:
- *need to discuss after doing a series of test runs, including v2.9 ambient settings SWPC will ultimately choose to use
Magnetogram input frequency:
- *Leila to upload test runs (DONE), SWPC will examine
- SWPC will let us know which ADAPT results to use
- arrival time error (average, absolute, rmse, and percent errors)
- we are not considering misses, we have a list of "hits" to validate. beyond scope/will do if we can:
- in addition to arrival time errors, consider validating the MC start time using the dp variable. Leila note: need observed MC start time from somewhere to validate against.
- max params in sheath regions
• Initially NOAA/SWPC selected a set of 33 historical events and 36 CME input parameters have been provided (3 events contain 2 CMEs) for the period of 2012-2014. Later, NOAA/SWPC added 5 more events in 2017-2019 to include events that can use more recent zero point corrected synoptic maps. All 38 events are available to view from CCMC's public DONKI database via an API in text or JSON format (if your are prompted for a password, please click the link again and it will bypass the login screen).
• Replication event subset for which input/output is available [TXT]
Previous Results for Annex Tasks a-c (Single and Time-dependent maps) - updated February 2019
⊳ Results: Year 1 interim report [PDF]
⊳ Download time-series plots comparing the benchmark single map simulations with time-dependent map simulations for all events (8 MB).
⊳ Same plots as above but with fixed y-axis scales for better visibility of OMNI data.
⊳ Full download of evo *dat, evo graphics, bnd graphics, and contour plot graphics for all runs (multi-block simulations for all cases) (7.3 GB).
⊳ Download pptx comparing simulation movies and timelines for events [PPTX]
⊳ Run replication outliers: See explanation google sheet and pptx
⊳ Logs of input map counts: (a) (b) 1(a) 1(b) 2(a) 2(b) 3(a) 3(b) d e
See page 1 of the year 1 report for a description of tasks a-e; see this spreadsheet for a description of tasks a-b and variations.
⊳ Repeat of one CME replication test swpc_test_070617_SH_2 with different grids.
Resolution: low (256x30x90), medium (512x60x180), high (1024x120x360)
Plot comparing temporal profiles at Earth for low, medium, and high resolution for three runs above.
⊳ Also see resolution comparison plots below.
Replication of event subset and resolution tests:
⊳ 2011_11_26 [ run files | evo files & plots | 2.6/2.9d replication comparison plots | 2.6/2.9e replication comparison plots | resolution comparison plot ]
⊳ 2012_01_23 [ run files | evo files & plots | 2.6/2.9d replication comparison plots | resolution comparison plot ]
⊳ 2012_03_13 [ run files | evo files & plots | 2.6/2.9d replication comparison plots | 2.6/2.9e replication comparison plots ]
⊳ 2012_06_14 [ run files | evo files & plots | 2.6/2.9d replication comparison plots | resolution comparison plot ]
⊳ 2012_08_31 [ run files | evo files & plots |2.6/2.9d replication comparison plots ]
⊳ 2012_09_28 [ evo files & plots | 2.6/2.9d replication comparison plots | resolution comparison plot ]
⊳ 2012_11_20 [ run files | evo files & plots | 2.6/2.9d replication comparison plots ]
Comparison of GONG "mrbqs" and "mrzqs" 2007-2017 (with metrics)
⊳ Time-depdendent 24hr cadence gongb (mrbqs) time series plot at Earth (low resolution) [GIF]
⊳ Time-depdendent 24hr cadence gongz (mrzqs) time series plot at Earth (low resolution) [GIF] and (medium resolution [GIF])
Also see other plots in Dusan's calibration folder on helioweather.net
Example of single map vs. time dependent inner boundary simulations:
All using gongb mrbqs: ftp://gong2.nso.edu/QR/bqs/.To see graphics for each run linked below, click on "list of quicklook graphics links" on the run page
Bain et al. ApJ 2016:
Luhmann et al. SWJ 2017:
Luhmann et al. SWJ 2018; Lee et al. GRL 2018 (submitted):
Other (mostly prelim):
Time-dependent (4hr cadence): June 2011
Time-dependent (24hr cadence): Nov 2011
Time-dependent (4hr cadence): Jan 2012
Time-dependent (24hr cadence): March 2012
Time-dependent (24hr cadence): May 2012
Time-dependent (24hr cadence): Sep 2014
Time-dependent (24hr cadence): Oct 2014
Time-dependent (24hr cadence): Dec 2014
Time-dependent (5hr cadence): May 2016
• CCMC v2.9d (version Dec 2017/Jan 2018) replication tests:
(repeat of tests below, added "b" label):
⊳ swpc_wsaenlil_bkg_20170720_0000 (18100) wcone_rep1b (tgz archive of run) and v2.6 and v2.9d temporal profile comparison gifs.
⊳ wsa_enlil_cme_09262017 (1446) wcone_rep2b (tgz archive of run) v2.6 and v2.9d temporal profile comparison gifs.
⊳ wsa_enlil_amb_09242017 wcone_rep3b (tgz archive of run) v2.6 and v2.9d temporal profile comparison gifs.
• From Dusan (September 2017):
⊳ (1) slides [PDF] and notes [TXT] on v2.9 and v2.6 replication tests.
⊳ (2) slides [PDF] and notes [TXT] on v2.9 and v2.6 replication tests.
• CCMC v2.9d (version Sep 2017) replication tests:
⊳ swpc_wsaenlil_bkg_20170720_0000 (18100) wcone_rep1 (tgz archive of run) and v2.6 and v2.9d temporal profile comparison gifs.
⊳ wsa_enlil_cme_09262017 (1446) wcone_rep2 (tgz archive of run) v2.6 and v2.9d temporal profile comparison gifs.
⊳ wsa_enlil_amb_09242017 wcone_rep3 (tgz archive of run) v2.6 and v2.9d temporal profile comparison gifs.
Magnetogram input time cadence tests:
CR 2070 high resolution ambient simulations - 3 hr output (ENLIL v2.8f):
Full rotation magnetograms:
6 hr cadence input: (Note: 1 hr cadence GONG maps are not available for this CR)