PredictWind Model Upgrade 

Evaluation Report

Report by Dr Jack Katzfey – CSIRO Project Leader

Dr Katzfey specializes in the use of climate modelling tools, in the development of high-resolution regional climate projections. He has managed the application of these tools in various uses including weather forecasts for the Australian Olympic sailing team in 2012 in Weymouth, UK and for the Alinghi America’s Cup Team from 2002-2010. For this project, Dr Katzfey has led the team in the development of updated climate projections for Vietnam and provided training on the use and application of CSIRO’s Conformal Cubic Atmospheric (CCAM) Model.

Overview

PredictWind has instituted a new forecast system, with updated, more efficient model code incorporating improved representation of factors such as a better land-sea mask and a new boundary-layer mixing scheme that affect the accuracy of wind speed and direction forecasts.

A series of experiments with different configurations of the Conformal Cubic Atmospheric Model (CCAM) were tested by PredictWind to evaluate which produced the best performance for wind forecasts.  Thirteen different experiments were run over 16 domains selected from the 362 domains available on the PredictWind website. Domains were scattered around the world and were chosen because they were representative of various forecasting challenges and had sufficient available station observation data to compare to the experimental forecasts.  Forecasts were completed at 50 km, 8 km, and 1 km resolution, and were validated against station observations and also compared to ECMWF and GFS wind forecasts.

The best model configuration chosen (experiment pw13) improved wind speed forecasts by 11% at 1km, 7% at 8 km and 18% at 50 km relative to the operational PWG runs at PredictWind.  Wind direction forecasts were also improved by 5% at 1km, 7% at 8 km and 10% at 50 km. 

Overall Results Summary

Table 1: Percent improvement in wind speed Mean Absolute Error (MAE) of pw13 relative to operational PWG runs averaged over all 77 stations associated with the 16 domains evaluated.

(00 is 00 UTC; 12 is 12 UTC, the initial times of the analyses that started the twice-daily runs)

Table 2: Percent improvement in wind direction Mean Absolute Error (MAE) of pw13 relative to operational PWG runs averaged over all 77 stations associated with the 16 domains evaluated.  

(00 is 00 UTC; 12 is 12 UTC, the initial times of the analyses that started the twice-daily runs)

Summary of experiments completed:

1. 13 different sets of new experimental forecasts were completed.
2. 16 different domains from different regions around the world were studied.
3. Forecasts were completed for the month of Sept 2017, though some runs for Oct and Nov. 2017 were included.  Only small differences were noted for Oct. and Nov. relative to Sept., so to minimize cost and time, the focus of the experiments was on Sept.
4. Forecasts were initialized with GFS analyses, though some experiments also tested using ECMWF datasets as initial conditions.  Only small differences were noted for the different initial conditions.
5. Two forecasts per day were completed for each day of the month for all domains evaluated:
   1. 1. 14-day global forecasts at 50 km (with output every 3 hours)
      2. 7-day forecasts at 8 km resolution, nudged to 50 km runs (with output every hour)
      3. 24-hour forecasts at 1 km resolution (with output every hour)
6. Data for same time period and domains from ECMWF and GFS forecasts were also evaluated (typically output was every 3 hourly) and compared to the CCAM experimental results.
7. Model forecasts were compared to 77 observing stations every 1-3 hours.
8. The main statistic to evaluate forecasts was the mean absolute error (MAE) of wind speed and wind direction.  Results shown are percent improvement relative to the operational PWG forecasts.

New features of the experiments:

1. Improved, more efficient model code – The CCAM model undergoes continuous development by researchers at CSIRO.
2. New radiation code – Provides improved forecasts of surface and atmospheric temperatures which affect wind speed and direction.
3. 50% more vertical levels in the model.
4. Non-hydrostatic code – To better capture vertical movement of wind for improved accuracy around mountainous areas.
5. Smaller time step of the model – Allows for more accurate time evolution.
6. More accurate topographical and land-water mask data is now used – For slightly more accurate/realistic topography and land-seas mask data in model.
7. Updated and more detailed land-use and soil type – There is constant exchange of moisture and energy between land and atmosphere, so more realistic representation increases accuracy of forecasts.
8. More sophisticated boundary-layer mixing scheme – Boundary layer mixing is the turbulent mixing of air in the lower part of the atmosphere, typically the lowest 1 km.  Previous PredictWind operational runs used a standard mixing parameterization based upon the Richardson number. The new scheme forecasts the development of turbulent eddies (based upon Turbulent Kinetic Energy, TKE) which then lead to mixing.  Non-local parameterization of mixing (which captures mixing between disparate levels) is also improved in the new boundary layer scheme.
9. High resolution observed Sea Surface Temperature (SST) data used over water – These SSTs provide more realistic water temperatures for the forecasts.
10. Initial analyses are now interpolated to each grid – Provide a more detailed initial condition file for the higher resolution runs.
11. Global 50km grid forecast now used to drive the 8km domains – For more accuracy and consistency between domains.
12. Improved model settings for treatment of Tropical Cyclones on the 50km global grid

Summary of validation procedures:

Key features used to select observational data for validation of the experiments were: 

1. Hourly time frequency was available for the period evaluated;
2. Location of the observing point was on or near water (no checking was done on the metadata provided);
3. The observing point was located within the 1km domains.

No assessment was done of the quality of the observations or the suitability of the location of the observing station.  

Note that most observing stations are located on land but were selected to be as close to the water as possible.  This can lead to some issues when comparing model results with observations due to slight differences in locations (wind over land is typically slower than over water).  The closest model grid point to the station location was used in the evaluation without regard to whether it was a land or ocean point.

Summary:

About a 10% improvement in wind speed was noted when going from 50km to 8km resolution and another 10% improvement when going from 8km to 1km.  Little change in accuracy of wind direction was noted when going from 8km to 1km resolution, but about 7% improvement when going from 50km to 8km. Finally, note that not all stations or domains showed the same improvement.  

Domains and Station data used for 1 km evaluation

Below is a list of the PredictWind domains chosen for the experiments and the number of stations with observational data available for validation found meeting the above requirements.  Most were located within the 1 km domains (about a 50 km x 50 km region), though some were slightly outside.

The USA NOAA Integrated hourly station data (https://www.ncdc.noaa.gov/isd) were used for most observations.  Over New Zealand, extra NZ Met Service station data were used, and some Australian BoM stations were also included

Table 3: List of domains and number of stations used

Different experimental forecast settings used

A summary of the various settings used in the forecast experiments are listed in Table 4.  Some of the nomenclature used in table is described below:

• Green shading is the selected best option
• G50 means run global 50km run; G50(3) means run global 50 km run with PG3 options; G50(11) means run global 50 km run with PG11 options.
• S60 means stretched C48 run.
• Nbd is grid point nudging with time scale of nud_hrs from sigma level kbotdav to ktopdav.  
• Mbd is large-scale nudging using a digital filter
• Modis uses high-resolution satellite land-use dataset with predominate land use type only
• New lsm means new high-resolution dataset used to determine model land sea mask 
• Nh means nonhydrostatic options used (1 or 5 are slightly different treatments)
• TKE mean prognostic Turbulent Kinetic Energy (TKE) scheme use to determine vertical mixing
• Ri means a Richardson based turbulent mixing parameterization used.
• NL means a non-local vertical mixing option is also included
• Epsp and epsu are different numerical treatments for semi-Lagrangian grids
• Two runs completed every day of month: 00Z and 12Z.
• Models output evaluated for all available hours between 1-24 of forecast.  
• For each hour of available data, the error (Model minus Observation) is calculated if data exists for both observations and model.  The absolute value of this difference is called the Absolute Error (AE).  
• Average over whole month is called Mean Absolute Error (MAE) which is the main statistic used to compare various model forecasts.  
• Errors (for both speed and direction) were set to zero if observed wind speed was less than 6 knots, assuming that winds less than this are classified light and variable. 

Table 4: PredictWInd 13 experimental forecast settings

Table 5: Speed mean absolute error (MAE) using station data from all 77 stations.  Res = resolution; PWG refers to operational PW GFS initialized forecasts, numbers 2-13 refer to the experimental forecasts; ec = ECMWF forecasts; gfs = the US-based GFS forecasts.  Percentages are mean % change in wind speed error relative to the operational PWG forecasts. Red highlighting indicates percent improvement (less error) relative to the operational PWG forecasts.  Note only 50 km forecasts were completed for exp14.

Table 6: Same as Table 4 but for wind direction.