Object

Compare wind observations from weather stations and wind forecasted by weather models for various locations around the world in order to rank weather models in terms of wind accuracy. This study focuses on the following weather models: ECMWF, SPIRE, UKMO and GFS.

–

Dataset

• The study covers a 4-month period (Feb 12th to June 13th, 2020)
• 441 land-based stations and 166 buoys available (for ECMWF). Land based stations are mostly close to coastline and buoys are in open ocean.
• For land based stations, the dataset is made of around 80,000 data points and for buoys around 15,000. By averaging those large dataset, we get a robust statistical value that removes any bad stations, bad forecast day, etc) in order to compare overall model performance.
• The wind forecasts cover a 7 day period (day 1 to day 7)
• ECMWF, SPIRE and GFS have a full day 7-day forecast, however UKMO however only has a 5-day forecast.

–

Methodology

For each model run (0Z and 12Z) we calculated the Mean Absolute Error (MAE) between the model and the observation. The MAE is calculated for each day of the forecast (day1 to day 7). 

The resulting MAE is then averaged for every station and every model run, to get a robust statistical value that truly represents the overall accuracy of the model. Example: TWS MAE for ECMWF on Day1 = 2.6kn for land based stations. This means that for 82000 data points the average error is 2.6kn between the ECMWF model and all the observations on land.

The weather model comparison is done for land-based stations and another model comparison for buoys.

–

Analysis TWS MAE

LAND BASED STATIONS analysis

TABLE: TWS MAE for Land-Based stations

Table analysis:

• ECMWF is ranked number 1. It is consistently the best on each day of the forecast (day+1 to day+7). 
• SPIRE is ranked number 2.
• UKMO average is only done over the first 5 days of the forecast so it is not fair to compare to other model average which include a 7 day average. However over the first 5 days, UKMO is slightly better than GFS.

–

BUOY analysis

TABLE: TWS MAE for Buoy

Table analysis:

• SPIRE is ranked number 1. 
• SPIRE and ECMWF are close whereas GFS is significantly less accurate.
• During the first 5 days UKMO is ranked number 3.

–

Analysis TWD MAE

LAND BASED STATIONS analysis

TABLE: TWD MAE for Land-Based stations

Table analysis:

• ECMWF and SPIRE are the best on average. 
• ECMWF is the best from day 1-3, and SPIRE is the best from Day 6-7–

–

BUOY analysis

Table analysis:

• SPIRE is ranked number 1. It is consistently the best on each day of the forecast.
• ECMWF and UKMO are close. GFS least accurate.

–

Conclusion

By comparing forecasts to hundreds of wind observations around the globe, ECMWF and SPIRE both perform better than GFS and UKMO at forecasting the wind.   

ECMWF and SPIRE are close in terms of ranking, and results suggest that ECMWF is slightly better close to shore (land-based station), and SPIRE is better on the open ocean (buoy).