Model Accuracy Guides

Model Accuracy Guides


Setup & Instruction Manual

Step 1 - Create a new race

Click the "New" button to create a new race. This creates a folder (named after the race) inside the Model Accuracy "Races" folder, in which prepared data, saved analyses, and analysis settings are saved, to allow you to quickly load your preferences in future analysis runs.

New Race
Step 2 - Locate raw log files.

Model Accuracy analyzes the log files in a folder that you specify. It does not have to be where Expedition stores the files, although that folder is referenced for easy operation. You can view this quick reference by clicking on the (?) symbol to the right of the "Log Directory" label. The folder that stores the log files must contain only log files, so we recommend that you create a separate folder for the log files you want to analyze.

The "subfolder" method is the recommended method of organizing log files (and is necessary any time there are log files from different sources). Model Accuracy treats each subfolder separately. The log source name is set by:

  • If the log files are in a subfolder, use the subfolder name.
  • If the log files are in the specified log folder, Model Accuracy creates a subfolder called "Boat" and moves the files into that folder.
  • If you are using Expedition as the log source, you can copy the log files from the Expedition log file folder to your log file folder. This make for a cleaner review process. Model Accuracy has a quick reference for the folder where Expedition stores its log files.

Tip: Even if you are selecting NOAA buoy data as log data to analyze, which requires fetching data from the internet, you still need to specify an empty folder where Model Accuracy can store the NOAA buoy log data

Log Folder

Step 3 - Locate raw GRIB files.

Model Accuracy analyzes the GRIB files in a folder that you specify. It does not have to be where Expedition stores the files, although that folder is referenced for easy operation. You can view this quick reference by clicking on the (?) symbol to the right of the words "Model Directory". The folder that stores the GRIB files must contain only GRIB files, so we recommend that you create a separate folder for the GRIB files you want to analyze.

The "subfolder" method is the recommended method of organizing GRIB files. Model Accuracy treats each subfolder separately. The GRIB source name is set by:

  • If the GRIB files are in a subfolder, use the subfolder name.
  • If the GRIB files are in the specified log folder, Model Accuracy uses the GRIB file name to work out the source name.
  • For some well known sources the names are automatically abbreviated.

For example, if the GRIB folder has these files and folder:

BIG_GRIB_1234.grb
PWG_All1/PWG-42424.grb
PWG_All1/BEST_PWG-64646.999.grb
GFSIsNotAlwaysTheBestGRIB.grb

there will then be three sources listed in the analysis:

BIG_GRIB (taken from the file name)
PWG_ALL1 (taken from the folder name)
GFS (GFS is well known)
GRIB Folder
Step 4 - Anemometer Setup.

Set mast height. This is the distance in meters from the water line to the anemometer where the logged wind data is captured.

Scale/Report Button:
If the Scale/Report button is set to Scale, the TWS of the anemometer and the TWS of the GRIB will both be scaled to the height that you have specified, using the same scaling formula as used in the Expedition Sailing Software. If the Scale/Report button is set to Report, no scaling is done (i.e., the raw wind speeds are compared directly even if they are predicted/logged at different heights), but the scaling factors that would be needed to bring them to the specified height are reported in the message window. We recommend selecting the REPORT option only for advanced level navigators/sailors who are comfortable with the equation of scaling. This is not required for NOAA buoys (when you click Prepare Data for NOAA buoys the mast height is set automatically)

True or Magnetic Degrees:
Select "True" or "Magnetic" mode. This is to determine the format of the wind direction stored in your log file. It is either True or Magnetic. If your log files capture TWD in degrees True, then select True. If your log files capture TWD in degree Magnetic, then select Magnetic. Consult Expedition, Adrena, ect... settings if necessary to learn if you are logging your information in True or Magnetic. NOAA buoy and Adrena log files are in True. Expedition files can be logged in Magnetic or True. NMEA log files can be logged in either True or Magnetic, consult you instrument settings before making this selection.

Scale Button


Report Button:

Step 5 - Put your log and GRIB files in the raw directories.

In this latest version of Model Accuracy you no longer have to select what type of log file format you are using in your analysis. Model Accuracy can now automatically detect the type of log file for Expedition, Adrena, NMEA, NOAA, AuBOM, and PredictWind file formats.

 
Step 5 - Adrena English version

Save the log files with a .adrena.csv extension.


Step 5 - Adrena French version

Adrena (French version) saves its log files as .csv files even though the fields are really separated by a semi-colon. Change the file extension to .adrena.txt.
Adrena (French version) Log file folder:

Adrena (French version) Log file folder: Step 2

Adrena (French version) Log file folder: Step 3

Step 5 - AuBOM

AuBOM (Australian Bureau of Meterorology) weather observation stations have TWS and TWD data that is capable of being downloaded similar to American NOAA weather buoys. However, the AuBOM weather observations have to be manually downloaded from each individual weather station's website. We hope to automate this soon!

The file format of the AuBOM weather observations is in the .axf file format. Model Accuracy will automatically convert the .axf file format into the .csv file format required for analysis.

As the user, you simply have to download the weather observation station .axf file and store on your computer in a log folder.

How to download an AuBOM weather observation stations log file:
  1. Navigate to http://www.bom.gov.au
  2. Click an area of the country
  3. Click the Observations tab
  4. Click Latest Coastal Observations
  5. Select an individual observation station
  6. Note the anemometer height of observation station - enter the height into Model Accuracy when you perform the analysis
  7. Download the .axf file
  8. Store .axf file in the specified log folder on your computer
Once the .axf file is downloaded and saved in the log folder Model Accuracy will create a .csv file in the same folder (do not delete the .csv file).
AuBOM Log file download process:

Step 1: Note height of observation station's anemometer - enter into Model Accuracy.

Step 2: Scroll to the bottom and download observation station's log file

Step 3: Select AuBOM file format in Model Accuracy when computing analysis


AuBOM Weather Station File setup
Step 5 - NMEA

The file must have a .nmea.txt file extension.

Your NMEA log file format should look like this format in the raw log file folder.
NMEA Log file folder


NMEA Log File setup
Step 5 - NOAA Buoy

If selecting a NOAA buoy log file you must enter the 5 digit code associated with the buoy, and select "Fetch" to the right of the code entry space. All NOAA buoy codes are listed here: https://www.ndbc.noaa.gov. NOAA buoy data is only available for the past 45 days from when selected. Example buoy code: PVGF1 is Port Everglades.

Not all NOAA buoys have TWS and TWD data, so make sure to confirm yourself on the website of that particular buoy before selection.

NOAA Buoy setup
Step 6 - Prepare data.

Set boat data smoothing window. This smooths out the boat log data, removing large changes in short time period that are caused by, e.g., the swinging of the mast, short gusts of wind, etc.

Prepare Data - This feature prepares the log and GRIB data. It determines the earliest and recommended analysis start times, and the latest and recommended analysis end times.

Model Accuracy recommends using the Prepare Data capability before every analysis!

Smoothing

Prepare Data
Step 7 - Select the analysis sources.

Model Accuracy permits you to prepare data for and then select more than one log file and more than one GRIB model at a time. Here you can select which log files and GRIB models you are going to analyze. Most analyses are complete by selecting "All". If you would like to focus in your analysis on specific logs or GRIBs you have that option in these drop down menus.

Select Logs

Select Models
Step 8 - Select analysis mode.

"All GRIB runs" creates an analysis on the freshest available data within the GRIB folder, reviewing all the GRIB files available.

"Single GRIB run" allows you to analyze only one GRIB file to compare against the boat log files. Model Accuracy will choose the GRIB file with a start time closest to the start time of the analysis window you select.

"Stepped" allows you to select time steps of 6, 12, 24, 48 hours within each GRIB file source to review its accuracy. This is helpful in determining the trends of each GRIB file for comparison.

Select Analysis Mode
Step 9 - Set analysis start and end times.

Here you tell Model Accuracy exactly from when to when you want the boat log files compared to the GRIB files. The times must be within the range of the boat log files' data and the GRIB files' data.

Example: If you have Logs which cover Friday and Saturday, and GRIBs which cover Saturday and Sunday, Model Accuracy can only provide you an analysis of what took place on Saturday as that is the only time that has logs and GRIBs.

By clicking the Prepare Data button earlier in the set up process Model Accuracy is capable of determining the earliest start and latest times possible for analysis, and will provide you a "no earlier than" and a "no later than" recommendation. These time recommendations are the earliest and latest time where both GRIBs and boat logs overlap.

The "Use" buttons to the right of the recommended times allow the user to immediately accept the recommendations of the start/end time and use at as the analysis start/end time. Consult the Model Accuracy Manual "Warnings - Notes - Definitions" which is located at the bottom of this document for more information regarding the meaning of the recommended times.

Set analysis granularity. Model Accuracy can produce results accurate to the minute, but if that's too slow for a very long log you change the granularity to take larger steps through the log data.

Start and End Times
Step 10 - Set offsets.

Here you can move the GRIB model in time and space, and provide offsets for the TWS and TWD.

  • The model can be brought forward or pushed back in time by giving a positive or negative number of minutes offset.
  • The model can be moved north/south and west/east by giving positive/negative minutes of latitude and longitude.
  • The model prediction for TWS can be increased or decreased by a percentage that is greater or less than 100%.
  • The model prediction for TWD can be increased or decreased by a positive or negative number of dgrees.
Providing offsets for the TWS and TWD can be thought of as fixing the errors in the model's prediction of those valuess. If the Calibration values provided by the analysis are put in here then the analysis will show no need for further calibration.

Providing offsets for the time/latitude/longitude can be thought of as fixing possible causes for the errors in the model's prediction of TWS and TWD. Typically either only one of the sets would be used: TWS and TWD offsets, or time/latitude/longitude offsets.

Select the "Use" button to apply the offsets during analysis.

Offsets
Step 11 - Select results.

Select "Component Plots" if you would like to see the TWS, TWD and BAR graphical plots in addition to the statistical analysis provided.

Select "Analysis Plots" if you would like to see the TWS, TWD and Model Accuracy % graphical plots in addition to the statistical analysis provided.

Select "Save" to save your results and data plots for future reference, in the SavedAnalyses subfolder of the race folder created in Step 1. You must click "Save" before you click "Analyse". After analysis you can select "View" to see the saved analysis and plots of your data.

Results Options
Step 11 - Click "Analyse!"

How to Understand the Analysis

Model Accuracy Results Page: Provides the statistical analysis, trend errors, and overall GRIB file recommendation.

----------------------------------------------------------------------------
Analysis for MATestDataFromWeb
Log data source is Boat
ALL_MODELS mode
Analysis start: 2016/06/03 23:00 UTC+0  Analysis end: 2016/06/05 06:00 UTC+0  Granularity: 00:01:00

COAMPS: Earliest step used:
COAMPS_20160603_1200_0009.grib    Updates 03:00 to 03:00 hh:mm apart. GRIB data point resolution 7.8 to 59.7 nm apart.
    TWS: Trend correlation  52%, RMS error   2.4, Average error   0.3 knots   above    log data. Calibrate  -0.3 knots ( 95%)
    TWD: Trend correlation  53%, RMS error  43.9, Average error   2.3 degrees right of log data. Calibrate  -2.3 degrees
    MA accuracy for Boat:COAMPS = 67% (1861 data points)

GFS: Earliest step used:
GFS_20160603_1800_0003.grib    Updates 03:00 to 03:00 hh:mm apart. GRIB data point resolution 13.0 to 60.1 nm apart.
    TWS: Trend correlation  38%, RMS error   4.4, Average error   1.8 knots   below    log data. Calibrate  +1.8 knots (133%)
    TWD: Trend correlation -39%, RMS error  67.9, Average error   0.4 degrees left of  log data. Calibrate  +0.4 degrees
    MA accuracy for Boat:GFS = 47% (1861 data points)

NAVGEM: Earliest step used:
NAVGEM_20160603_1200_0009.grib    Updates 03:00 to 03:00 hh:mm apart. GRIB data point resolution 26.5 to 60.1 nm apart.
    TWS: Trend correlation  55%, RMS error   2.4, Average error   0.8 knots   below    log data. Calibrate  +0.8 knots (112%)
    TWD: Trend correlation  10%, RMS error  63.8, Average error   9.7 degrees right of log data. Calibrate  -9.7 degrees
    MA accuracy for Boat:NAVGEM = 59% (1861 data points)

NDFD: Earliest step used:
NDFD_20160603_1800_0003.grib    Updates 03:00 to 03:00 hh:mm apart. GRIB data point resolution 26.0 to 30.1 nm apart.
    TWS: Trend correlation  61%, RMS error   2.5, Average error   0.9 knots   below    log data. Calibrate  +0.9 knots (114%)
    TWD: Trend correlation  12%, RMS error  58.0, Average error  20.7 degrees right of log data. Calibrate -20.7 degrees
    MA accuracy for Boat:NDFD = 61% (1861 data points)

Model Accuracy recommends COAMPS. MA accuracy for Boat:COAMPS = 67% (1861 data points)
----------------------------------------------------------------------------

----------------------------------------------------------------------------
Analysis settings:
    MA version                     1.600
    Race name                      MATestDataFromWeb
    Log data sources               [Boat]
    Model sources                  [COAMPS, GFS, NAVGEM, NDFD]
    Raw log directory              LogsAndModels/LogFiles
    Raw model directory            LogsAndModels/ModelFiles
    Wind height scaling            Scale
    Wind direction type            True
    Anemometer height              33.0m
    Data smoothing window          0s
    Logs analysed                  [Boat]
    Models analysed                All
    Analysis mode                  ALL_MODELS
    Analysis start time            2016/06/03 23:00 UTC+0
    Analysis window size           31:00 hh:mm
    Analysis end time              2016/06/05 06:00 UTC+0
    Analysis granularity           60s
    Offsets                        Time     0min, LAT    0min, LON    0min, TWS 100%kts, TWD   +0°
----------------------------------------------------------------------------
The Model Accuracy % is a value that can be interpreted combination of a wind speed and direction error. The higher the Model Accuracy % the more accurate the GRIB file. The Model Accuracy % is the most important number in determining the accuracy of a GRIB file.


True Wind Speed Graphical Plot: Allows the user to visually track the forecasted accuracy of each GRIB vs. the logged TWS captured by the instruments onboard. Each GRIB is color coded and the black line is the boat's TWS plot.

  • Red Squares: represent GRIB delivery times (when you got a new GRIB). This is most commonly every 6hrs or 12hrs.
  • Green Circles: represent each "time step" inside the respective GRIB file. Example: 3hr steps within the forecast.

True Wind Direction Graphical Plot: Displayed as a rolling figure within the chart plot. The data can "roll" from 000 degrees to 359 degrees, and the plot wraps around.

  • Red Squares: represent GRIB delivery times (when you got a new GRIB). This is most commonly every 6hrs or 12hrs.
  • Green Circles: represent each "time step" inside the respective GRIB file. Example: 3hr steps within the forecast.

Barometric Pressure Graphical Plot: Displayed values in millibars of pressure. Helpful to understand overall accuracy of large weather systems.

  • Red Squares: represent GRIB delivery times (when you got a new GRIB). This is most commonly every 6hrs or 12hrs.
  • Green Circles: represent each "time step" inside the respective GRIB file. Example: 3hr steps within the forecast.

True Wind Direction Error Graphical Plot: Displayed as degrees of error per GRIB file left or right of the boat's instrument data plotted in time. The closer the plot is to a straight line the less TWD error of that particular GRIB.

  • Red Squares: represent GRIB delivery times (when you got a new GRIB). This is most commonly every 6hrs or 12hrs.
  • Green Circles: represent each "time step" inside the respective GRIB file. Example: 3hr steps within the forecast.

True Wind Speed Error Graphical Plot: Displayed as knots of error per GRIB file above or below of the boat's instrument data plotted in time. The closer the plot is to a straight line the less TWS error of that particular GRIB.

  • Red Squares: represent GRIB delivery times (when you got a new GRIB). This is most commonly every 6hrs or 12hrs.
  • Green Circles: represent each "time step" inside the respective GRIB file. Example: 3hr steps within the forecast.

Model Accuracy % Graphical Plot: Displayed as values 0-100% of accuracy per GRIB file plotted in time. The closer the line is to 100% the more accurate that respective GRIB file. Helps understand exactly when the accuracy of a GRIB file was best or worst and which GRIB file source performed the best in what conditions.

  • Red Squares: represent GRIB delivery times (when you got a new GRIB). This is most commonly every 6hrs or 12hrs.
  • Green Circles: represent each "time step" inside the respective GRIB file. Example: 3hr steps within the forecast.


Analysis Mode Definitions

All GRIB runs Analysis mode: Compares the freshest GRIB data to the boat log data. For example, if you get a new GRIB every 6 hours then the oldest GRIB data that will be analyzed in this mode is 6 hours old. This is a broad and general analysis to determine which GRIB is doing well at predicting weather. (most common analysis)

Single GRIB Analysis mode: Compares a single chosen GRIB file (one source, one delivery time) to the boat log data.

Stepped Analysis mode: Compares a chosen GRIB delivery time of all the GRIBs in the folder, by examining in small blocks of time within the  analysis time frame set by the user. The user can choose the size of the blocks of time to compare within the overall analysis (6 hr, 12 hr, 24 hr, 48 hr). This mode is helpful to see the trend more clearly of when each GRIB's forecast gets inaccurate. The stepped mode outputs the results in blocks of times analyzed. Example:


Warnings and Notes

  1. Your computer must be set to Zulu time or the appropriate time zone for your location. This allows the boat log file to be time-stamped in the same way that the GRIB files are created (i.e., in Zulu time).
  2. Model Accuracy applies the appropriate variation as applicable to the GRIB files. Make sure you understand how your TWD is being stored in your GRIB files as either True or Magnetic. Consult your settings in Expedition to make sure you select the proper setting in Model Accuracy.
  3. The height of your anemometer (wind instruments on your mast) must be entered in meters. This measurement is from the waterline to the anemometer (birdie and cups).
  4. Model Accuracy will read only .grb and .grb2 type GRIB files.
  5. The time selected for the analysis must be covered by both the boat log files and GRIB files, or else the analysis will not work. Use the Advanced Data Preparation button/feature to help learn what are the earliest and latest times where both the GRIB and log files overlap in your folder.
  6. Analysis Start Times: The EARLIEST start time is constrained by the start of the boat log data & the single earliest GRIB run. If you start the analysis at this time only that one GRIB source gets analyzed (because the others start later). The RECOMMENDED earliest start time is constrained by the start of the boat log data and the latest GRIB run. If you start the analysis at this time all GRIB sources get analyzed.
  7. Analysis End Times: The LATEST end time is constrained by the end of the boat log data and the single latest GRIB step time. If you end the analysis at this time only that GRIB has data (in the form of a GRIB step) at the end time. The others will have older data (because their last GRIB steps are earlier). The recommended latest end time is constrained by the end of the boat log data and the earliest last GRIB step time. If you end the analysis at this time all the GRIBs have data (in the form of a GRIB step) at the end time.
  8. It is recommended that you keep all of your GRIB files organized by Zulu run time in separate folders, as you download them. When you run an analysis, create a new folder of all the GRIBs you want to analyze and specify that folder as outlined in Step 3 of the instruction manual.
  9. It is recommended that you rename each GRIB file that you download. This is can be tedious, however it helps you quickly identify each individual GRIB file for future analysis, and ensures as a navigator you always know which GRIB you are using. For example:
    Here the navigator has renamed a GFS GRIB file so that it is clear that this GRIB is the 0000 Zulu run from June 4th 2016 and has a 96hr forecast. This provides quick reference not only for the Model Accuracy analysis, but also for navigating.