Model Accuracy Guides

Model Accuracy Guides


Setup & Instruction Manual

Step 1.

Click "New Race" to create a new race file.

This creates a folder (named after the race) inside the Model Accuracy "Races" folder, in which prepared data, saved analyses, and analysis settings are saved, to allow you to quickly load your preferences in future analysis runs.

Select what type of Log data you will be comparing to GRIB files. The options now are Expedition log files, Adrena log files, or NOAA buoy data.

Select “True” or “Magnetic” mode. This is to determine the format of the wind direction stored in your log file. It is either True or Magnetic. If your log files capture TWD in degrees True, then select True. If your log files capture TWD in degree Magnetic, then select Magnetic. Consult Expedition settings if necessary. NOTE: NOAA buoy and Adrena log files are in True. Expedition files can be logged in Magnetic or True.

If selecting a NOAA buoy log file you must enter the 5 digit code associated with the buoy, and select "fetch" to the right of the code entry space. All NOAA buoy codes are listed here: https://www.ndbc.noaa.gov/ **NOAA buoy data is only available for the past 5 days from when selected** Example bouy code: PVGF1 is Port Everglades, FL **not all NOAA buoys have TWS and TWD data, confirm before selection**

Set mast height. This must be the distance in meters from the water line to the anemometer. Not required for NOAA buoys (when you click Prepare Data for NOAA buoys the mast height is set automatically)

Set boat data smoothing window. This smooths out the boat log data, removing large changes in short time period that are caused by, e.g., the swinging of the mast, short gusts of wind, etc.

 Expedition set up

Adrena set up


NOAA Buoy setup



Step 2.

Tell Model Accuracy where to find the boat's log files that you want to evaluate.

The folder that stores the log files must contain only log files. 

Tip: Click the folder icon to open up the file explorer.

Tip: Model Accuracy has a quick reference for the folder where that Expedition Sailing software package always stores the log files.

Tip: Even if you are selecting NOAA buoy data to analyze which require fetching data from the internet you still need to create and map an empty folder where Model Accuracy can store the NOAA buoy log data.

Model Accuracy will analyze the boat's log files wherever you specify their folder. It does not have to be where Expedition stores the files, although that folder is referenced for easy operation.

 

Step 2 -*Adrena only*

Adena log files are saved as a .csv file even though they are really separated by a semi-colon.
In order to allow Model Accuracy to view the log file in a standard format we require the user to change the file format for all Adrena log files by adding "adrena.txt" to the file.

This is accomplished in the following way.
1. Open the log file in Excel
2. Click "Save-as" function and simply add ".adrena.txt" to the file name
*this can not be done manually while the file is i the folder, but instead has to be completed in Excel.

You will then see the adrena.txt file in the same folder, which is the file Model Accuracy will need to view for the analysis

  Adrena Log file directory


Adrena Log file"adrena.txt" file extension



.txt file in Adrena Log file directory

Step 3.

Tell Model Accuracy where to find the GRIB files that you want to evaluate.

The folder that stores the GRIB files must contain only GRIB files, or sub-folders that contain only GRIB files. 

Tip: click the folder icon to open up the file explorer.

Tip: Model Accuracy has a quick reference for the folder where that Expedition Sailing software package always stores the GRIB files.

The GRIB source name is set by:

  • If the GRIB files are in a subdirectory, use the subdirectory name.
  • If the GRIB files are in the specified raw GRIB directory, Model Accuracy uses the GRIB file name to work out the source name.
  • For some well known sources the names are automatically abbreviated.

For example, if the GRIB directory has this structure:

BIG_GRIB_1234.grb
PWG_All1/PWG-42424.grb
PWG_All1/BEST_PWG-64646.999.grb
GFSIsNotAlwaysTheBestGRIB.grb

There will be three sources:

BIG_GRIB (taken from the file name)
PWG_ALL1 (taken from the directory name)
GFS (GFS is well known)
 

Step 4.

Prepare Data - This feature prepares the boat log and GRIB data. It determines the earliest and recommended analysis start times, and the latest and recommended analysis end times.

Model Accuracy recommends using the Prepare Data capability before EVERY analysis!

 

Step 5.

Select your preferred Analysis Mode.

“All GRIB runs” creates an analysis on the freshest available data within the GRIB folder, reviewing all the GRIB files available.

“Single GRIB run” will allow you to analyze only one GRIB file to compare against the boat log files. Model Accuracy will choose the GRIB file with a start time closest to the start time of the analysis window you select.

“Stepped” will allow you to select time steps of 6, 12, 24, 48 hours within each GRIB file source to review its accuracy. This is helpful in determining the trends of each GRIB file for comparison.

 

Step 6.

Set your analysis start and end times. Here you tell Model Accuracy exactly from when to when you want the boat log files compared to the GRIB files. The times must be within the range of the boat log files' data and the GRIB files' data.

By clicking the Prepare Data button earlier in the set up process Model Accuracy is capable of determining the earliest start and latest times possible for analysis, and will provide you a “no earlier than” and a “no later than” recommendation. These time recommendations are the earliest and latest time where both GRIBs and boat logs overlap, which is critical to the analysis process.

The “Use” buttons to the right of the recommended times allow the user to immediately accept the recommendation of the start/end time and use at as the analysis start/end time. Consult the Model Accuracy Manual – Warnings – Notes – Definitions for more information regarding the meaning of the recommended times.

Set analysis granularity. Model Accuracy can produce results accurate to the minute, though the computation time is longer.

 

Step 7.

Select “Plots” if you would like to see the graphical plots in addition to the statistical analysis provided.

Click "Analyze!"

 

Optional Step "Save"

Clicking this option automatically saves your results and data plots for future reference, in the SavedAnalyses subfolder of the race folder created in Step 1.

Select “View” to see the saved analysis and plots of your data stored neatly in a folder for the race you created.

 


How to Understand the Analysis

True Wind Speed (TWS) Graphical Plot: Allows the user to visually track the forecasted accuracy of each GRIB vs. the logged TWS captured by the instruments onboard. Each GRIB is color coded and the black line is the boat's TWS plot.

  • Red Squares: represent GRIB delivery times (when you got a new GRIB). This is most commonly every 6hrs or 12hrs.
  • Green Circles: represent each "time step" inside the respective GRIB file. Example: 3hr steps within the forecast.

True Wind Direction (TWD) Graphical Plot: Displayed as a rolling figure within the chart plot. The data can "roll" from 000 degrees to 359 degrees, and the plot wraps around.

  • Red Squares: represent GRIB delivery times (when you got a new GRIB). This is most commonly every 6hrs or 12hrs.
  • Green Circles: represent each "time step" inside the respective GRIB file. Example: 3hr steps within the forecast.

Model Accuracy Results Page: Provides the statistical analysis, trend errors, and overall GRIB file recommendation.

Correlation: Tells you if the GRIB prediction followed the boat log data in parallel. Example: If the TWS was predicted to increase and the boat data did in fact increase the correlation will be high. HIGH is good.
RMS Error: LOW is good. “Above’s” and “Below’s” do not cancel each other out! This is the most accurate error measure, taking into account how much time the prediction spent not aligning with the boat data.
Average Error: LOW is good. This is the simple comparison if you averaged all the differences between the GRIB data and the boat log data. In theory here you can have a GRIB prediction way above and then way below but then averages to the exact average number of the boat data and have a zero Average error.” “Above’s” and “Below’s” do cancel each other out in this analysis! Simple sentences are provided to outline the analysis of data being "xx degrees left/right/above/below of log data".
Calibrate: Provides the user the simple calibration for optimal routing uses in the direction required. EX: if the average error of the TWS forecast was 1.4 knots below log data then the recommendation would be to "Calibrate +1.4 knots" meaning increase TWS that factor to achieve a perfectly adjusted GRIB file.

MA Error is the most important number in determining the accuracy of a GRIB file.

The MA Error is a value that can be interpreted as a wind speed and direction error (it can be thought of as being in knots of wind)


Analysis Mode Definitions

All GRIB runs Analysis mode: Compares the freshest GRIB data to the boat log data. For example, if you get a new GRIB every 6 hours then the oldest GRIB data that will be analyzed in this mode is 6 hours old. This is a broad and general analysis to determine which GRIB is doing well at predicting weather. (most common analysis)

Single GRIB Analysis mode: Compares a single chosen GRIB file (one source, one delivery time) to the boat log data.

Stepped Analysis mode: Compares a chosen GRIB delivery time of all the GRIBs in the folder, by examining in small blocks of time within the  analysis time frame set by the user. The user can choose the size of the blocks of time to compare within the overall analysis (6 hr, 12 hr, 24 hr, 48 hr). This mode is helpful to see the trend more clearly of when each GRIB's forecast gets inaccurate. The stepped mode outputs the results in blocks of times analyzed. Example:


Warnings and Notes

  1. Your computer must be set to Zulu time or the appropriate time zone for your location. This allows the boat log file to be time-stamped in the same way that the GRIB files are created (i.e., in Zulu time).
  2. Model Accuracy applies the appropriate variation as applicable to the GRIB files. Make sure you understand how your TWD is being stored in your GRIB files as either True or Magnetic. Consult your settings in Expedition to make sure you select the proper setting in Model Accuracy.
  3. The height of your anemometer (wind instruments on your mast) must be entered in meters. This measurement is from the waterline to the anemometer (birdie and cups).
  4. Model Accuracy will read only .grb and .grb2 type GRIB files.
  5. The time selected for the analysis must be covered by both the boat log files and GRIB files, or else the analysis will not work. Use the Advanced Data Preparation button/feature to help learn what are the earliest and latest times where both the GRIB and log files overlap in your directory.
  6. Analysis Start Times: The EARLIEST start time is constrained by the start of the boat log data & the single earliest GRIB run. If you start the analysis at this time only that one GRIB source gets analyzed (because the others start later). The RECOMMENDED earliest start time is constrained by the start of the boat log data and the latest GRIB run. If you start the analysis at this time all GRIB sources get analyzed.
  7. Analysis End Times: The LATEST end time is constrained by the end of the boat log data and the single latest GRIB step time. If you end the analysis at this time only that GRIB has data (in the form of a GRIB step) at the end time. The others will have older data (because their last GRIB steps are earlier). The recommended latest end time is constrained by the end of the boat log data and the earliest last GRIB step time. If you end the analysis at this time all the GRIBs have data (in the form of a GRIB step) at the end time.
  8. It is recommended that you keep all of your GRIB files organized by Zulu run time in separate folders, as you download them. When you run an analysis, create a new folder of all the GRIBs you want to analyze and specify that folder as outlined in Step 3 of the instruction manual.
  9. It is recommended that you rename each GRIB file that you download. This is can be tedious, however it helps you quickly identify each individual GRIB file for future analysis, and ensures as a navigator you always know which GRIB you are using. For example:
    Here the navigator has renamed a GFS GRIB file so that it is clear that this GRIB is the 0000 Zulu run from June 4th 2016 and has a 96hr forecast. This provides quick reference not only for the Model Accuracy analysis, but also for navigating.