Time Based Data – Importing Data Logger Files

Time Based Data – Importing Data Logger Files – Overview

Tool Location: Data > Import Data > From Data Loggers...

This section describes the procedure for loading digital logger data into the database. Logger data is typically in the format of a text file or Excel file that contains a few dozen lines of header information followed by the data sets. An example logger file is shown below.

image

During importing, SiteFX can:

·         Check to see if the logger file is already loaded

·         Match the logger file to a specific well

·         Calibrate the logger reading with manual readings (optional)

·         Screen the data for spikes, steps and flat-lining (optional)

 

In additional, SiteFX can:

·         Apply corrections – add or subtract a fixed value between a specified start and end date

·         Interactive Point Deletion – graph the data and then click on and delete points as necessary

Time Based Data – Importing Data Logger Files – General Usage

The general usage is that you can choose one or more files to be imported. The files get loaded into the database and the import log gets updated.

If a file doesn’t pass one of the screens in the Screening Options area, the file is converted to an Excel file and placed in the folder "SiteFX_Rejected_Logger_Files". The user can then review the file and see what lines of data triggered the screen. The user can then correct or edit the file as necessary and then load the Excel file directly into the database.

If a file with the same data is selected, SiteFX will first remove any existing files that have the same name.

Time Based Data – Importing Data Logger Files – Features

The general usage is that you can choose one or more files to be imported. The files get loaded into the database and the import log gets updated.

Time Based Data – Importing Data Logger Files – Importing

Tool Location: Data > Import Data > From Data Loggers > Importing

image

Settings to use for import

The import options include the following functions.

·         Select a setting

User can select a setting that has the appropriate calibration and screening options previously saved for importing a certain type of logger files.

·         Save a setting

The current selected calibration and screening options can be saved with a user specified name for later usage.

·         Delete a setting

A saved setting can be deleted by user.

Import Files From > Last Folder Used...

When pressed, SiteFX automatically displays the last folder that was used for loading logger files. Intended to save the user time when searching for folders. One or more files can be selected at a time.

Import Files From > Rejected File Folder...

Files that do not pass the screening test get converted to Excel and placed in a special folder. When pressed, SiteFX automatically displays this folder. Intended to save the user time when searching for folders. One or more files can be selected at a time.

Status

Displays the status of the current import.

Open Rejected File Folder

Opens Windows Explorer to the folder used for rejected files (those that don’t pass the screening tests).

Time Based Data – Importing Data Logger Files – File Testing

Tool Location: Data > Import Data > From Data Loggers > File Testing

This tool allows you to check your file for compatibility with the SiteFX importer and provide feedback as to whether the file can be used with SiteFX or not. Data logger files differ from logger to logger and model to model. Even identical loggers can produce different file formats depending on how the logger is configured.

Note: If your logger file is not recognized, contact EarthFX to get support for your file.

Select Files To Test

Press the Select Files To Test button and choose one or more files. If you choose more than one file, SiteFX will check each file individually.

In order for a file to be loaded, the answer to following three questions must be true and the color is green:

1.        Is File Format Recognized?

2.        Is All Data Readable?

3.        Matching Logger Found in Database?

If the answer to any of these questions is no, during import SiteFX will skip the file and write the issue to import log.

In order to match your logger file to an existing location in the database, you must set up an alias for your well. SiteFX will attempt to extract the logger name from several locations in the logger file.

image

For more information on setting up linkages between logger files and wells in your database, see the section Linking Logger Files to Wells.

Time Based Data – Importing Data Logger Files – Calibration

Tool Location: Data > Import Data > From Data Loggers > Calibration

SiteFX can adjust your logger data according to manual water levels if you enter calibration data. Calibration happens DURING import of your logger data. As a result, the calibration data must be entered prior to importing of the logger files. 

Calibration is optional. In order for calibration to happen, user must do the following 2 things before importing:

1.        Enter calibration data.

2.        Choose "Calibrate Data during Import" option on the Settings tab.

The calibration data entry form is as follows.

image

 

Logger List

This is a list of all loggers in the database. In the case where a logger has multiple alias names, the first name ever used is the name that appears in this list.

Logger Aliases List

A list of alternate names for the logger. These are the same entries you would see if viewing the location with the Location Editor.

Add New Logger

Prompts you for a name and then adds a new logger to the database by automatically adding a location and interval with the name you just entered.

Note: Only use if the location does not already exist in your database. If the location is already in your database, find the location using the Location Explorer and then use the Location Editor to turn on the logger flag and add a logger name.

See the section Linking Logger Files to Wells for more details.

Add Logger Alias

Opens the Location Editor. Once the Location Editor is open, you can highlight the appropriate interval from the list at the top left and then enter alternate logger names.

Date

Date that the calibration reading was taken. Double click in this field for a calendar.

 

Time

Time that the calibration ready was taken.

Manual Reading

The value of the manual reading in masl.

Logger Reading

The value of the logger reading in masl. The difference between the manual reading and the logger reading is the value used for the correction.

Offset

A value that will be added to all values in the log file, in metres.

Logger Reading (Comp)

The value used for calibrating ‘compensated’ files. A file is flagged as compensated if the word ‘compensated’ appears in the file name. The difference between the manual reading and the logger reading (comp) reading is the value used for the correction.

Add Reading button

Press to add the current values to the calibration register.

Delete Reading button

Press to delete the current values from the calibration register.

Time Based Data – Importing Data Logger Files – Corrections

Tool Location: Data > Import Data > From Data Loggers > Corrections

This tool is used to apply a positive or negative correction to data that was imported using the Data Logger Import Tool. The correction tool is show below.

Applying a correction involves four steps running from the top to the bottom of the form.

Step 1

Choose a single location from the logger list. Click on a logger will update section two to show all available data for that location.

Step 2

Select the readings you wish to correct. The water level readings will automatically become selected by default if these can be auto detected, but you can select any readings you wish. The first reading date, last reading date and units are provided to you.

Step 3

Enter a positive or negative correction and a start and end date for the ranges of values that the correction should be applied to. If times are required, use a 24 hr time format (all 4 digits of the time must be entered, even if there is a proceeding zero). Starting and ending date/times are included when the correction is applied.

Step 4

Press the Apply button. You will be asked to confirm the correction details before the correction is applied. A status box will appear during the correction showing you the current state of the correction.

image

 

Time Based Data – Importing Data Logger Files – Summaries

Tool Location:

Data > Import Data > From Data Loggers > Summaries

Summary > Import Log > Last 50 Imports

Shows the import log for the last 50 imports.

Summary > Import Log > Last Import

Shows the import log for the most recently performed import.

Summary > Import Log > All Imports

Shows the entire import log history.

Summary > Import Log > Loaded Logger Data

Brings up a table that summarizes all loaded data logger data in the database.

The summary data looks as follows:

image

 

Refresh Grid

Load data from database to the grid.

Detach Grid

Open the grid in a separate window.

Save to Excel

Export the grid into an Excel file.

Time Based Data – Importing Data Logger Files – Settings

Tool Location: Data > Import Data > From Data Loggers > Settings

Parameters

The parameter options match the parameters in the database with the parameters in a logger file.

Filename Keywords

User can specify special keywords included in the file names for compensated or uncompensated water level data.

Data Storage

Data can be stored in one of 2 options: Normal or Compressed.

Calibration Options

Calibration data gets applied during the import of the logger data. As a result, the calibration must be entered prior to import. The process of retrieving and applying the calibration is described below.

image

Calibration Portion of Logger Import Tool

If the user chooses to turn Calibrate Data during Import on, SiteFX will adjust values from the data logger file accordingly.      

Retrieving Calibration Data

Example Scenario:

·         Calibrate Data during Import is turned ON

·         Include ‘time’ element when searching for calibration data is turned OFF

·         Search calibration data once per file is turned ON

These are the most commonly used options when using the Calibration Options in SiteFX. As a starting point to describing the process of data calibration, assume the options are set in this way.

Starting at the beginning of the logger file, SiteFX will look at the date of the first record. It will then look for the first calibration data available on or before this date. This is called the low-end calibration point. Next, SiteFX will look for the first calibration data available after the current date (which would be on or after the next day). This is called the high-end calibration point. These calibration points will be used to calibrate the entire file.

If there is more than one calibration point available at the low end, SiteFX will use the time portion of the calibration data to sort the calibration data and select the latest  one. Therefore if there are 2 available on the same day, SiteFX will choose the later of the two.  If there is more than one calibration point available at the high end, SiteFX will use the time portion of the calibration data to determine which one is earliest. Again, if there are two on the same day, SiteFX will choose the earliest one of the two.

If no calibration data is found, SiteFX will reject the file and notify you.

Include ‘time’ element when searching for calibration data

In general, this option can be left off.

The option called Include ‘time’ element when searching for calibration data refers to the time element of the logger data and not the calibration data. The time element of the calibration data is always considered.

If the time element option is turned on, then SiteFX looks at both the date AND time of the first record in the logger file. It will then look for the first calibration data available on or before this date/time for the low-end calibration point and look for the first calibration data available after the current date/time for the high end calibration point.

Search for Calibration Data Once per File

If this option is turned off, SiteFX will search for low-end and high-end calibration points for every line of data in the logger file. This will increase the time necessary to process the file.

In general, if calibration readings correspond to the beginning and end of the logger files, searching for calibration data once per file will suffice. If you have additional calibration readings taken at intervals more often then this, turn this option off.

When this option if off, the search for low-end and high-end calibration data changes somewhat. The low-end search is still for calibration data on or before the current date and time but the search for high-end calibration data is now for data on or after the current date and time as opposed to just ‘after’. As a result, the time element option MUST be turned on. SiteFX will automatically set this option for you and provide a notification. This ensures perfect calibration if the calibration date/time fall on the exact same date/time in the logger file.

Calibration Window

SiteFX starts at the first line of data and moves through the file line by line. For each line of data, SiteFX checks to see how many days are between the current date and the date of the low-end and high-end calibration point. If the number of days exceeds the number of days the user has entered as the Calibration Window, the record is flagged and the file is rejected.

Applying the Calibration Values - Offsets

After retrieving the low and high-end calibration points. SiteFX obtains the offset that the user entered for both the low and high end points. Offsets are used for shifting the entire file by the same amount. If the low end and high end offsets are not the same value, SiteFX will notify you and the file will be rejected.

Note – offsets handled in this manner are typically only temporary in nature until the user can input the appropriate information into the data logger.

Applying the Calibration Values - Corrections

SiteFX assumes a straight line correction for logger values. The calibration performed is day based, therefore all readings on the same day get the same calibration adjustment, regardless of time.

The formula used for corrections is described below. The procedure is repeated for each line of the logger file starting at the first row. Selection of the calibration data to use for the calibration is described above. Once the calibration data has been selected, the following formulas are applied.

LowEndCorrection:

·         LowEndCorrection = Calibration Manual Reading – Calibration Logger Reading

·         taken from low end calibration data entered by user (the manual and logger readings are the values entered by the user into the calibration tool and not ones that may or may not appear elsewhere in the database or logger file)

HighEndCorrection:

·         HighEndCorrection = Calibration Manual Reading – Calibration Logger Reading

·         taken from high end calibration data entered by user (the manual and logger readings are the values entered by the user into the calibration tool and not ones that may or may not appear elsewhere in the database or logger file)

DateDiffCalibration:

·         DateDiffCalibration = High-End Calibration Date - Low-End Calibration Date

·         times excluded

DateDiffCurrentToLow

·         DateDiffCurrentToLow = Current Date From File - Low-End Calibration Date

·         times excluded

·         SiteFX moves line by line through the file so this value will change for each line

LineSlope:

·         If DateDifference > 0

o        LineSlope = (HighEndCorrection-LowEndCorrection) /  DateDiffCalibration

·         If DateDifference = 0:

                                LineSlope = 0

Correction:

·         Correction = LineSlope * DateDiffCurrentToLow + LowEndCorrection

CorrectedValue:

·         CorrectedValue = OriginalValue + Correction

 

Screening Options

When turned on, SiteFX can scan your file for 2 different types of data anomalies. If a file triggers one of the screening checks, the file is converted to an Excel file and placed in a special directory. The user can then view and see which lines of data triggered the screen (a flag column indicates which trigger was fired). From there, they may delete lines from the file or edit as necessary. Last, SiteFX can load these reviewed Excel files directly. When loading the Excel version of the file, no screening is performed as it is assumed the user has verified the data.

Maximum Step

For each data point SiteFX looks at the data point immediately before the current point. If the difference between these two points (i.e. current value minus previous value) is greater than the maximum step value, the step flag is generated. It does not matter whether the difference is positive or negative.

Consecutive Values (Flat Lining)

At the beginning of the import, the minimum and maximum values are located for that file. If the same value (min. or max.) is produced consecutively more times than the user has specified, the consecutive value flag is triggered. This type of screen essentially checks for loggers that have come out of water or are malfunctioning.

 

Save Defaults Button

The save default button saves the currently displayed calibration and screening options as the default settings for the user.

Time Based Data – Importing Data Logger Files – Notes / Tips

·         Whenever a file is loaded, SiteFX will first check and remove any existing files with the same name. You will be notified before SiteFX removes any data. As a result, it is important to use a naming convention that generates unique file names for your data logger files.

·         The folder used for files that are rejected due to the screening tests is by default set to C:\SiteFX_Rejected_Logger_Files. You can changes this folder under Tools > Options > Advanced if you wish.

·         Screening happens before any sort of calibration takes place.

·         Calibration data with no time specified is assumed to have been taken at 12:00 AM on the selected date. 

·         Calibration data must be entered prior to logger file importing.

·         Logger data is assumed to be in centimeters above sea level.

·         If data is detected to be in PSI, it is converted to masl assuming 1 psi = 0.704 m.

·         If the work ‘compensated’ is located in the file name, SiteFX will not import the temperature or conductivity in this file. This is because it is assumed the temperature and conductivity will be loaded from the uncompensated version of the file.

Time Based Data – Importing Data Logger Files – Linking Logger Files to Wells

In order for SiteFX to link the data from your logger file to an existing location in your database, you must set up a logger connection. To do this, bring up the Location Editor for your location (Locations > Location Explorer > Location Editor button).

Once the Location Editor is open, click on the interval that represents the well. This will put you on the Interval Details tab. Switch to Edit mode (lower left) and place a checkmark inside the ‘Has Logger’ checkbox. Next press the ‘Add’ button and add the logger name that can be used link the file to the wells. A screen shot of this is shown below.

image

For example, if after checking a logger file for compatibility, the results look like the following:

image

 

In this case, you could use ‘WELL-18’ or ‘MW-18’ for the logger name.

image

 

Time Based Data – Importing Data Logger Files – Running an Import

Tool Location: Data > Import Data > From Data Loggers

The main data logger import form is shown.

1.        Use File Testing to check your file first if you are uncertain if the file is compatible and/or has been linked to an existing well in your database.

Read the section called File Testing and Linking Logger Files to Wells for more information.

2.        Turn on or off the Calibrate Data during Import option.

If you choose this option, refer to the sections of the manual referring to calibration data first. This includes the section Importing Data Logger Files - Calibration.

3.        Turn on or off the Screen Data during Import option.

For more information, reading about the screening options in the Importing Data Logger Files - Settings section of the manual.

4.        Press the Last Folder Used button from the Import Files From section located on the right side of the import form. Select one or more logger files to import. As the import proceeds you will receive informational messages as deemed necessary.