Live File Processor Agent Folder

<< Click to Display Table of Contents >>

Navigation:  Live > Agents > File Agents > File Processor > Settings >

Live File Processor Agent Folder

 

Setting

Type

Description

folderWatcherFilter

String

Only process files matching this wildcard (e.g., *.csv) and the folderWatcherFilterRegex regex pattern.

 

Default Value: *.*

folderWatcherFilterRegex

Regex

Only process files matching the folderWatcherFilter wildcard and this regex pattern (see E-BAM example).

 

If this setting is empty or not defined, it does not filter anything out.

 

The FileSystemWatcher (and EnumerateFiles, used by the loaddata command line option) does not support regex; it uses the folderWatcherFilter wildcard. This regex is the secondary filter.

folderWatcherPath

String

Watch this folder path for new files, or new file data, and automatically load the data into the EQuIS Live database.

 

Type

Example

Local

C:\EarthSoft\Live\HOBO

UNC

\\servername\path

FTP

ftp://user:password@host:port/path

 

 

Local and UNC

 

Setting

Type

Description

folderWatcherBufferSize

Integer

The folder watcher's FileSystemWatcher.InternalBufferSize, in bytes. The default is 8192 bytes (8 KB), the minimum is 4096 bytes (4 KB), and the maximum is 65536
(64 KB). If there are many file changes in a short time, the buffer can overflow, and some files may not be processed. Increasing the size of the buffer can prevent missing file system change events. However, it is expensive, because it comes from non-paged memory that cannot be swapped out to disk. Therefore, keep the buffer as small as possible.

 

If the buffer overflows, an error is logged, e.g., InternalBufferOverflowException (Too many changes at once in directory:C:\EarthSoft\Live\Lorentz.), in file %env{temp}/equislogs/{process}.log.

 

Default Value: 8192 (bytes)

folderWatcherDelay

Integer

Delay loading file data, by this number of milliseconds, after a file change is detected.

 

The FileSystemWatcher, used by the agent to watch the files and folders, may detect MULTIPLE near-same-time OnChanged events for ONE file system operation (e.g., copying a file). To prevent a file from being processed multiple times for one file system operation, the agent starts/resets a timer when it detects an OnChanged event that fires after this number of milliseconds (e.g., 500 ms) and processes the file.

 

The timer also frees up the FileSystemWatcher thread, by processing the files on a ThreadPool thread. Thus, the folder watcher's FileSystemWatcher.InternalBufferSize does not need to be too big.

 

This setting may need to be increased for larger files that take longer to copy. (The FileSystemWatcher may not get an OnChanged event when the copying completes.)

 

Default Value: 1000 (milliseconds)

 

 

FTP

 

Note: A local or UNC folder path can also be used for an FTP folder, if the FTP server is running on the same computer as this agent (use a local folder path), or if the FTP folder is shared over the network (use a UNC folder path). (It will be more efficient because it will not need to connect to, and download the files from, the FTP server.)

 

 

Setting

Type

Description

ftpDeleteFiles

Boolean

Delete files from FTP folder after they're processed?

ftpMode


Active or Passive.

ftpPollingInterval

Time Span

[d.]hh:mm:ss, e.g., 00:03:21 = 3 minutes + 21 seconds, or 01:23:45 = 1 hour + 23 minutes + 45 seconds.

 

This agent checks (polls) the FTP folder periodically (at this interval, using a timer), and downloads and processes files that have been added to, or modified in, the FTP folder since the last time it checked. (The time between checks will be this ftpPollingInterval, plus the time that it takes to list, download, and process the files.)

 

Default Value: 00:15:00

ftpEncryptedPassword

String

The user and password can also be added to the FTP URI, in folderWatcherPath, if they don't contain any disallowed characters (e.g., : or @ or #), and don't need to be encrypted.

 

The encrypt command line option can be used to encrypt the user and password.

ftpEncryptedUser

ftpPassword

ftpUser

logFtpFileListings

Logging Level

Log the list of files in the FTP folder, every ftpPollingInterval (even if no files have changed)?

 

Default Value: Off