Module: LogStash

Defined in:

lib/logstash/inputs/file.rb,
lib/logstash/inputs/file_listener.rb,
lib/logstash/inputs/friendly_durations.rb,
lib/logstash/inputs/log_completed_file_handler.rb,
lib/logstash/inputs/delete_completed_file_handler.rb

Overview

If you would like to join multiple log lines into one event, you’ll want to use the multiline codec or filter.

The plugin aims to track changing files and emit new content as it’s appended to each file. It’s not well-suited for reading a file from beginning to end and storing all of it in a single event (not even with the multiline codec or filter).

Reading from remote network volumes

The file input is not tested on remote filesystems such as NFS, Samba, s3fs-fuse, etc. These remote filesystems typically have behaviors that are very different from local filesystems and are therefore unlikely to work correctly when used with the file input.

Tracking of current position in watched files

The plugin keeps track of the current position in each file by recording it in a separate file named sincedb. This makes it possible to stop and restart Logstash and have it pick up where it left off without missing the lines that were added to the file while Logstash was stopped.

By default, the sincedb file is placed in the home directory of the user running Logstash with a filename based on the filename patterns being watched (i.e. the ‘path` option). Thus, changing the filename patterns will result in a new sincedb file being used and any existing current position state will be lost. If you change your patterns with any frequency it might make sense to explicitly choose a sincedb path with the `sincedb_path` option.

A different ‘sincedb_path` must be used for each input. Using the same path will cause issues. The read checkpoints for each input must be stored in a different path so the information does not override.

Sincedb files are text files with four columns:

. The inode number (or equivalent). . The major device number of the file system (or equivalent). . The minor device number of the file system (or equivalent). . The current byte offset within the file.

On non-Windows systems you can obtain the inode number of a file with e.g. ‘ls -li`.

File rotation

File rotation is detected and handled by this input, regardless of whether the file is rotated via a rename or a copy operation. To support programs that write to the rotated file for some time after the rotation has taken place, include both the original filename and the rotated filename (e.g. /var/log/syslog and /var/log/syslog.1) in the filename patterns to watch (the ‘path` option). Note that the rotated filename will be treated as a new file so if `start_position` is set to ’beginning’ the rotated file will be reprocessed.

With the default value of ‘start_position` (’end’) any messages written to the end of the file between the last read operation prior to the rotation and its reopening under the new name (an interval determined by the ‘stat_interval` and `discover_interval` options) will not get picked up.

Defined Under Namespace

Modules: Inputs