Summary -

In this topic, we described about the below sections -

The interceptors used to modify/drop events in-flight. Flume has the capability as it uses the interceptors. The interceptor also decides what sort of data should pass through to the Channel.

An interceptor can modify/drop events based on any criteria chosen by the developer of the interceptor. Flume supports binding of interceptors. Interceptors are stated as a whitespace separated list in the source configuration.

The interceptors are specified in the order and in the same order which they are invoked. The list of events returned by one interceptor is passed to the next interceptor in the chain.

If an interceptor drop events, it does not return the event in the list that it should returns. If it is to drop all events, then it simply returns an empty list. The interceptors are themselves configurable and can be passed configuration values.

The below are the list of interceptors available

  1. Timestamp Interceptor
  2. Host Interceptor
  3. Static interceptor
  4. Regex filtering interceptor

Timestamp Interceptor -

This interceptor inserts the time in milli-seconds into the event headers, at which it processes the event. This interceptor inserts a header with the relevant timestamp. If the event already contains a timestamp header, it will be overwritten with the current time unless configured to preserve the original value by settings.

<agent-name>.sources = <source_name>
<agent-name>.channels = <channel-name>
<agent-name>.sources.<source_name>.channels =  <channel-name>
<agent-name>.sources.<source_name>.type = seq
<agent-name>.sources.<source_name>.interceptors = <interceptor_name>
<agent-name>.sources.<source_name>.interceptors.<interceptor_name>.type 
			= timestamp

Example for agent named agt -

agt.sources = sr1 
agt.channels = chn1
agt.sources.sr1.channels =  chn1
agt.sources.sr1.type = seq
agt.sources.sr1.interceptors = ic1
agt.sources.sr1.interceptors.ic1.type = timestamp

Below are the properties for Timestamp Interceptor -

Property NameDefaultDescription
typeThe component type name (timestamp)
preserveExistingfalseIf the timestamp already exists, should it be preserved

Host Interceptor -

This interceptor inserts the host name or IP address into the event headers, at which it processes the event. This interceptor inserts a header with the host name or IP address.

If the event already contains a IP address or host name in header, it will be overwritten with the current IP address or host name unless configured to preserve the original value by settings.

<agent-name>.sources = <source_name>
<agent-name>.channels = <channel-name>
<agent-name>.sources.<source_name>.interceptors = <interceptor_name>
<agent-name>.sources.<source_name>.interceptors.<interceptor_name>.type
					= host
<agent-name>.sources.<source_name>.interceptors.<interceptor_name>.hostHeader
					= hostname

Example for agent named agt -

agt.sources = sr1 
agt.channels = chn1
agt.sources.sr1.interceptors = ic1
agt.sources.sr1.interceptors.ic1.type = host
agt.sources.sr1.interceptors.ic1.hostHeader = hostname

Below are the properties for host Interceptor -

Property NameDefaultDescription
typeThe component type name(host)
preserveExistingfalseIf the host header already exists, should it be preserved
useIPtrueIP Address if true, else use hostname.
hostHeaderhostThe header key to be used.

Static Interceptor -

Static interceptor allows user to append a static header to all events. Static interceptor allows user to append a static value to all events.

The current implementation does not allow multiple headers at one time. Instead user might chain multiple static interceptors each defining one static header.

<agent-name>.sources = <source_name>
<agent-name>.channels = <channel-name>
<agent-name>.sources.<source_name>.channels =  <channel-name>
<agent-name>.sources.<source_name>.type = seq
<agent-name>.sources.<source_name>.interceptors = <interceptor_name>
<agent-name>.sources.<source_name>.interceptors.<interceptor_name>.type
						= timestamp
<agent-name>.sources.<source_name>.interceptors.<interceptor_name>.key
						= <key-name>
<agent-name>.sources.<source_name>.interceptors.<interceptor_name>.value
						= <static-value>

Below are the properties for static Interceptor -

Property NameDefaultDescription
typeThe component type name(static)
preserveExistingtrueIf configured header already exists, should it be preserved
keykeyName of header
valuevalueStatic value

Regex filtering Interceptor -

This interceptor filters events selectively by understanding the event body as text. This interceptor matches the text against a configured regular expression. The provided regular expression can be used to include events or exclude events.

Property NameDefaultDescription
typeThe component type name (regex_filter)
regex”.*”Regular expression
excludeEventsfalseIf true, regex determines events to exclude, otherwise regex determines events to include.