Apache Flume Interceptors
Interceptors are used to modify or drop events as they are being processed. Flume has this capability because it utilizes interceptors. These interceptors also determine which types of data should pass through to the channel.
The interceptors are invoked in the order they are set. The list of events returned by one interceptor is passed on to the next interceptor in the chain. If an interceptor drops events, it does not include those events in the list that it returns. If it is designed to drop all events, it simply returns an empty list.
Below is a list of available interceptors:
- Timestamp Interceptor
- Host Interceptor
- Static interceptor
- Regex filtering interceptor
Timestamp Interceptor -
This interceptor adds the current timestamp to the event headers when processing the event. If the event already has a timestamp header, it will be overwritten with the current time, unless it is configured to preserve the original value through specific settings.
<agent-name>.sources = <source_name>
<agent-name>.channels = <channel-name>
<agent-name>.sources.<source_name>.channels = <channel-name>
<agent-name>.sources.<source_name>.type = seq
<agent-name>.sources.<source_name>.interceptors = <interceptor_name>
<agent-name>.sources.<source_name>.interceptors.<interceptor_name>.type
= timestamp
Example for agent named agt -
agt.sources = sr1
agt.channels = chn1
agt.sources.sr1.channels = chn1
agt.sources.sr1.type = seq
agt.sources.sr1.interceptors = ic1
agt.sources.sr1.interceptors.ic1.type = timestamp
Below are the properties for Timestamp Interceptor -
| Property Name | Default | Description |
|---|---|---|
| type | – | The component type name (timestamp) |
| preserveExisting | false | If the timestamp already exists, should it be preserved |
Host Interceptor -
This interceptor adds the host name or IP address to the event headers during event processing.
If the event headers already contain an IP address or host name, the current IP address or host name will overwrite the existing value, unless configured otherwise to preserve the original value through specific settings.
<agent-name>.sources = <source_name>
<agent-name>.channels = <channel-name>
<agent-name>.sources.<source_name>.interceptors = <interceptor_name>
<agent-name>.sources.<source_name>.interceptors.<interceptor_name>.type
= host
<agent-name>.sources.<source_name>.interceptors.<interceptor_name>.hostHeader
= hostname
Example for agent named agt -
agt.sources = sr1
agt.channels = chn1
agt.sources.sr1.interceptors = ic1
agt.sources.sr1.interceptors.ic1.type = host
agt.sources.sr1.interceptors.ic1.hostHeader = hostname
Below are the properties for host Interceptor -
| Property Name | Default | Description |
|---|---|---|
| type | – | The component type name(host) |
| preserveExisting | false | If the host header already exists, should it be preserved |
| useIP | true | IP Address if true, else use hostname. |
| hostHeader | host | The header key to be used. |
Static Interceptor -
The static interceptor allows users to append a static header to all events.
The current implementation does not support multiple headers simultaneously. Instead, users can chain multiple static interceptors, each defining a single static header.
<agent-name>.sources = <source_name>
<agent-name>.channels = <channel-name>
<agent-name>.sources.<source_name>.channels = <channel-name>
<agent-name>.sources.<source_name>.type = seq
<agent-name>.sources.<source_name>.interceptors = <interceptor_name>
<agent-name>.sources.<source_name>.interceptors.<interceptor_name>.type
= timestamp
<agent-name>.sources.<source_name>.interceptors.<interceptor_name>.key
= <key-name>
<agent-name>.sources.<source_name>.interceptors.<interceptor_name>.value
= <static-value>
Below are the properties for static Interceptor -
| Property Name | Default | Description |
|---|---|---|
| type | – | The component type name(static) |
| preserveExisting | true | If configured header already exists, should it be preserved |
| key | key | Name of header |
| value | value | Static value |
Regex filtering Interceptor -
This interceptor selectively filters events by analyzing the event body as text. It matches the text against a configured regular expression that can include or exclude specific events.
| Property Name | Default | Description |
|---|---|---|
| type | – | The component type name (regex_filter) |
| regex | ”.*” | Regular expression |
| excludeEvents | false | If true, regex determines events to exclude, otherwise regex determines events to include. |