LimaCharlie Universal Sensor Protocol Adapter
USP Adapters to allow ingest from various sources into LimaCharlie.
Usage
The adapter can be used to access many different sources and many different event types. The main mechanisms specifying the source and type of events are:
- Adapter Type: this indicates the technical source of the events, like syslog or S3 buckets.
- Platform: the platform indicates the type of events that are acquired from that source, like text or carbon_black.
Depending on the Adapter Type specified, configurations that can be specified will change. Running the adapter with no command line arguments will list
all available Adapter Types and their configurations.
Configurations can be provided to the adapter in one of three ways:
- By specifying a configuration file.
- By specifying the configurations via the command line in the format
config-name=config-value.
- By specifying the configurations via the environment variables in the format
config-name=config-value.
All Adapter Types support the same client_options, plus type-specific configurations. You should always specify the following configurations:
client_options.identity.oid: the LimaCharlie Organization ID (OID) this adapter is used with.client_options.identity.installation_key: the LimaCharlie Installation Key this adapter should use to identify with LimaCharlie.client_options.platform: the type of data ingested through this adapter, like text, json, gcp or carbon_black.client_options.sensor_seed_key: an arbitrary name for this adapter which Sensor IDs (SID) are generated from, see below.
Other common configurations that are not required include:
client_options.mapping.parsing_re: regular expression with named capture groups. The name of each group will be used as the key in the converted JSON parsing.client_options.mapping.sensor_key_path: indicates which component of the events represent unique sensor identifiers.client_options.mapping.sensor_hostname_path: indicates which component of the event represents the hostname of the resulting Sensor in LimaCharlie.client_options.mapping.event_type_path: indicates which component of the event represents the Event Type of the resulting event in LimaCharlie.client_options.mapping.event_time_path: indicates which component of the event represents the Event Time of the resulting event in LimaCharlie.client_options.mapping.rename_only: if true, indicates the field mappings defined here should only be renamed fields from the original event (and not completely replacing the original event).client_options.mapping.mappings: a list of field remapping to be performed:
src_field: the source component to remap.dst_field: what the SourceField should be mapped to in the final event.
All other configurations in client_options should only be set for advanced ingestion schemes.
Graceful shutdown
Adapter Close stops its producer and waits up to 60 seconds for the USP proxy
to acknowledge accepted events. A producer blocked by a full buffer is unblocked
after 10 seconds. Mailbox clients drain concurrently. Pub/Sub also gets up to
10 seconds to flush source acknowledgements before its RPC transport closes.
Current proxies acknowledge on receipt. An already-written idle tail can wait
for their 30-second periodic ACK, so the 60-second drain timeout must remain
comfortably above that interval. Proxies that acknowledge downstream acceptance
provide stronger confirmation through the same protocol.
If confirmation times out, Close returns an error and OnWarning reports the
unconfirmed count and sequence range without payloads. Sources already
acknowledged or checkpointed cannot be replayed generically; operators should
treat this as uncertain delivery. Hosts using go-uspclient directly can recover
the unacknowledged messages returned by its Close method.
Logging
The adapter writes its debug (DBG) and warning (WRN) lines to stdout and
its errors (ERR) to stderr. So that a failure retried in a loop (an
unreachable API, a TLS error) cannot flood the host's logs, the output is
deduplicated:
- The first occurrence of a message is always printed as is.
- Repeats of it within one minute are counted, then reported in a single
last message repeated N more time(s) between <first> and <last>: <message>
line carrying the most recent text. A message still repeating gets one such
line per minute. A message that went quiet for one to two minutes is
printed in full the next time it happens.
- Ephemeral ports (32768 and up) following an IP address are ignored when
comparing messages, so the same error from the same host on a new
connection counts as a repeat. A different address or service port is a
different message.
- Each adapter has its own limits, and each level is rate limited: up to 3000
lines at once, refilled at 3000 per minute. Lines over the limit are counted
and reported within seconds in a
log rate limit ... reached: dropped N ... line(s) line that includes the last dropped message. A flood of debug lines
never delays warnings or errors, and one adapter never uses up another's
budget.
- Pending counts are reported when the adapter exits.
Set the environment variable USP_LOG_NO_DEDUP=1 to print every line
unchanged.
Multi-Adapter
If running using a configuration file, it is possible to run multiple instances of the same adapter type within a single instance of the adapter
process. To do this, simply make the config file multiple "documents" (in JSON or YAML format) within the same file, like:
file:
client_options.identity.oid: XXXXX
file_path: ./file_1
...
---
file:
client_options.identity.oid: XXXXX
file_path: ./file_2
...
---
...
This is the beginning of the README. Read the rest at the original. Read the full README ↗
