yaf - Yet Another Flowmeter
yaf [--in INPUT_SPECIFIER] [--out OUTPUT_SPECIFIER] [--live LIVE_TYPE] [--ipfix TRANSPORT_PROTOCOL] [--filter BPF_FILTER] [--rotate ROTATE_DELAY] [--lock] [--caplist] [--group SPREAD_GROUP_NAME(s)] [--groupby GROUPBY_TYPE] [--stats INTERVAL][--no-stats] [--noerror] [--dag-interface] [--napatech-interface] [--gre-decode] [--no-frag] [--max-frags FRAG_TABLE_MAX] [--ip4-only] [--ip6-only] [--idle-timeout IDLE_TIMEOUT] [--active-timeout ACTIVE_TIMEOUT] [--udp-temp-timeout TEMPLATE_TIMEOUT] [--max-payload PAYLOAD_OCTETS] [--udp-payload] [--max-flows FLOW_TABLE_MAX] [--export-payload] [--silk] [--udp-uniflow PORT] [--uniflow] [--mac] [--force-ip6-export] [--observation-domain DOMAIN_ID] [--entropy] [--applabel] [--applabel-rules RULES_FILE] [--ipfix-port PORT] [--tls] [--tls-ca CA_PEM_FILE] [--tls-cert CERT_PEM_FILE] [--tls-key KEY_PEM_FILE] [--become-user UNPRIVILEGED_USER] [--become-group UNPRIVILEGED_GROUP] [--log LOG_SPECIFIER] [--loglevel LOG_LEVEL] [--verbose] [--version] [--p0fprint] [--fpexport] [--plugin-name LIBPLUGIN_NAME[,LIBPLUGIN_NAME...]] [--plugin-opts "OPTIONS[,OPTIONS...]"]
yaf
is Yet Another Flowmeter and yaf
is a suite of tools to do flow metering.
yaf
is used as a sensor to capture flow information on a network and export that
information in IPFIX format. It reads packet data from pcap(3) dumpfiles as
generated by tcpdump(1), from live capture from an interface using pcap(3),
an Endace DAG capture device, or a Napatech adapter, aggregates these packets
into flows, and exports flow records via IPFIX over SCTP, TCP or UDP, Spread, or into
serialized IPFIX message streams (IPFIX files) on the local file system.
Since yaf
is designed to be deployed on white-box sensors attached to local
network segments or span ports at symmetric routing points, it supports
bidirectional flow assembly natively. Biflow export is done via the export
method specified in RFC 5103 Bidirectional Flow Export using IPFIX. See the
OUTPUT section below for information on this format.
yaf
also supports experimental partial payload capture, specifically for
banner-grabbing applications and protocol verification purposes.
The output of yaf
is designed to be collected and manipulated by flow
processing toolchains supporting IPFIX. The yafscii(1) tool, which is
installed as part of YAF, can also be used to print yaf
output in a
human-readable format somewhat reminiscient of tcpdump(1). yaf
output
can also be analyzed using the SiLK suite, and the nafalize(1) tool, both
available from the CERT NetSA group.
These options control where yaf
will take its input from. yaf
can read packets
from a pcap dumpfile (as generated by tcpdump -w
) or live from an interface
via libpcap, libdag, or libpcapexpress (Napatech Adapter library).
By default, if no input options are given, yaf
reads a pcap dumpfile
on standard input.
INPUT_SPECIFIER is an input specifier. If --live is given, this is the
name of an interface (e.g. eth0
, en0
, dag0
) to capture packets from.
Otherwise, it is a filename; the string - may be used to read from
standard input (the default).
If present, treat the filename in INPUT_SPECIFIER as an ordered newline-delimited list of pathnames to pcap(3) dumpfiles. Blank lines and lines beginning with the character '#' within this are ignored. All pathnames are evaluated with respect to the working directory yaf is run in. These dumpfiles are processed in order using the same flow table, so they must be listed in ascending time order. This option is intended to ease the use of yaf with rotated or otherwise split tcpdump(1) output.
Used with the --caplist option. When present, this prevents yaf
from
exiting when processing a list of dumpfiles in the middle due to an error
in a file. yaf
will continue to process all files given in the
INPUT_SPECIFIER despite errors within those files.
If present, capture packets from an interface named in the INPUT_SPECIFIER.
LIVE_TYPE is one of pcap for packet capture via libpcap, or dag for
packet capture via an Endace DAG interface using libdag, or napatech for
packet capture via a Napatech Adapter using libpcapexpress. dag is only
available if yaf
was built with Endace DAG support. napatech is only
available if yaf
was built with libpcapexpress, a library maintained by nPulse
Technologies for use with a Napatech adapter.
If present, the interface on which a packet was received will be noted internally
within yaf
. When flow records are exported from yaf
, an ingressinterface
and an egressinterface
set of fields will be added to the output. The
ingressinterface
field will be the physical interface which captured the packet
while the egressinterface
will be the physical interface | 0x100. This can
be used to separate traffic based on DAG physical ports. This option requires
building yaf
with the --enable-daginterface
switch.
Similar to --dag-interface, if present, the interface on which a packet was
received will be noted internally within yaf
. When flow records are exported
from yaf
, an ingressinterface
and an egressinterface
set of fields will
be added to the output. The ingressinterface
field will be the physical interface
which captured the packet while the egressinterface
will be the the physical
interface | 0x100. This can be used to separate traffic based on Napatech physical
ports. This option requires building yaf
with the --enable-napatechinterface
switch.
If present, enable Berkeley Packet Filtering (BPF) in yaf
with FILTER_EXPRESSION as the incoming traffic filter. Syntax of FILTER_EXPRESSION follows the expression format described in the tcpdump(1) man page. This option is not currently supported if --live is set to dag or napatech as BPF filtering is implemented with libpcap. This option is also not supported on a Bivio due to the Bivio supplied version of libpcap that supports the zero-copy driver.
These options control where yaf
will send its output. yaf
can write flows
to an IPFIX file or export flows to an IPFIX collector over SCTP, TCP, UDP, or Spread.
By default, if no output options are given, yaf
writes an IPFIX file to
standard output.
OUTPUT_SPECIFIER is an output specifier. If --ipfix is present, the OUTPUT_SPECIFIER specifies the hostname or IP address of the collector to which the flows will be exported. Otherwise, if --rotate is present, OUTPUT_SPECIFIER is the prefix name of each output file to write to. If --ipfix is present and set to spread, then OUTPUT_SPECIFIER should be set to the name of the Spread daemon to connect to (See below examples of spread daemon names). Otherwise, OUTPUT_SPECIFIER is a filename in which the flows will be written; the string - may be used to write to standard output (the default).
If present, causes yaf
to operate as an IPFIX exporter, sending
IPFIX Messages via the specified transport protocol to the collector (e.g.,
SiLK's rwflowpack or flowcap facilities) named in the OUTPUT_SPECIFIER.
Valid TRANSPORT_PROTOCOL values are tcp, udp, sctp, and spread;
sctp is only available if yaf
was built with SCTP support; spread is only available if yaf
was built with Spread support. UDP is not recommended, as it is not a reliable transport protocol, and cannot guarantee delivery of messages. As per the recommendations in RFC 5101, yaf
will retransmit templates three times within the template timeout period (configurable using --udp-temp-timeout or by default, 10 minutes). Use the
--ipfix-port, --tls, --tls-ca, --tls-cert, --tls-key, --tls-pass, and --group options to further configure the connection to the
IPFIX collector.
If present, causes yaf
to write output to multiple files, opening a new output
file every ROTATE_DELAY seconds in the input data. Rotated files are named
using the prefix given in the OUTPUT_SPECIFIER, followed by a suffix
containing a timestamp in YYYYMMDDhhmmss
format, a decimal serial number,
and the file extension .yaf.
Use lockfiles for concurrent file access protection on output files. This is recommended for interoperating with the Airframe filedaemon facility.
If present, causes yaf
to export process statistics every INTERVAL seconds.
The default value for INTERVAL is 300 seconds or every 5 minutes. yaf
uses IPFIX Options Templates and Records
to export flow, fragment, and decoding statistics. If INTERVAL is set
to zero, stats will not be exported.
If present, yaf
will not export process statistics. yaf
uses IPFIX
Options Templates and Records to export flow, fragment, and decoding
statistics. --no-stats takes precedence over --stats.
These options are used to modify the yaf
packet decoder's behavior. None of
these options are required; the default behavior for each option when not
present is noted.
If present, ignore all fragmented packets. By default, yaf
will reassemble
fragments with a 30 second fragment timeout.
If present, limit the number of outstanding, not-yet reassembled fragments
in the fragment table to FRAG_TABLE_MAX by prematurely expiring fragments
from the table. This option is provided to limit yaf
resource usage when
operating on data from very large networks or networks with abnormal
fragmentation. The fragment table may exceed this limit slightly due
to limits on how often yaf
prunes the fragment table (every 5 seconds).
By default, there is no fragment table limit, and the fragment
table can grow to resource exhaustion.
If present, ignore all IPv6 packets and export IPv4 flows only. The default is to process both IPv4 and IPv6 packets.
If present, ignore all IPv4 packets and export IPv6 flows only. The default is to process both IPv4 and IPv6 packets.
If present, attempt to decode GRE version 0 encapsulated packets. Flows will be created from packets within the GRE tunnels. Undecodeable GRE packets will be dropped. Without this option, GRE traffic is exported as IP protocol 47 flows. This option is presently experimental.
These options are used to modify the flow table behavior within yaf
. None of
these options are required; the default behavior for each option when not
present is noted.
Set flow idle timeout in seconds. Flows are considered idle and flushed from the flow table if no packets are received for IDLE_TIMEOUT seconds. The default flow idle timeout is 300 seconds (5 minutes).
Set flow active timeout in seconds. Any flow lasting longer than ACTIVE_TIMEOUT seconds will be flushed from the flow table. The default flow active timeout is 1800 seconds (30 minutes).
Set UDP template timeout in seconds if --ipfix is set to udp. As per RFC 5101 recommendations, yaf
will attempt to export templates three times within TEMPLATE_TIMEOUT. The
default template timeout period is 600 seconds (10 minutes).
If present, capture at most PAYLOAD_OCTETS octets from the start of
each direction of each flow. Non-TCP flows will only capture payload
from the first packet unless --udp-payload is set. If not present, yaf
will not attempt to capture
payload. Payload capture must be enabled for payload export
(--export-payload), application labeling (--applabel), and entropy
evaluation (--entropy). Note that payload capture is still an
experimental feature.
If present, limit the number of open flows in the flow table to
FLOW_TABLE_MAX by prematurely expiring the flows with the least
recently received packets; this is analogous to an adaptive idle
timeout. This option is provided to limit yaf
resource usage when
operating on data from large networks. By default, there is no flow
table limit, and the flow table can grow to resource exhaustion.
If present, capture at most PAYLOAD_OCTETS octets fom the start of each direction of each UDP flow, where PAYLOAD_OCTETS is set using the --max-payload flag.
If present, export flows in "SiLK mode". As of YAF 2.0, this will export TCP information (flags, ISN) in the main flow record instead of within the SubTemplateMultiList. This flag must be used when exporting to SiLK for it to collect TCP flow information. This also introduces the following incompatibilities with standard IPFIX export:
totalOctetCount and reverseTotalOctetCount are clamped to 32 bits. Any packet that would cause either of these counters to overflow 32 bits will cause the flow to close with flowEndReason 0x02 (active timeout), and will become the first packet of a new flow. This is analogous to forcing an active timeout when the octet counters overflow.
The high-order bit of the flowEndReason IE is set on any flow created on a counter overflow, as above.
The high-order bit of the flowEndReason IE is set on any flow created on an active timeout.
Since this changes the semantics of the exported flowEndReason IE, it should only be used when generating flows and exporting to rwflowpack, flowcap, or writing files for processing with rwipfix2silk.
These options are used to modify the the data exported by yaf
.
If present, export at most PAYLOAD_OCTETS (the argument to
--max-payload) octets from the start of each direction of each
flow. Non-TCP flows will only export payload from the first packet. By default,
yaf
will not export flow payload.
If present, export biflows using the Record Adjacency method in section 3 of RFC 5103. This is useful when exporting to IPFIX Collecting Processes that are not biflow-aware.
If present, export MAC-layer information; presently, exports source and destination MAC addresses.
If present, force IPv4 flows to be exported with IPv6-mapped IPv4 addresses in ::FFFF/96. This will cause all flows to appear to be IPv6 flows.
Set the observationDomainID on each exported IPFIX message to the given
integer value. If not present, the observationDomainId defaults to 1. This
value is also used as the exportingProcessId in the yaf
statistics Option
Record as a Scope Field.
If present, export each UDP packet on the given port (or 1 for all ports) as a single flow, with flowEndReason set to YAF_END_UDPFORCE (0x1F). This will not close the flow. The flow will stay open until it closes naturally by the idle and active timeouts. Most useful with --export-payload in order to export every UDP payload on a specific port.
If yaf
is built with application labeler support enabled (using the
--enable-applabel
option to ./configure
when yaf
is built), then
yaf
can examine packet payloads and determine the application protocol
in use within a flow, and export a 16-bit application label with
each flow.
The exported application label uses the common port number for the
protocol. For example, HTTP traffic, independent of what port the
traffic is detected on, will be labeled with a value of 80, the
default HTTP port. Labels and rules are taken from a configuration
file read at yaf
startup time.
Application labeling requires payload capture to be enabled with the --max-payload option. A minimum payload capture length of 384 octets is recommended for best results.
Application labeling is presently experimental. SiLK does support
IPFIX import and translation of the application label via
rwflowpack
, flowcap
, and rwipfix2silk
.
If present, export application label data. Requires --max-payload to enable payload capture.
Read application labeler rules from RULES_FILE. If not present, rules are read by default from /usr/local/etc/yafApplabelRules.conf.
If yaf
is built with entropy measurement enabled (using the
--enable-entropy
option to ./configure
when yaf
is built,)
then yaf
can examine the packet payloads and determine a Shannon
Entropy value for the payload. The entropy calculation does not
include the network (IP) or transport (UDP/TCP) headers. The entropy
is calculated in terms of bits per byte, (log base 2.) The
calculation generates a real number value between 0.0 and 8.0. That
number is then converted into an 8-bit integer value between 0 and
255. Roughly, numbers above 230 are generally compressed (or encrypted)
and numbers centered around approximately 140 are English text. Lower
numbers carry even less information content. Another useful piece of
information is that SSL/TLS tends to zero pad its packets, which causes
the entropy of those flows to drop quite low.
If present, export the entropy values for both the forward and reverse payloads. Requires the --max-payload option to operate.
These options are used to configure the connection to an IPFIX collector.
If --ipfix is present, export flows to TCP, UDP, or SCTP port PORT. If not present, the default IPFIX port 4739 is used. If --tls is also present, the default secure IPFIX port 4740 is used.
If --ipfix is present, use TLS to secure the connection to the IPFIX collector. Requires the TRANSPORT_PROTOCOL to be tcp, as DTLS over UDP or SCTP is not yet supported. Requires the --tls-ca, --tls-cert, and --tls-key options to specify the X.509 certificate and TLS key information.
Use the Certificate Authority or Authorities in CA_PEM_FILE to verify the remote IPFIX Collecting Process' X.509 certificate. The connection to the Collecting Process will fail if its certificate was not signed by this CA (or by a certificate signed by this CA, recursively); this prevents export to unauthorized Collecting Processes. Required if --tls is present.
Use the X.509 certificate in CERT_PEM_FILE to identify this IPFIX Exporting Process. This certificate should contain the public part of the private key in KEY_PEM_FILE. Required if --tls is present.
Use the private key in KEY_PEM_FILE for this IPFIX Exporting Process. This key should contain the private part of the public key in CERT_PEM_FILE. Required if --tls is present. If the key is encrypted, the password must be present in the YAF_TLS_PASS environment variable.
If --ipfix is present and set to spread, use --group to specify the spread group name(s) to publish output.
It is possible to list more than one group name in a comma-seperated list.
To use Spread as a manifold for different types of flows, use the format
GROUP, GROUP_NAME:VALUE, GROUP_NAME:VALUE as the argument to --group
and use the --groupby switch. This list should be contained in quotes if it contains spaces (yaf
will ignore spaces in quotes). It is suggested to use one group
as the catchall for all flows (no value listed) so flows are not lost.
The --groupby switch must be used if --group uses GROUP:VALUE
format. See the Spread Documentation, www.spread.org,
for more details on Spread.
If --group is used with group values, use --groupby to specify what type of value should be used. Options are port, vlan, applabel, protocol, version. --groupby accepts only one argument. The port option is destination transport port of the flow. version is the IP version of the flow.
These options are used to cause yaf
to drop privileges when running as root
for live capture purposes.
After opening the live capture device in --live mode, drop
privilege to the named user. Using --become-user requires yaf
to
be run as root or setuid root. This option will cause all files
written by yaf
to be owned by the user UNPRIVILEGED_USER and the
user's primary group; use --become-group as well to change the group
yaf
runs as for output purposes.
If running as root for live capture purposes and --become-user is not
present, yaf
will warn that privilege is not being dropped. We highly
recommend the use of this option, especially in production environments,
for security purposes.
--become-group can be used to change the group from the default of the user given in --become-user. This option has no effect if given without the --become-user option as well.
These options are used to specify how log messages are routed. yaf
can
log to standard error, regular files, or the UNIX syslog facility.
Specifies destination for log messages. LOG_SPECIFIER can be a syslog(3)
facility name, the special value stderr for standard error, or the
absolute path to a file for file logging. The default log
specifier is stderr if available, user otherwise.
Specify minimum level for logged messages. In increasing levels of verbosity, the supported log levels are quiet, error, critical, warning, message, info, and debug. The default logging level is warning.
Equivalent to --loglevel debug.
If present, print version and copyright information to standard error and exit.
These options are used to load, configure, and run a yaf
plugin.
Specify the plugin to load. The loaded plugin must follow the yaf
plugin framework. LIBPLUGIN_NAME must be the full path to the plugin library name. The only plugin currently supported is the Deep Packet Inspection plugin. This flag will only be recognized if yaf
is configured with --enable-plugins. There are also configure options to export only DNS Authoritative and NXDomain responses. Read the document on the DPI Plugin for more information.
Specify the arguments to the plugin given to --plugin-name. This flag will only be recognized if yaf
is configured with --enable-plugins and --plugin-name is set to a valid plugin. For example, the DPI Plugin takes the well-known port of a protocol(s) to enable DPI (default for DPI is all protocols).
These options are used to enable p0f in yaf
. p0f is presently experimental. There is no support in yafscii
or SiLK for printing p0f related data. Currently, yaf
uses the p0f Version 2 SYN fingerprints (see p0f.fp).
If present, export p0f data. This data consists of three related information elements; osName, osVersion, osFingerPrint. This flag requires yaf
to be configured with --enable-p0fprinter.
If present, enable export of handshake headers for external OS fingerprinters. The related information elements are firstPacketBanner and secondPacketBanner. This flag requires yaf
to be configured with --enable-fpexporter.
yaf
's output consists of an IPFIX message stream. yaf
uses a variety of
templates for IPFIX data records; the information elements that may appear
in these templates are enumerated below. For further information about the IPFIX
information model and IPFIX message stream, see RFC 5102, RFC 5101, and
RFC 5103. As of yaf
2.0, yaf
nests some templates in an IPFIX
subTemplateMultiList. In order to retain compatibility with the SiLK Tools,
use --silk to prevent yaf
from nesting TCP Information Elements. Below
are descriptions of each of the templates YAF will export. See the
Internet-Draft Export of Structured Data in IPFIX for more information on IPFIX lists.
yaf
assigns information element numbers to reverse flow elements in biflow
capture based on the standard IPFIX PEN 29305. This applies only for
information elements defined in the standard IPFIX Information Model
(RFC 5102) that do not have a reverse information element already defined.
For information elements defined under the CERT PEN, a standard method is
used to calculate their reverse element identifier. The method is that
bit fourteen is set to one in the IE field, (e.g. 16384 + the forward IE number.)
Flow start time in milliseconds since 1970-01-01 00:00:00 UTC. Always present.
Flow end time in milliseconds since 1970-01-01 00:00:00 UTC. Always present.
Number of octets in packets in forward direction of flow. Always present. May be encoded in 4 octets using IPFIX reduced-length encoding.
Number of octets in packets in reverse direction of flow. Present if flow has a reverse direction. May be encoded in 4 octets using IPFIX reduced-length encoding.
Number of packets in forward direction of flow. Always present. May be encoded in 4 octets using IPFIX reduced-length encoding.
Number of packets in reverse direction of flow. Present if flow has a reverse direction. May be encoded in 4 octets using IPFIX reduced-length encoding.
Difference in time in milliseconds between first packet in forward direction and first packet in reverse direction. Correlates with (but does not necessarily represent) round-trip time. Present if flow has a reverse direction.
IPv4 address of flow source or biflow initiator. Present for IPv4 flows without IPv6-mapped addresses only.
IPv4 address of flow source or biflow responder. Present for IPv4 flows without IPv6-mapped addresses only.
IPv6 address of flow source or biflow initiator. Present for IPv6 flows or IPv6-mapped IPv4 flows only.
IPv6 address of flow source or biflow responder. Present for IPv6 flows or IPv6-mapped IPv4 flows only.
TCP or UDP port on the flow source or biflow initiator endpoint. Always present.
TCP or UDP port on the flow destination or biflow responder endpoint. Always present. For ICMP flows, contains ICMP type * 256 + ICMP code. This is non-standard, and an open issue in YAF.
Miscellaneous flow attributes for the forward direction of the flow. Always present (YAF 2.1 or later). If all packets in the forward direction have fixed size, flowAttributes will be set to 1.
Miscellaneous flow attributes for the reverse direction of the flow. Always present (YAF 2.1 or later). If all packets in the reverse direction have fixed size, reverseFlowAttributes will be set to 1.
IP protocol of the flow. Always present.
Flow end reason code, as defined by the IPFIX Information Model. Always present. In --silk mode, the high-order bit is set if the flow was created by continuation.
Application label, defined as the primary well-known port associated with a given application. Present if the application labeler is enabled, and was able to determine the application protocol used within the flow.
802.1q VLAN tag of the first packet in the forward direction of the flow.
802.1q VLAN tag of the first packet in the reverse direction of the flow. Present if the flow has a reverse direction.
Represents a list of zero or more instances of a structured data type, where the data type of each list element can be different and corresponds with different template definitions. The Information Element Number will change upon updates to the IPFIX lists specification and libfixbuf releases.
The following six Information Elements will be exported as a template within the subTemplateMultiList unless --silk is used.
Initial sequence number of the forward direction of the flow. Present if the flow's protocolIdentifier is 6 (TCP). This element is contained in the yaf TCP template within the subTemplateMultiList unless --silk is used.
Initial sequence number of the reverse direction of the flow. Present if the
flow's protocolIdentifier is 6 (TCP) and the flow has a reverse direction.
This element is contained in the yaf
TCP template within the subTemplateMultiList unless --silk is used.
TCP flags of initial packet in the forward direction of the flow. Present if the flow's protocolIdentifier is 6 (TCP). This element is contained in the yaf TCP template within the subTemplateMultiList unless --silk is used.
Union of TCP flags of all packets other than the initial packet in the forward direction of the flow. Present if the flow's protocolIdentifier is 6 (TCP). This element is contained in the yaf TCP template within the subTemplateMultiList unless --silk is used.
TCP flags of initial packet in the reverse direction of the flow. Present if the flow's protocolIdentifier is 6 (TCP) and the flow has a reverse direction. This element is contained in the yaf TCP template within the subTemplateMultiList unless --silk is used.
Union of TCP flags of all packets other than the initial packet in the reverse direction of the flow. Present if the flow's protocolIdentifier is 6 (TCP) and the flow has a reverse direction. This element is contained in the yaf TCP template within the subTemplateMultiList unless --silk is used.
The following two Information Elements will be exported as a template within the subTemplateMultiList.
Source MAC Address of the first packet in the forward direction of the flow. This element is contained in the yaf MAC template within the subTemplateMultiList.
Destination MAC Address of the first packet in the reverse direction of the flow. This element is contained in the yaf MAC template within the subTemplateMultiList.
The following two Information Elements will be exported as a template within the subTemplateMultiList.
Initial n bytes of forward direction of flow payload.
Present if payload collection is enabled and payload is present
in the forward direction of the flow. This element is contained in the
yaf
Payload template within the subTemplateMultiList.
Initial n bytes of reverse direction of flow payload.
Present if payload collection is enabled and payload is present
in the reverse direction of the flow. This element is contained in the yaf
Payload template within the subTemplateMultiList.
The following two Information Elements will be exported as a template within the subTemplateMultiList.
Shannon Entropy calculation of the forward payload data. This element is
contained in the yaf
Entropy template within the subTemplateMultiList.
Shannon Entropy calculation of the reverse payload data. This element is
contained in the yaf
Entropy template within the subTemplateMultiList.
The following six Information Elements will be exported as a template within the subTemplateMultiList if present and only if p0f is enabled.
p0f OS Name for the forward flow based on the SYN packet and p0f SYN Fingerprints. Present only if p0f is enabled. This element is contained in the yaf
p0f template within the subTemplateMultiList.
p0f OS Name for the reverse flow based on the SYN packet and p0f SYN Fingerprints. Present only if p0f is enabled. This element is contained in the yaf
p0f template within the subTemplateMultiList.
p0f OS Version for the forward flow based on the SYN packet and p0f SYN Fingerprints. Present only if p0f is enabled. This element is contained in the
yaf
p0f template within the subTemplateMultiList.
p0f OS Version for the reverse flow based on the SYN packet and p0f SYN fingerprints. Present only if p0f is enabled. This element is contained in the
yaf
p0f template within the subTemplateMultiList.
p0f OS Fingerprint for the forward flow based on the SYN packet and p0f SYN fingerprints. Present only if p0f is enabled. This element is contained in the yaf
p0f template within the subTemplateMultiList.
p0f OS Fingerprint for the reverse flow based on the SYN packet and p0f SYN Fingerprints. Present only if p0f is enabled. This element is contained in the yaf
p0f template within the subTemplateMultiList.
The following four Information Elements will be exported as a template within the subTemplateMultiList if present and only if fpexport is enabled.
IP and transport headers for first packet in forward direction to be used for external OS Fingerprinters. Present only if fpexport is enabled. This element is contained in the yaf
FPExport template within the subTemplateMultiList.
IP and transport headers for first packet in reverse direction to be used for external OS Fingerprinters. Present only if fpexport is enabled. This element is contained in the yaf
FPExport template within the subTemplateMultiList.
IP and transport headers for second packet in forward direction (third packet in sequence) to be used for external OS Fingerprinters. Present only if fpexport is enabled. This element is contained in the yaf
FPExport template within the subTemplateMultiList.
IP and transport headers for second packet in reverse direction (currently not used). Present only if fpexport is enabled. This element is contained in the yaf
FPExport template within the subTemplateMultiList.
yaf
can export other templates within the subTemplateMultiList
if plugins are enabled in yaf
. See <a href=../yafdpi.html>yafdpi
</a> for descriptions
of the yaf
Deep Packet Inspection Information Elements.
yaf
will export information about its process periodically
using IPFIX Options Template Record. This record gives information
about the status of the flow and fragment table, as well as decoding
information. This can be turned off using the --no-stats option.
The following Information Elements will be exported:
Total amount of exported flows from yaf
start time.
Total amount of packets processed by yaf
from yaf
start time.
Total amount of dropped packets according to statistics given by libpcap, libdag, or libpcapexpress.
Total amount of packets ignored by the YAF packet decoder, such as
unsupported packet types and incomplete headers, from yaf
start time.
Total amount of fragments that have been expired since yaf
start time.
Total number of packets that been assembled from a series of
fragments since yaf
start time.
Total number of times the yaf
flow table has been flushed
since yaf
start time.
The maximum number of flows in the yaf
flow table at any
one time since yaf
start time.
The IPv4 Address of the yaf
flow sensor.
Set the ID of the yaf
flow sensor by giving a value to
--observation-domain. The default is 1.
The mean flow rate of the yaf
flow sensor since yaf
start time,
rounded to the nearest integer.
The mean packet rate of the yaf
flow sensor since yaf
start time,
rounded to the nearest integer.
yaf
responds to SIGINT or SIGTERM by terminating input processing,
flushing any pending flows to the current output, and exiting. If --verbose
is given, yaf
responds to SIGUSR1 by printing present flow and fragment table
statistics to its log. All other signals are handled by the C runtimes in
the default manner on the platform on which yaf
is currently operating.
To generate flows from an pcap file into an IPFIX file:
yaf --in packets.pcap --out flows.yaf
To capture flows from a pcap interface and export them to files in the current directory rotated hourly:
yaf --live pcap --in en1 --out en1_capture --rotate 3600
To capture flows from an Endace DAG card and export them via IPFIX over TCP:
yaf --live dag --in dag0 --ipfix tcp --out my-collector.example.com
To convert a pcap formatted packet capture and convert that into IPFIX:
yaf <packets.pcap >flows.yaf
To publish to spread group TST_SPRD for a spread daemon running locally on port 4803:
yaf --live pcap --in eth1 --out 4803@localhost --ipfix spread --group TST_SPRD
To publish to spread groups based on application label for spread daemon running locally on port 4803:
yaf --live pcap --in eth1 --out 4803@localhost --ipfix spread --group "SPRD_CATCHALL, SPRD_DNS:53, SPRD_HTTP:80, SPRD_SMTP:25" --groupby applabel --applabel --max-payload=400
To run yaf
with application labeling enabled and export via IPFIX over TCP:
yaf --live pcap --in eth1 --out 127.0.0.1 --ipfix tcp --ipfix-port=18001 --applabel --applabel-rules=/usr/local/etc/yafApplabelRules.conf --max-payload=300
To run yaf
with BPF on UDP Port 53
yaf --live pcap --in en1 --out /path/to/dst/ --rotate 120 --filter="udp port 53"
To run yaf
with Deep Packet Inspection enabled for HTTP, IMAP, and DNS:
yaf --in packets.pcap --out flows.yaf --applabel --max-payload=400 --plugin-name=/usr/lib/dpacketplugin.la --plugin-opts="80 143 53"
YAF BPF Filtering does not operate with the Bivio Zero-Copy Library.
YAF BPF Filtering is ignored when using --live dag or napatech because libpcap is not used.
YAF 2.x requires libfixbuf 1.0.0 or later.
YAF 2.0 will not interoperate with the SiLK tools unless --silk is used, due to the TCP Information Elements being nested in the subTemplateMultiList. YAF 2.0 must be used with an IPFIX Collecting Process that can handle IPFIX lists elements, especially the subTemplateMultiList Information Element in order to retrieve certain flow information. Older versions of YAF can read YAF 2.0 flow files, but will ignore anything contained in the subTemplateMultiList.
The plugin infrastructure has been modified in YAF 2.0 to export templates in YAF's subTemplateMultiList element.
YAF 2.0 will export statistics in an Options Template and Options Data Records unless the --no-stats switch is given. The IPFIX Collecting Process should be able to differentiate between options records and flow records in order to prevent incorrect transcoding of statistics records into flow records.
YAF 0.7 does not interoperate with previous versions, because it no longer uses provisional information elements for the reverse direction of a biflow. YAF 0.7 must be used with an IPFIX Collecting Process that uses PEN 29305 for reverse information elements. For export to SiLK, this implies that the SiLK packer or rwipfix2silk utility must be built against libfixbuf 0.7.0 or later.
Presently, the destinationTransportPort information element contains ICMP type and code information for ICMP or ICMP6 flows; this is nonstandard and may not be interoperable with other IPFIX implementations.
Bug reports and feature requests may be sent via email to <netsa-help@cert.org>.
Brian Trammell, Chris Inacio <inacio@cert.org>, Michael Duggan <mwd@cert.org>, Emily Sarneso <ecoff@cert.org>, Dan Ruef <druef@cert.org>, and the CERT Network Situational Awareness Group Engineering Team, <http://www.cert.org/netsa>.
yafscii(1), tcpdump(1), pcap(3), nafalize(1), Spread Documentation at www.spread.org, libp0f at <https://tools.netsa.cert.org/confluence/display/tt/libp0f>, and the following IETF Internet RFCs: Specification of the IPFIX Protocol for the Exchange of IP Traffic Flow Information RFC 5101, Information Model for IP Flow Information Export RFC 5102, Bidirectional Flow Export using IPFIX RFC 5103, Export of Structured Data in IPFIX RFC 6313