Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[8.1](backport #30564) [Filebeat] Remove RecordedFuture dataset from Threat Intel module #30568

Merged
merged 1 commit into from
Feb 24, 2022
Merged

[8.1](backport #30564) [Filebeat] Remove RecordedFuture dataset from Threat Intel module #30568

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CHANGELOG.next.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ https:/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d

*Filebeat*

- Remove Recorded Future integration from threatintel module. {pull}30564[30564]

*Heartbeat*

Expand Down Expand Up @@ -108,7 +109,6 @@ https:/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Add support in httpjson input for oAuth2ProviderDefault of password grant_type. {pull}29087[29087]
- Add support for filtering in journald input with `unit`, `kernel`, `identifiers` and `include_matches`. {pull}29294[29294]
- Add new `userAgent` and `beatInfo` template functions for httpjson input {pull}29528[29528]
- threatintel module: Add new Recorded Future integration. {pull}30030[30030]
- Add pipeline in FB's supported hints. {pull}30212[30212]

*Auditbeat*
Expand Down
37 changes: 0 additions & 37 deletions filebeat/docs/fields.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -151909,43 +151909,6 @@ type: keyword
The indicator type, can for example be "domain, email, FileHash-SHA256".


type: keyword

--

[float]
=== recordedfuture

Fields for Recorded Future Threat Intel



*`recordedfuture.evidence_details`*::
+
--
List of sightings used as evidence for this indicator.


type: flattened

--

*`recordedfuture.name`*::
+
--
Indicator value.


type: keyword

--

*`recordedfuture.risk_string`*::
+
--
Details of risk rules observed.


type: keyword

--
Expand Down
Binary file removed filebeat/docs/images/filebeat-threatintel-recordedfuture.png
View file
Open in desktop
Binary file not shown.
131 changes: 0 additions & 131 deletions filebeat/docs/modules/threatintel.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,6 @@ The available filesets are:
* <<otx,otx>>: Supports gathering threat intel attributes from AlientVault OTX.
* <<anomali,anomali>>: Supports gathering threat intel attributes from Anomali Limo.
* <<anomalithreatstream,anomalithreatstream>>: Supports gathering threat intel attributes from Anomali ThreatStream.
* <<recordedfuture,recordedfuture>>: Supports gathering threat intel attributes from Recorded Future.
* <<threatq,threatq>>: Supports gathering threat intel attributes from ThreatQuotient.

include::../include/gs-link.asciidoc[]
Expand Down Expand Up @@ -515,124 +514,6 @@ Anomali ThreatStream fields are mapped to the following ECS fields:
[[a]]
[small]#[1]: Field is used to derive a value for the ECS field but its original value is kept under `threatintel.anomalithreatstream`.#

[[recordedfuture]]
[float]
==== `recordedfuture` fileset settings

The `recordedfuture` fileset fetches risklists from the Recorded Future Connect API.
It supports `domain`, `hash`, `ip` and `url` entities.

In order to use it you need to define the `entity` and `list` to fetch. Check the
https://api.recordedfuture.com/index.html[Recorded Future API Explorer] for the
available lists for each entity.

Sample configuration:
[source,yaml]
----
- module: threatintel
recordedfuture:
enabled: true
var.input: httpjson
var.interval: 1h
var.api_token: "<RF_TOKEN>"
var.list: default
var.entity: domain
----

To fetch threat intelligence from multiple entities and/or lists, you must define more
than one instance of the module. The following configuration fetches the default list
for domains every hour and the rfTrending list for hashes every 12 hours:

[source,yaml]
----
- module: threatintel
recordedfuture:
enabled: true
var.input: httpjson
var.interval: 1h
var.api_token: "<RF_TOKEN>"
var.list: default
var.entity: domain
- module: threatintel
recordedfuture:
enabled: true
var.input: httpjson
var.interval: 12h
var.api_token: "<RF_TOKEN>"
var.entity: hash
var.list: rfTrending
----

Alternatively, you can use the module to fetch custom Fusion files by setting
`var.custom_url` to the URL of the Fusion File:

[source,yaml]
----
- module: threatintel
recordedfuture:
enabled: true
var.input: httpjson
var.interval: 1h
var.api_token: "<RF_TOKEN>"
var.custom_url: 'https://api.recordedfuture.com/v2/fusion/files/?path=%2Fpublic%2Frisklists%2Fdefault_domain_risklist.csv'
----

It's also possible to load CSV risklists from a file:

[source,yaml]
----
- module: threatintel
recordedfuture:
enabled: true
var.input: file
var.paths:
- /path/to/risklist.csv
----

*`var.input`*::

The input to use to fetch indicators. Use `httpjson` to query
Recorded Future API or `file` to load the indicators from a file.

*`var.api_token`*::

The API token used to access Recorded Future API (RF-Token).

*`var.interval`*::

How often the API is polled for updated information. It is recommended to set this
to `1h`. For `hash` entities, it's recommended to set this to `12h`.

*`var.entity`*::

The type of entity to fetch. One of `domain`, `hash`, `ip` or `url`.

*`var.list`*::

The indicator list to fetch.

*`var.proxy_url`*::

Optional URL to use as HTTP proxy.

*`var.custom_url`*::

An alternative URL pointing to a CSV risklist. Use this option
to fetch custom Fusion Files.

Recorded Future fields are mapped to the following ECS fields:

[options="header"]
|=============================================================
| Recorded Future fields | ECS Fields
| entity.name | threat.indicator.{url,ip,domain,file.hash}
| entity.type | threat.indicator.type
| fileHashes | threat.indicator.file.hash
| risk.score | event.risk_score
|=============================================================

:has-dashboards!:

[float]
=== Dashboards

Expand Down Expand Up @@ -675,12 +556,6 @@ image::./images/filebeat-threatintel-misp.png[]
[float]
Overview of the information provided by the MISP feed.

[role="screenshot"]
image::./images/filebeat-threatintel-recordedfuture.png[]

[float]
Overview of the information provided by the Recorded Future feed.

[[threatq]]
[float]
==== `threatq` fileset settings
Expand Down Expand Up @@ -787,12 +662,6 @@ image::./images/filebeat-threatintel-misp.png[]
[float]
Overview of the information provided by the MISP feed.

[role="screenshot"]
image::./images/filebeat-threatintel-recordedfuture.png[]

[float]
Overview of the information provided by the Recorded Future feed.

[role="screenshot"]
image::./images/filebeat-threatintel-threatq.png[]

Expand Down
26 changes: 0 additions & 26 deletions x-pack/filebeat/filebeat.reference.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2225,32 +2225,6 @@ filebeat.modules:
# var.ssl_certificate: path/to/server_ssl_cert.pem
# var.ssl_key: path/to/ssl_key.pem

recordedfuture:
enabled: false

# Input used for ingesting threat intel data
var.input: httpjson

# Set your API Token.
var.api_token: "<RF_TOKEN>"

# The interval to poll the API for updates
var.interval: 1h

# The kind of entity to fetch. One of domain, hash, ip or url.
var.entity: domain

# The list to fetch. See the Recorded Future API Explorer for
# valid lists for each kind of entity.
var.list: default

# Uncomment to use a different API endpoint.
# The API endpoint used for Recorded Future API calls.
# var.endpoint: "https://api.recordedfuture.com/v2"

# Uncomment to fetch a custom CSV file via URL. Useful for custom Fusion Files.
# var.custom_url: "https://api.recordedfuture.com/v2/fusion/files/?path=%2Fhome"

threatq:
enabled: false

Expand Down
26 changes: 0 additions & 26 deletions x-pack/filebeat/module/threatintel/_meta/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,32 +138,6 @@
# var.ssl_certificate: path/to/server_ssl_cert.pem
# var.ssl_key: path/to/ssl_key.pem

recordedfuture:
enabled: false

# Input used for ingesting threat intel data
var.input: httpjson

# Set your API Token.
var.api_token: "<RF_TOKEN>"

# The interval to poll the API for updates
var.interval: 1h

# The kind of entity to fetch. One of domain, hash, ip or url.
var.entity: domain

# The list to fetch. See the Recorded Future API Explorer for
# valid lists for each kind of entity.
var.list: default

# Uncomment to use a different API endpoint.
# The API endpoint used for Recorded Future API calls.
# var.endpoint: "https://api.recordedfuture.com/v2"

# Uncomment to fetch a custom CSV file via URL. Useful for custom Fusion Files.
# var.custom_url: "https://api.recordedfuture.com/v2/fusion/files/?path=%2Fhome"

threatq:
enabled: false

Expand Down
Loading