IOC Feed (MISP Threat Lists)

The IOC component watches incoming logs and marks documents whose content matches the Malware Information Sharing Platform (MISP) threat lists delivered with the Energy Security Feed. Marked events can be routed to alert rules or used to track incident behaviour.

How it works

1. A shell script (misp_threat_lists.sh) downloads the eight blacklist files over HTTPS with Basic authentication.

2. An ELS Network Node pipeline (blacklists.conf) reads the downloaded files and stores each entry in the .blacklists index (used as an intermediate store).

3. The same pipeline re-queries .blacklists daily and writes YAML dictionary files (misp_<type>.yml) consumed by:

   • Logstash translate filters in integration pipelines, which add a per-event field (for example [source][badip] => "true") when an indicator matches,

   • the Blacklist-IOC alert rule type, which reads the YAML files directly (see Alert Rule Types).

4. At the end of the run, after all eight categories have been fetched, the script issues a single _delete_by_query that removes entries in .blacklists older than five minutes and tagged misp_blacklist, so the intermediate index does not grow indefinitely.

5. The shell script uses curl over HTTPS with default TLS certificate validation. Only HTTP 200 responses trigger the rename of the downloaded file to the timestamped form (misp_<type>-<epoch>.blacklist) that the ELS Network Node pipeline picks up — any other response (401, 403, timeout) skips the rename, so the pipeline does not ingest new data for that category.

 repository.energylogserver.pl/ioc/
              │  misp_threat_lists.sh (HTTPS + Basic auth)
              ▼
 /etc/logserver-probe/lists/misp_<type>.blacklist
              │  blacklists.conf ELS Network Node pipeline
              ▼
        .blacklists index (intermediate store)
              │  daily export
              ▼
 /etc/logserver-probe/lists/misp_<type>.yml ──┬─► Blacklist-IOC alert rules
              │                               │
              │  translate filter             │
              ▼                               │
    Logs tagged "misp_threat_detected"        │

Feed categories

Eight category files are fetched from https://repository.energylogserver.pl/ioc/misp_<type>.blacklist:

Category file	Indicator type
misp_domain.blacklist	Domain names
misp_email.blacklist	Email addresses
misp_filename.blacklist	File names
misp_ip.blacklist	IP addresses
misp_url.blacklist	URLs
misp_filehash.blacklist	File hashes
misp_certhash.blacklist	Certificate hashes
misp_regkey.blacklist	Windows registry keys

Feed file format

Each .blacklist file is tab-separated. The blacklists.conf ELS Network Node pipeline renames the five columns as follows:

Column	Renamed to
1	object
2	type
3	source
4	description
5	url

Note

This is a custom Energy Logserver TSV format. It is not the standard MISP feed format (which supports MISP JSON, CSV or freetext per misp-project.org/feeds).

Generated dictionary format

The pipeline writes one line per indicator into /etc/logserver-probe/lists/misp_<type>.yml:

"<indicator value>": "bad_<type>"

The value is a literal presence label (bad_domain, bad_ip, bad_url, …) taken from the pipeline tag list, not a threat description. The source, description and url columns from the .blacklist file are stored in the .blacklists index but are not written into the YAML dictionary.

Note

Shipped translate integrations (Barracuda, Ruckus) therefore use the dictionary as a presence check and attach their own badip = "true" field via add_field. Custom integrations can read the dictionary value directly if a simple category label is sufficient; for richer threat context (source, description, url), query the .blacklists index instead.

Installation paths

On the ELS Network Node host the IOC components live at the following paths:

Path	Purpose
/etc/logserver-probe/lists/bin/misp_threat_lists.sh	Download script
/etc/logserver-probe/lists/	Downloaded .blacklist files and generated .yml dictionaries
/etc/logserver-probe/conf.d/blacklists/blacklists.conf	ELS Network Node pipeline
/etc/logserver-probe/templates.d/blacklists_template.json	Index template for .blacklists

The index template declares four properties: object (keyword), status (keyword), name (keyword) and description (text). The current pipeline stores five fields per entry — object, type, source, description, url — so type, source and url are mapped dynamically by the data engine, and status and name from the template are reserved but not populated by the pipeline.

Configure feed credentials

Edit /etc/logserver-probe/lists/bin/misp_threat_lists.sh and fill in the variables at the top of the file. A valid SIEM-PLAN license is required; IOC repository credentials are provided with it.

Variable	Purpose	Shipped default
DEST	Directory for downloaded blacklist files	/etc/logserver-probe/lists
IOC_USER	Basic-auth user for repository.energylogserver.pl	empty
IOC_PASS	Basic-auth password for repository.energylogserver.pl	empty
LOGSERVER_USER	Local account used to prune .blacklists	commented out, default logserver
LOGSERVER_PASS	Password for LOGSERVER_USER	commented out, default logserver
LOGSERVER_HOST	Data node address	127.0.0.1
LOGSERVER_PORT	Data node port	9200
LOGSERVER_SSL	Use TLS when querying the data node	false

In addition, the pipeline output step uses ${PROBE_USER} / ${PROBE_PASS} passed from the ELS Network Node environment when writing to .blacklists.

Scheduling

The blacklists.conf pipeline carries its own schedule through the Logstash exec and logserver input plugins — no external cron is required when the pipeline is enabled on the ELS Network Node host. Cron expressions use the format m h * * *.

Cron	Local time	Action
1 2 * * *	02:01	Run misp_threat_lists.sh to refresh the eight .blacklist files
1 3 * * *	03:01	Query .blacklists and write unsorted misp_<type>.tmp files; in parallel fetch .blacklists-exclude → misp_exclude.tmp and .blacklists-include → misp_include.tmp
1 4 * * *	04:01	Sort and deduplicate each .tmp file into the final misp_<type>.yml used by translate filters and Blacklist-IOC rules
5 4 * * *	04:05	Remove entries listed in .blacklists-exclude from misp_domain.yml (grep -Fvxf)
10 4 * * *	04:10	Append entries from .blacklists-include to misp_domain.yml

If the script must run from a host that does not run the pipeline, schedule it externally. See Installation — Blacklist and Threat Intelligence Setup for the crontab example and the ELS Console Scheduler configuration.

Custom exclude and include lists

The pipeline reads two auxiliary indices once a day and applies them to misp_domain.yml only:

Index	Field read by pipeline	Effect
.blacklists-exclude	excluded_domains	Domains listed here are removed from the translate dictionary — use this to eliminate false positives
.blacklists-include	included_domains	Domains listed here are appended to the translate dictionary — use this to add local indicators

Write documents to these indices through the data API. Changes take effect at the next apply step (04:05 for excludes, 04:10 for includes).

Using IOC in alert rules

The Blacklist-IOC rule type (see Alert Rule Types) reads the generated YAML dictionaries directly. The shipped Watchguard alert rules illustrate the expected configuration:

blacklist-ioc:
  - "!yaml /etc/logserver-probe/lists/misp_ip.yml"
compare_key: src.ip
realert:
  minutes: 60

• blacklist-ioc: — list of YAML dictionary files prefixed with !yaml

• compare_key: — event field compared against the dictionary keys

• realert: — minimum interval between repeated alerts for the same match

Point compare_key at the field you want to check (for example src.ip, dst.ip, dst.domain) and blacklist-ioc at the matching misp_<type>.yml file.

Using IOC in ELS Network Node pipelines

The generated YAML files in /etc/logserver-probe/lists/ can be consumed by any ELS Network Node pipeline through the Logstash translate filter. Two shipped integrations use this pattern: Barracuda CloudGen Firewall (barracuda/conf.d/barracuda/096-filter-misp.conf) and Ruckus (ruckus/conf.d/ruckus/090-filter-misp.conf). The Barracuda filter is shown below:

filter {
  if "barracuda" in [tags] {
    if [source][locality] == "public" {
      if [source][ip] {
        translate {
          field => "[source][ip]"
          dictionary_path => "/etc/logserver-probe/lists/misp_ip.yml"
          add_field => {
            "[source][badip]" => "true"
          }
          refresh_behaviour => "replace"
          override => true
        }
      }
    }
    if ![source][badip] {
      mutate {
        add_field => { "[source][badip]" => "false" }
      }
    }

    # (symmetric block for [destination][ip] omitted for brevity)

    if [source][badip] == "true" or [destination][badip] == "true" {
      mutate {
        add_tag => [ "misp_threat_detected" ]
      }
    }
  }
}

Other integrations can adopt the same approach by pointing dictionary_path at the relevant misp_<type>.yml file.

Note

The Barracuda example above uses the field => parameter, which is deprecated in Logstash translate 3.3.0+ (use source => in new pipelines). Existing shipped integrations mix both forms.

Verification

Confirm the feed is flowing after installation:

# Index exists and holds documents
curl -sS -u $CREDENTIAL -XGET '127.0.0.1:9200/_cat/indices/.blacklists?v'

# YAML dictionaries were generated
ls -l /etc/logserver-probe/lists/misp_*.yml

Troubleshooting

Symptom	What to check
.blacklists index missing or empty	Credentials IOC_USER/IOC_PASS in misp_threat_lists.sh, SIEM-PLAN license status
YAML dictionaries not updating	The blacklists pipeline is running on the ELS Network Node host; check pipeline status
False positive for a domain	Add the domain to .blacklists-exclude (field excluded_domains) — applied at 04:05
Missing internal indicator	Add the domain to .blacklists-include (field included_domains) — applied at 04:10
HTTP 401/403 from the feed URL	Verify IOC_USER/IOC_PASS values
.blacklists index not refreshing for a category	misp_threat_lists.sh only renames the download to the timestamped file on HTTP 200; other codes skip the rename silently and the ELS Network Node pipeline sees nothing new. Run bash /etc/logserver-probe/lists/bin/misp_threat_lists.sh manually — curl prints each HTTP status to stdout

Further reading

The feed uses MISP category naming (domain, email, url, ip, filehash, filename, certhash, regkey). For background on what a MISP feed is and how public MISP feeds are distributed, see the MISP project public feeds page.