262COS Custom Elastic Data Pipelines
tip
On a a DIP configured IAW 262COS-DIP-SOP-001, Logstash is able to listen on TCP/6000 - 6009 and UDP/7000 - 7009 for custom inputs, such as custom logs ingested by Filebeat - but proper ingestion requires that your Data Engineer create custom Logstash Input, Filter, and Output configurations.
This chart illustrates how host data is collected from a source and processed through the Elastic stack to populate an index in Elasticsearch:
| Data Location | Data Source | Data Shipper | Logstash Input | Elasticsearch Index | Index Aliases | |||
|---|---|---|---|---|---|---|---|---|
| Remote Host | Windows Event Logs | ⇨ | Winlogbeat | ⇨ | logstash:5045 | ⇨ | cvah-winlogbeat* | winlogbeat-cvah* |
| Remote Host | Log Files | ⇨ | Filebeat | ⇨ | logstash:5045 | ⇨ | cvah-filebeat* | filebeat-external-cvah* |
| Remote Host | System Auditing | ⇨ | Auditbeat | ⇨ | logstash:5045 | ⇨ | cvah-auditbeat* | auditbeat-external-cvah* |
| Remote Host | System Auditing | ⇨ | Syslog TCP | ⇨ | tcp:\\logstash:5047 | ⇨ | cvah-syslog | syslog-cvah* |
| Remote Host | System Auditing | ⇨ | Syslog UDP | ⇨ | udp:\\logstash:514 | ⇨ | cvah-syslog | syslog-cvah* |
| Remote Host | Endgame Agent | ⇨ | Endgame SMP | ⇨ | http:\\logstash:9800 | ⇨ | cvah-endgame.stream* | endgame-cvah* |
| Remote Host | Endgame Agent | ⇨ | Endgame SMP | ⇨ | syslogtcp:\\logstash:9801 | ⇨ | cvah-endgame.forwarded | endgame-cvah* |
| CVAH (Metasponse) | Metasponse Jobs | ⇨ | Metasponse | ⇨ | N/A (Straight to Elasticsearch) | ⇨ | ms-* | |
| Custom Locations | Custom Sources | ⇨ | Custom TCP Inputs | ⇨ | tcp:\\logstash:6000-6009 | ⇨ | Custom Indices | |
| Custom Locations | Custom Sources | ⇨ | Custom UDP Inputs | ⇨ | udp:\\logstash:7000-7009 | ⇨ | Custom Indices |