At Datadog’s scale, with over 18,000 customers sending trillions of data points per day, analyzing the volume of data coming in can be challenging. One of the largest log sources internally at Datadog are networking logs. Being able to analyze and make sense of them is critical to keep Datadog secure. To help with the task, we have built a flow analysis pipeline that alerts against network level Indicators of Compromise (IOCs) like IP address, port combinations, and data exchanged.

In this session, Andrew Krug, Technical Evangelist, will chat with Théo Guidoux, Software Engineer in the Threat-Detection Platform team and Anna Pauxberger, Software Engineer in the Security Platform team. They will discuss how they built the pipeline leveraging Datadog and Apache Spark and how they optimized the process by taking advantage of new AWS features and the Datadog Cloud Security Platform.

By the end of the session you will understand better how Network Traffic Flows are a critical tool to detect security threats and how you can start collecting, optimizing, and analyzing them at your organization.