Data Engineer

Data Engineer, Data Pipelines

Recorded Future harvests data from many public web sources and structured data feeds. Data rich web sources such as public forums, social media sites and digital marketplaces provide cornerstone information for current cyber intelligence. As an engineer, you will join a team of motivated people working on building reliable data pipelines that ingest raw data and convert it into actionable analytics to identify and combat cyber security threats.

What you’ll do as a Data Engineer:

• You will develop, productionalize and maintain high volume data extraction via custom harvesters, APIs, and data stores.
• Create well monitored production data pipelines which convert raw data into well organized and actionable information.
• Maintain and upgrade existing data flow production systems.
• Collaborate with senior engineers through design reviews, code reviews, pair programming, and other software engineering best practice collaboration.

What you’ll bring as a Data Engineer:

• 1+ years of Python experience: You are comfortable writing in Python and have used it in personal projects or previous professional experiences. Your debugging skills are strong.
• 1+ years of working with large datasets: You are adept at working with large, complex data structures. You have experience transforming large datasets in Python and querying with modern datastores such as MongoDB and RedShift.
• Desire to learn: Coding is a passion of yours and you are excited to learn new skills and practices. You are willing to gain an understanding of how your work impacts clients and the larger intelligence community.