Responsibilities:
Leadership & Strategy: Lead and mentor a team of data engineers, driving best practices in data engineering and ensuring alignment with the overall data strategy.
Architecture & Development: Architect and develop highly scalable data pipelines and infrastructure using Apache Kafka, Apache Spark or Databricks, Python, Scala, and other relevant technologies.
Streaming & Real-Time Processing: Design and implement streaming data solutions using technologies such as Spark Structured Streaming, Kafka Streams, and others to process millions of events per day.
Data Governance & Quality: Ensure the integrity, quality, and consistency of data across our Data Lake and Data Warehouse, implementing rigorous data governance practices.
Cloud & NoSQL Management: Oversee the use of AWS managed services and manage NoSQL databases such as ElasticSearch, Redis, MongoDB, or ScyllaDB.
Operational Excellence: Continuously monitor and improve processes and best practices for managing data pipelines, ETL operations, CI/CD deployment and data infrastructure.
Cross-Functional Collaboration: Collaborate with stakeholders across the organization to understand data needs and deliver solutions that drive business value.
Requirements:
Expert Engineering Skills: A strong background in software engineering with deep expertise in abstraction, coding, and building distributed systems.
Advanced Programming Proficiency: Exceptional programming skills in Python or Scala, with a strong understanding of multi-threading and object-oriented programming.
Data Systems Expertise: Extensive experience with Spark, Kafka, real-time streaming systems, AWS, and Cloud Data Warehouses like Starburst Trino. Familiarity with containerization technologies such as Docker and Kubernetes.
Proven Leadership: Demonstrated ability to lead and develop high-performing data engineering teams, particularly within startup or high-growth environments.
Collaborative Team Player: Strong interpersonal skills, with the ability to collaborate effectively across teams and communicate complex technical concepts to non-technical stakeholders.
Visionary & Strategic Thinker: Capable of seeing the big picture, with the ability to align technical initiatives with broader business goals.