The Wikimedia Foundation is seeking an experienced engineering leader to serve as Director of Data Engineering for our Technology department. At the Wikimedia Foundation, we operate the world’s largest collaborative project: a top ten website, reaching a billion people globally every month, while incorporating the values of privacy, transparency and community that are so important to our users.
Reporting to the VP of Data Science & Engineering, the Director of Data Engineering is a key member of the Foundation’s data team and an active participant in the strategic decision making framing the work of the technology department, the Wikimedia Foundation and the Wikimedia movement. Working closely with other Technology and Product teams, as well as our community of contributors and readers, you will help shape the next generation of data usage, analysis and access across all Wikimedia projects.
This role is responsible for planning and executing on key data engineering initiatives spanning our work in artificial intelligence, machine learning, search, natural language processing and product analytics. As Director, you will lead a team of data engineers, site reliability engineers and software engineers in building and maintaining our entire analytics and events infrastructure, including key systems like Kafka, Cassandra, Druid and our Hadoop/Spark cluster. We build our stacks using fully open source components, we release our data publicly and our puppet repo is public.
Here are some examples of projects we have tackled that you may be excited to lead:
- Releasing public data to the Wikimedia community and the world at large. Our public data offerings are used all over the world by companies and research institutions big and small. A popular example is the Wikipedia Clickstream (a.k.a. Wikipedia’s rabbit hole)
- Deploying an anomaly detection system to monitor Wikipedia accessibility over the world and detect possible outages (or censorship events)
- Integrating an open-source GPU software platform like AMD ROCm in Hadoop and in the Tensorflow-related ecosystem
- Building the Foundation’s new event data platform infrastructure
- Migrate our on-prem Hadoop infrastructure that holds petabytes of data to Hadoop3
- Evangelize privacy conscious ways to compute metrics. Privacy is key to the work we do
We’d like you to do these things:
- Partner closely with other teams and departments across the Wikimedia Foundation to define and experiment with machine learning products. These could be brand new feature offerings in Wikipedia or augmentation of existing workflows.
- Review and advise in code changes and technical decisions made by the team
- Represent team members within the organization and Wikimedia community
- Support and coach your team members in the development of their career paths
- Recruit and hire new team members
- Work closely with our Research, Architecture, Security, Site Reliability and Platform teams to define our next generation of data architecture
- Ensure data is available, reliable, consistent, accessible, secure, and available in a timely manner for external and internal stakeholders and in accordance with our privacy policy
- Contribute to our culture by managing, coaching and developing team members
Skills and Experience:
- Deep experience in leading data science, machine learning, search or data engineering teams that is able to separate the hype in the artificial intelligence space from the reality of delivering production-ready data systems
- 5+ years engineering leadership experience
- Demonstrated ability to balance competing interests in a complex technical and social environment
- Proven success at all stages of the engineering process and product lifecycle, leading to significant, measurable impact
- Previous hands-on experience in production big data and machine learning environments at scale.
- You are no stranger to error budgets, operations at scale and efforts to reduce toil.
- Experience building and leading diverse, international and remote-first teams
Qualities that are important to us:
- You take a solutions-focused approach to challenging data and technical problems
- A passion for people development, team culture and the management of ideas
- You have a desire to show the world how data can be done while honoring the user’s right to privacy
Additionally, we’d love it if you have:
- Experience with modern machine learning, search and natural language processing platforms
- A track record of open source participation
- Fluency or familiarity with languages in addition to English
- Spent time having lived or worked outside your country of origin
- Experience as a member of a volunteer community
The Wikimedia Foundation is the nonprofit organization that hosts and operates Wikipedia and the other Wikimedia free knowledge projects. Our vision is a world in which every single human can freely share in the sum of all knowledge. We believe that everyone has the potential to contribute something to our shared knowledge, and that everyone should be able to access that knowledge, free of interference.
As an equal opportunity employer, the Wikimedia Foundation values having a diverse workforce and continuously strives to maintain an inclusive and equitable workplace. We encourage people with a diverse range of backgrounds to apply. We do not discriminate against any person based upon their race, religion, color, national origin, sex, pregnancy or related medical conditions, parental status, sexual orientation, gender identity, gender expression, age, status as a protected veteran, status as an individual with a disability, genetic information, or any other legally protected characteristics.
If you are a qualified applicant requiring assistance or an accommodation to complete any step of the application process due to a disability, you may contact us at or (415) 839-6885.
#J-18808-Ljbffr