Job Title: Lead/Senior BigData Engineer
Location: Bedminster, NJ • Work onsite - 3 days a week is required
Type : #W2 Position
Experience:- 10 to 15+
Visa :- USC, GC Only
Keyskills - Must Have • Apache Spark, ElasticSearch, OpenShift, Hadoop, ETL, Scala, AWS ,Python Programming ,SQL Programming, ETL , Databricks
Project summary :
The project is in telecom sector and involves collecting, processing, and analyzing large volumes of network performance data from various sources, including mobile devices, CPEs, scanners, and third party testing tools. Project features with complexity of its data sources, the volume of data processed, and the need for real-time analysis.
Job Description
• Bachelor’s degree in Computer Science, Software Engineering or related field
• 10+ Years of experience working on any public cloud platforms, preferably AWS and an excellent grasp of the following:
Development of Complex Data Ingestion & Transformation Pipelines
o Spark Programming hands-on experience (Scala, Python, PySpark, SparkSQL)
o Strong Apache Spark and Elasticsearch expertise .
o Good understanding of OpenShift, specifically running ElasticSearch on OpenShift o DevOps skills, preferably ArgoCD would be a big plus o Databricks/Hive/Data lake/Delta lake
• Excellent grasp of data warehousing and ETL/ELT use cases and patterns at scale • The ability to apply such methods to solve business problems in combination with building data pipelines, data streams, and system integration.
• Support the planning and implementation of data design services, providing sizing and configuration assistance and performing needs assessments. • Translate business requirements to technical solutions leveraging strong business acumen.
• Analyze current business practices, processes, and procedures as well as identify future business opportunities for leveraging cloud services (Microsoft, Amazon, and/or Google). • Deliver architectures for transformations and modernizations of enterprise data solutions using cloud data technologies.
• Develop and maintain data warehouse schematics, layouts, architectures, and relational/non(1)relational databases for data access and Advanced Analytics. • Implement effective metrics and monitoring processes. • Demonstrated experience of turning business use cases and requirements to technical solutions. • Experience in business process mapping of data and analytics solutions. • Experience in leading and mentoring development teams.
• Ability to model business processes using industry best practices.
• Ability to conduct data profiling, cataloging, and mapping for technical design and construction of technical data flows. • Amazon AWS Experience a big plus.
Job Responsibilities
• Design and develop scalable data processing systems using Apache Spark and ElasticSearch,
• Deploy and manage ElasticSearch on OpenShift for indexing and data storage.
• Create customer libraries for high-bandwidth data processing and ETL • Develop solution for handling large data processing and transfer distributed system and storage environments • Implement and maintain ETL processes to integrate large datasets into data lakes and data warehouses. • Lead and mentor our development teams to deliver a quality data solution. • Work closely with other app developers and engineers to ensure project success.
• Exercise a high degree of autonomy and ownership • Research approaches to solve unfamiliar problems
• You must be available for meeting and team collaboration during the day starting 8:00AM EST/EDT
Best Regards
AppStream