AWS Data Bricks Spark/Scala Engineer

Job Title: AWS Data Bricks Spark/Scala Engineer

Job ID: 2023-11933

Job Location: Morris Plains, NJ (remote to start); Remote, Memphis, TN (preferred), Morris Plains, NJ, Austin, TX (preferred), St. Louis, MO

Employment Type: W2

Duration: Long Term

Work Eligibility: All Work Authorizations are Permitted

Key Technology: DataBricks, AWS, Spark

Job Responsibilities:

• Drive the mentality of building well-architected applications for AWS Cloud.
• Promote quality ownership across the entire team.
• Identify code defects and collaborate with developers to address quality issues in product code.
• Find bottlenecks and thresholds in existing code using automation tools.
• Articulate clear business objectives aligned with technical specifications and work in an iterative agile pattern daily.
• Take ownership of tasks and engage with all team levels, raising challenges when necessary.

Skills and Experience Required:

• 6-8 years of strong experience in Spark, Python, Shell Scripting, PostgreSQL, Hadoop, AWS (S3, EC2, SNS, SQS, Lambda, ECS, Glue, IAM, and CloudWatch), Databricks (Delta lake, Notebooks, Pipelines, cluster management, Azure/AWS integration).
• In-depth knowledge of Hadoop architecture and its components, with the ability to troubleshoot Hadoop Yarn jobs. Experience in implementing Hadoop job orchestration using Shell scripting and Airflow.
• In-depth understanding of Spark Architecture, including Spark Core, Spark SQL, Data Frames, Spark Streaming, RDD caching, and Spark MLib.
• Strong experience in writing complex and effective SQLs and Stored Procedures, with query optimization skills to improve ETL performance.
• Hands-on cloud experience with the AWS suite.
• Demonstrated work experience in Hadoop, Spark, and Hive.
• Strong development experience on Databricks components.
• Significant hands-on experience with Spark.
• Experience with Hadoop as a data warehouse/Data Lake implementation.
• Moderate to senior Terraform development experience.
• Health care domain knowledge is a plus.