Python+Spark Scala

Key Responsibilities:

• Big Data Spark Development
• Develop and maintain data processing pipelines using Apache Spark PySpark Scala
• Work with Spark DataFrames RDDs and Spark SQL
• Implement transformations joins aggregations and optimizations
• Tune Spark jobs for performance scalability and reliability
• Python Scala Programming
• Write clean efficient and scalable code in Python and Scala
• Develop modular and reusable components
• Integrate data pipelines with various applications and APIs
• ETL Data Engineering
• Design and build ETL workflows for structured and unstructured data
• Extract data from multiple sources databases APIs flat files
• Perform data cleansing transformation and validation
• Ensure data accuracy consistency and completeness
• Data Platforms Integration
• Work with Hadoop ecosystem HDFS Hive Spark
• Handle large datasets in data lakes and warehouses
• Process data in formats like Parquet ORC JSON CSV
• Collaboration Support
• Work with data engineers analysts and business stakeholders
• Troubleshoot pipeline issues and provide production support
• Participate in Agile Scrum processes
• Maintain technical documentation

Technical Requirements:

• Primary skills Technology Big Data Data Processing Spark Technology Java Apache Technology Machine Learning Python

Additional Responsibilities:

• Core Skills
• 2 5 years of experience in Python development
• Hands on experience with Apache Spark PySpark and or Scala
• Strong understanding of data processing and ETL concepts
• Good knowledge of SQL and relational databases

Preferred Skills:

Technology->AI-Data science->PYTHON,Technology->Big Data - Data Processing->Spark->SparkSQL,Technology->Java->Apache->Scala