Warm welcome from SP Staffing Services! We're excited to tell you about a permanent opportunity for a PySpark Developer to join our team.
- Experience: We're seeking professionals with 4 to 6 years of experience.
- Location: This role is open in Chennai, Hyderabad, Bangalore.
What You'll Do
- Implement data ingestion pipelines from various sources, including databases, S3, and files.
- Build ETL/Data Warehouse transformation processes.
- Develop Big Data and non-Big Data cloud-based enterprise solutions using PySpark and SparkSQL, along with related frameworks and libraries.
- Create scalable, reusable, and self-service frameworks for data ingestion and processing.
- Integrate end-to-end data pipelines to move data from source to target repositories, ensuring data quality and consistency.
- Conduct processing performance analysis and optimization.
- Implement best practices in design & analysis, automation (Pipelining, IaC), testing, monitoring, and documentation.
- Work with both structured and unstructured data.
Good to Have
- Experience with cloud-based solutions.
- Knowledge of data management principles.
