I
Spark
I
Spark
Infosys3-5 Years
- Posted 12 hours ago
- Be among the first 10 applicants
Job Description
- Primary skills:Technology->Big Data - Data Processing->Spark
- Design, develop, and maintain scalable data processing jobs using Spark for batch and/or near-real-time workloads.
- Analyze large datasets to identify trends, anomalies, and data quality issues; implement validation and reconciliation checks.
- Optimize Spark applications for performance by tuning partitions, caching strategies, memory usage, and execution plans.
- Collaborate with cross-functional teams to translate business requirements into technical solutions and well-defined deliverables.
- Implement robust error handling, logging, and monitoring to ensure reliability and easier troubleshooting.
- Participate in code reviews, follow engineering best practices, and contribute to reusable components and standards.
- Support deployments and production issues by performing root-cause analysis and implementing preventive fixes. Minimum Qualifications:
- Bachelor's degree (or equivalent) in Engineering/Technology/Computer Science or related field (BTech/BE/MSc or equivalent).
- 3–5 years of experience working on data engineering or big data processing initiatives.
- Hands-on experience building and maintaining Spark-based data processing solutions.
- Strong understanding of distributed computing concepts and data processing fundamentals.
- Ability to work independently on assigned modules and collaborate effectively within a team. Preferred Qualifications:
- Master's degree (MTech/MCA or equivalent) in a relevant discipline.
- Proven experience delivering end-to-end Spark pipelines, including development, testing, and production support.
- Experience improving job performance and stability through Spark tuning and structured troubleshooting practices.
- Familiarity with building reusable frameworks/components to standardize Spark development across projects.
- Strong communication skills to explain technical trade-offs and align solutions with stakeholder expectations. Good to have skills: Hadoop, Hive, Kafka, Airflow, Delta Lake
More Info
About Company
Skills:
Apache Airflow, Git, Scala, Spark, Databricks, Sql, Etl, Delta Lake
Skills:
Cloud Composer (Airflow), BigQuery, Scala, Apache Spark, Dataproc, Sql, Cloud Storage, Terraform, Apache Beam, DataFlow, Python, Pub Sub, GCP AI Platform, Cloud SQL, Vertex AI, Spanner
Skills:
Spark SQL, Agile Methodologies, Pyspark, Performance Tuning, Apache Spark, Data Warehousing, Azure Databricks, Sql, Query Optimization, Spark Core, Git, Hive, Hadoop Ecosystem, Python, AWS EMR, RDDs, HDFS, DataFrames
Skills:
Java, Test Automation, Graphql, Kafka, Spring Boot, Microservice Architecture, Sql, Nosql, Rest Api, Docker, Spark, Helm, Kubernetes, Micronaut, Observability