Job Title: Data Engineering + Data Modelling Lead
Experience: 6–8 Years
Location: Bangalore (Preferred) / Remote
Job Responsibilities:
- Lead end-to-end data engineering and data modeling initiatives, including requirements gathering, solution design, development, testing, deployment, andoperational support.
- Develop scalable, high-performance data pipelines using PySpark and Apache Spark for data ingestion, transformation, integration, and loading acrossenterprise platforms.
- Define and implement enterprise-wide conceptual, logical, and physical data models to support business, analytical, and operational requirements.
- Design and maintain database schemas, tables, views, indexes, and data structures that support both transactional and analytical workloads.
- Lead the development of ETL/ELT frameworks and reusable components to ensure efficient and standardized data processing.
- Implement data quality frameworks, validation rules, profiling techniques, and monitoring processes to ensure data accuracy, completeness, consistency, andreliability.
- Optimize data models, ETL pipelines, database performance, and Spark workloads to improve scalability, processing efficiency, and query performance.
- Document data models, data lineage, ETL workflows, metadata, and technical designs in a clear and comprehensive manner.
- Provide technical leadership and mentorship to development teams.
Required Experience & Skills:
- 6-8 years of experience in Data Engineering, ETL/ELT Development, and Data Modeling, with demonstrated experience leading enterprise-scale data initiatives and deliveryteams.
- Strong expertise in Databricks/Snowflake(or similar ETL tools), PySpark, Apache Spark, Spark SQL, and Python for designing, developing, and optimizing high-volume datapipelines and distributed data processing solutions.
- Proven experience in designing Conceptual, Logical, and Physical Data Models, including ER Modeling, Dimensional Modeling, Data Vault, and normalization/denormalizationtechniques.
- Hands-on experience with data lake, data warehouse, and cloud-based data platforms such as Databricks, Snowflake, AWS, or Azure.
- Strong knowledge of relational databases and performance tuning, including Oracle, SQL Server, PostgreSQL, Snowflake, schema design, indexing, and query optimization.
- Experience with data integration and orchestration technologies such as Apache Airflow, Kafka, NiFi, or equivalent enterprise data movement tools.
- Solid understanding of data governance, metadata management, data lineage, and data quality frameworks, with the ability to establish enterprise data standards and bestpractices.
Job Description – PySpark ETL Developer
Job Title: PySpark ETL Developer
Experience: 6+ Years
Location: Bangalore (Preferred)
Job Summary
We are seeking an experienced PySpark ETL Developer with 6+ years of experience in designing, developing, and optimizing enterprise ETL pipelines using PySpark. The ideal candidate should have strong expertise in Python, Apache Spark, Databricks, and Snowflake, along with hands-on experience in processing large-scale data and building scalable data engineering solutions. You will work closely with business stakeholders and cross-functional teams to develop high-performance data pipelines that support business intelligence and analytics initiatives.
Key Responsibilities
- Design, develop, and maintain scalable ETL pipelines using PySpark and Spark SQL.
- Collaborate with stakeholders to gather business requirements and translate them into efficient data engineering solutions.
- Extract data from multiple sources, including databases, APIs, data lakes, files, and streaming platforms.
- Transform, cleanse, and validate data using PySpark to ensure high data quality and consistency.
- Develop and optimize Spark jobs for performance, scalability, and efficient resource utilization.
- Build and maintain batch and streaming data pipelines to support real-time and near real-time processing.
- Load transformed data into data lakes, data warehouses, and analytical platforms.
- Implement robust error handling, logging, monitoring, and troubleshooting mechanisms for ETL workflows.
- Document ETL processes, data lineage, transformation logic, and technical specifications.
- Develop unit, integration, and performance tests to ensure reliable and accurate data processing.
- Collaborate with cross-functional teams to deliver scalable, secure, and high-quality data solutions.
Required Skills
- 6+ years of experience as a Data Engineer or ETL Developer.
- Strong hands-on experience with PySpark, Apache Spark, Spark SQL, and Python.
- Expertise in Databricks and/or Snowflake.
- Strong understanding of ETL design, data transformation, and data integration.
- Experience with Data Lakes, Data Warehouses, and Big Data technologies.
- Knowledge of distributed computing, parallel processing, and data partitioning concepts.
- Strong SQL skills with experience in performance tuning and query optimization.
- Experience working with batch and streaming data processing.
- Excellent analytical, debugging, and problem-solving skills.
- Strong verbal and written communication skills.
Preferred Skills
- Experience with cloud platforms such as AWS or Azure.
- Knowledge of Hadoop ecosystem and modern data engineering technologies.
- Experience with API integration and data ingestion from multiple data sources.
- Familiarity with CI/CD pipelines and Agile development methodologies.
- Exposure to enterprise-scale data engineering projects.
Why Join Us
- Opportunity to work on cutting-edge Big Data and Cloud Data Engineering projects.
- Exposure to modern technologies including PySpark, Databricks, Snowflake, and distributed computing.
- Collaborative work environment with strong learning, innovation, and career growth opportunities.
Job Title: PySpark ETL Developer
Experience: 4+ Years
Location: Bangalore
Job Responsibilities:
- Collaborate with stakeholders to gather and understand data requirements, business rules, and transformation logic. Analyze source data structures andformats to design ETL pipelines that meet business needs.
- Extract data from various sources such as databases, data lakes, APIs, files, and streaming sources using PySpark. Develop custom connectors and readers toaccess and ingest data into Spark DataFrames or RDDs.
- Transform and cleanse raw data using PySpark transformations and functions to prepare it for analysis and reporting.
- Process large volumes of data efficiently using distributed computing capabilities of Apache Spark. Optimize Spark jobs for performance, scalability, andresource utilization to meet SLAs and throughput requirements.
- Load transformed data into target systems such as data lakes, data warehouses, or analytical databases using PySpark. Implement batch and streaming dataloading strategies to support real-time and near-real-time data integration.
- Implement data quality checks and validation rules within ETL pipelines to ensure the accuracy, completeness, and consistency of data. Handle exceptions,errors, and anomalies gracefully and log diagnostic information for troubleshooting.
- Implement error handling mechanisms and logging frameworks to capture errors, warnings, and debugging information during ETL execution. Monitor jobstatus, progress, and performance metrics using logging and monitoring tools.
- Document ETL processes, data lineage, and metadata in clear and comprehensive documentation. Maintain data dictionaries, schema definitions, andtransformation logic to facilitate data governance and lineage tracking.
- Develop unit tests, integration tests, and end-to-end tests to validate ETL pipelines and ensure data accuracy and reliability. Conduct performance testing andoptimization to identify and resolve bottlenecks.
Required Experience & Skills:
- Minimum of 4 years of experience as a data engineer, ETL developer with specific expertise in designing and implementing ETL pipelines using PySpark.
- Strong expertise in Databricks/Snowflake(or similar ETL tools)
- Strong proficiency in Python programming and experience with PySpark, Spark SQL.
- Familiarity with distributed computing principles, data partitioning, and parallel processing techniques.
- Excellent analytical, problem-solving, and communication skills.
