Senior Data Engineer

Company: Tredence

About Us: Tredence differentiates itself from other analytics services firms by enabling last mile adoption of insights. We drive true business impact by uniting our strengths in business analytics, data science, and software engineering. We help our clients deliver on the artificial intelligence imperative in a practical and demystified way.

To know more about us, visit: www.tredence.com

Job Overview:

A Data Engineer specializing in Kafka, SQL, Python, and DBT Core is responsible for building and maintaining the infrastructure that handles both real-time and batch data processing. This role bridges the gap between raw data sources and actionable business insights by designing scalable pipelines and robust transformation layers.

Core Responsibilities

Streaming Pipeline Development (Kafka): Design and manage Kafka-based data pipelines to handle high-throughput, real-time data streams. This includes configuring Kafka brokers, producers, and consumers, and integrating them with downstream storage like data lakes or warehouses.

Modular Data Transformation (dbt Core): Implement and maintain data models and schemas using dbt to turn raw data into analysis-ready datasets. This involves applying software engineering practices like version control, automated testing, and documentation to the SQL transformation layer.

Complex Data Modeling (SQL): Write advanced, optimized SQL queries for data manipulation, cleaning, and aggregation. Responsibilities include designing efficient schemas (e.g., star schemas or medallion architecture) and performing query tuning to ensure high performance.

Develop robust Python scripts and classes for data ingestion, parallel processing, and error handling. Python is typically used for complex logic that SQL cannot handle alone, such as interacting with APIs or building custom orchestrators.

Continuously monitor and fine-tune big data systems for maximum scalability and reliability, reducing processing times and costs.

Success in this role means delivering high-quality, maintainable data solutions that drive actionable insights and business value.

You will collaborate closely with data scientists, analysts, and software engineers to ensure seamless data flow and integrity.

This position plays a crucial role in enabling AI-driven decision-making and last-mile adoption of analytics within our client engagements.

What will your role look like

Design and implement robust data models and architecture within cloud envirment to support analytics and reporting requirements.

Develop, monitor, and optimize ETL/ELT pipelines ensuring data quality and integrity using Python, Snowpipe, SnowSQL, and Snowpark.

Collaborate with cross-functional teams to understand data requirements and translate business needs into technical solutions.

Troubleshoot and resolve data-related issues, ensuring high availability and performance of data infrastructure.

Implement data quality frameworks and best practices to maintain accuracy and consistency across datasets.

Maintain documentation of data workflows, models, and architecture standards for knowledge sharing and operational continuity.

You will need:

Hold a BE/BTech degree in Computer Science, Information Technology, or a related field.

Expert proficiency in Python (specifically libraries like pandas or PySpark) and SQL (PostgreSQL, Snowflake, or BigQuery).

Deep knowledge of Apache Kafka, including Kafka Connect, Schema Registry, and stream processing tools like Kafka Streams or KSQL.

Hands-on experience with dbt Core, including mastery of macros, packages, and materialization strategies.

Familiarity with workflow tools such as Apache Airflow or Prefect to schedule and monitor complex pipelines.

Experience managing data on cloud environments such as AWS, Azure, or Google Cloud Platform.

Solid understanding of Git for version control, CI/CD pipelines, and writing unit tests for data logic

Demonstrate 59 years of strong experience in data engineering, specifically working with cloud platforms and its architecture components

Exhibit expertise in data modelling, data quality management, and SQL scripting.

Write efficient, reusable, and scalable Python code to support data transformation and pipeline automation.

Communicate effectively with technical and non-technical stakeholders to drive alignment on data solutions and project deliverables.

Good to Have Skills:

Gain familiarity with cloud platforms and their integration with Snowflake (e.g., AWS, Azure, or GCP).

Develop knowledge of advanced analytics or machine learning workflows that leverage engineered data sets.

Acquire certifications related to Snowflake or data engineering such as SnowPro Core Certification or equivalent.

Cultivate experience working in agile teams and using collaboration tools like JIRA or Confluence.

Why you will love this job:

Engage in impactful projects within the dynamic retail sector, directly contributing to business transformations powered by AI and analytics.

Benefit from a hybrid work model that offers flexibility and work-life balance.

Grow your career with continuous learning opportunities, including mentorship and access to cutting-edge technology and tools.

Join an inclusive, collaborative environment where your expertise is valued and your ideas can drive real-world change.

Location: Bangalore, Chennai, Pune, Gurgaon, Kolkata, Hyderabad

Years of experince-59 years