StreamSets Developer/Engineer

Job Title: StreamSets Developer/Engineer

Location: Chennai/Bangalore/Hyderabad (Hybrid)

Duration: Full time

Notice period: Immediate joiner only/Currently serving notice with less than 45 days/Notice is less than 30 days

Relevant : 3+ Years

StreamSets Developer/Engineer:

➢ Designing, developing, and deploying data pipelines using StreamSets

➢ Configuring data sources, transformations, and destinations within pipelines.

➢ Monitoring and troubleshooting pipeline performance and errors.

➢ Collaborating with data architects and stakeholders to understand data requirements and ensure pipeline efficiency.

➢ Implementing best practices for data governance, security, and compliance.

➢ Basic understanding of Apache Kafka architecture, including topics, partitions, brokers, and consumer groups.

Responsibilities:

Pipeline Development:

➢ Designing and developing data pipelines using StreamSets based on business requirements.

➢ Implementing data transformations, validations, and enrichments within pipelines.

➢ Testing pipelines for performance, reliability, and data quality. Pipeline Monitoring and Maintenance:

➢ Monitoring pipeline performance metrics such as throughput, latency, and error rates.

➢ Troubleshooting and resolving issues related to pipeline failures or data inconsistencies.

➢ Optimizing pipelines for efficiency and scalability as data volumes grow. Security and Compliance:

➢ Ensuring that data handling practices within StreamSets comply with organizational security policies and regulatory requirements.

➢ Implementing encryption, access controls, and auditing mechanisms to protect sensitive data. Documentation and Knowledge Sharing:

➢ Documenting pipeline configurations, dependencies, and operational procedures.

➢ Sharing knowledge and best practices with team members through training sessions, documentation, and collaboration tools.

Performance Optimization:

➢ Analyzing pipeline performance bottlenecks and optimizing configurations for improved efficiency.

➢ Scaling pipelines to handle increasing data volumes and processing requirements. Collaboration and Communication:

➢ Collaborating with cross-functional teams including developers, data scientists, and business analysts to understand data requirements and ensure alignment with business objectives.

➢ Communicating updates, issues, and resolutions effectively with stakeholders.