TECHNICAL ANALYST

• Design and develop data ingestion pipelines using Apache NiFi for structured, semi-structured, and unstructured data sources.

• Build and maintain real-time and batch processing workflows to ingest data into AWS-based data lakes or warehouses.

• Configure and optimize NiFi processors, templates, and flows for scalability and reliability.

• Implement monitoring, alerting, and recovery strategies for NiFi flows and downstream processes.

• Integrate NiFi with AWS services such as S3, Lambda, Redshift, Glue, Kinesis, and RDS.

• Ensure data quality, integrity, and compliance across pipelines. • Work closely with data analysts, architects, and business teams to define data requirements and delivery models.

• Document workflows, metadata, and lineage using tools like Apache Atlas or custom catalogs.

Required Skills & Qualifications

• 6-7 years of experience in data engineering or data integration roles.

• Strong hands-on experience with Apache NiFi including custom processor development and flow orchestration.

• Solid experience with AWS services, especially: o S3, Lambda, Glue, Kinesis, Redshift, RDS

• Strong understanding of data ingestion patterns (CDC, batch, streaming).

• Proficiency in SQL, Python, or Java for data transformations or scripting.

• Experience in handling large datasets and performance optimization in data workflows.

Good to Have

• Exposure to Apache Kafka, Airflow, or Step Functions.

• Experience with data lake architectures and tools like Delta Lake, Lake Formation, or Athena.

• Knowledge of data governance and cataloging tools (AWS Glue Data Catalog, Apache Atlas).

• Experience working in Agile teams with tools like Jira, Confluence.