Key Responsibilities
Infrastructure Design and Management:
- Design and provision scalable, secure, and cost-efficient infrastructure onAWSusingTerraform(mandatory).
- Deploy and manage services such asDatabricks (on AWS),Amazon Redshift,EMR, andRDSusing Infrastructure-as-Code.
- Maintain version-controlled, reusable infrastructure modules in alignment with IaC best practices.
DevOps and CI/CD Automation:
- Develop and maintainCI/CD pipelinesusingJenkins(mandatory) integrated with Git-based workflows.
- Automate infrastructure provisioning, testing, and deployment processes.
- Support CI/CD for data orchestration tools such asAirflow,Databricks, andRedshift.
Data Pipeline Platform Management:
- Collaborate with data engineers to build and deploy pipelines usingAWS Glue,Kinesis,Lambda,S3, andRedshift.
- Implement orchestration usingApache Airflow(open-source and AWS MWAA).
- Ensure reliability, scalability, and security throughout the data lifecycle.
Monitoring, Logging Optimization:
- Set up monitoring, logging, and alerting tools usingCloudWatch,Prometheus,Grafana, andCloudTrail.
- Perform root-cause analysis, troubleshoot issues, and optimize infrastructure and pipeline performance.
Security and Compliance:
- Apply AWS security best practices across environments, includingIAM,VPC,KMS, encryption, and network segmentation.
- Conduct audits and implement governance policies.
Documentation Collaboration:
- Work cross-functionally with DevOps, data engineering, and cloud security teams.
- Document architecture, configurations, workflows, and reusables to drive team efficiency and knowledge sharing.
Required Skills Qualifications
Cloud Infrastructure-as-Code (IaC):
- Strong hands-on experience with AWS (EC2, S3, Lambda, RDS, Glue, Redshift, IAM, VPC, Kinesis).
- Expertise in Terraformfor multi-account, multi-service AWS infrastructure.
DevOps CI/CD:
- Proficiency with Jenkins(mandatory).
- Experience with GitLab CI/CD, GitHub Actions, or AWS CodePipeline is a plus.
- Familiarity with Docker andAmazon EKSis desirable.
Data Platform Experience:
- Experience provisioning and operating platforms likeDatabricks,Amazon Redshift, and optionallySnowflake.
- Experience building and managing data pipelines usingGlue,Airflow, andStep Functions.
Monitoring Troubleshooting:
- Strong understanding of observability tools and frameworks (e.g., CloudWatch, Grafana, Prometheus).
Scripting Automation:
- Proficient inPython,Bash, or similar scripting languages.
Security Governance:
- Hands-on with IAM roles/policies, VPC security,KMS encryption, and compliance controls.
Preferred Qualifications
- Certifications:AWS Solutions Architect, DevOps Engineer, or Data Analytics Specialty.
- Databricks, Redshift, or Snowflake certifications are a plus.
- Experience with other IaC tools (Ansible, CloudFormation) and workflow engines (Apache Airflow, MWAA).
- Exposure to big data platforms like EMR, Spark, or Hadoop.
