JD – Big Data Engineer
Experience: 5+ years
Work Mode:Remote
Work Timings: 11:00 PM – 7:00 AM EST
Job Summary:
We are seeking a mid-level Big Data Engineer / Support Engineer to provide operational support for Market Regulation's big data processing and hybrid infrastructure. The role focuses on monitoring and maintaining high-volume batch and data pipelines in AWS, troubleshooting database and scheduling issues, automating monitoring and remediation, and maintaining enterprise Linux and Windows systems across on-premises and cloud environments. The ideal candidate combines systems administration, DevOps practices, containerization, cloud operations (AWS/Azure), and scripting to ensure reliability, availability, and serviceability for critical regulatory systems. This position requires working the night shift (11:00 PM – 7:00 AM EST) with occasional weekend support.
Required Skills & Experience:
- A minimum of 5 years of related experience as a big data engineer.
- Production experience as a systems administrator or support engineer across Linux and Windows environments.
- Hands-on experience with AWS services (EC2, ECS, Lambda, CloudWatch, IAM) and exposure to Azure (VMs, Container Instances, Monitor, AD).
- Strong SQL skills with experience in Postgres, Oracle, or equivalent relational databases; ability to build and run complex queries.
- Proficient scripting skills in Python and Bash; PowerShell experience desirable.
- Experience with workload automation/schedulers such as JAMS, Broadcom AutoSys, or equivalent.
- Familiarity with containerization (Docker, ECS or Kubernetes) and CI/CD tools (Jenkins, GitLab); strong understanding of git branching and deployment pipelines.
- Demonstrated troubleshooting and debugging skills for OS, applications, and infrastructure, including patching, deployment, and server hardware maintenance.
- Experience with monitoring, observability, and performance tuning across servers, containers, and cloud resources.
- Knowledge of change management, incident and problem management, metrics reporting, and SOX/DR/BCP support.
- Strong written and verbal communication and stakeholder collaboration skills
Good to Have:
- Experience with AI-enhanced observability and automation tools.
- Experience with infrastructure-as-code and enterprise automated deployment pipelines.
- Certifications: AWS SysOps, AWS DevOps, or relevant Linux/DB certifications.
- Prior experience supporting financial services, market regulation, or other regulated environments.
- Familiarity with monitoring platforms (CloudWatch, Prometheus, Datadog, Grafana) and log analysis tools.
Key Responsibilities:
- Monitor and support big data operations and batch processing in AWS for Market Regulation; detect, remediate, and escalate processing failures.
- Run and troubleshoot SQL/Postgres/Oracle queries to validate data processing status and integrity.
- Automate data monitoring, alerting, and remediation tasks using Python, Bash, and PowerShell scripts.
- Implement and manage job schedules using JAMS, Broadcom AutoSys, or equivalent tools; resolve scheduling conflicts and failed jobs.
- Maintain hybrid infrastructure across enterprise Linux and Windows systems in on-premises and cloud environments.
- Operate cloud resources in AWS and Azure, manage containers, and support server and application deployments.
- Support CI/CD pipelines (Jenkins, GitLab), identify and resolve deployment failures, and follow git best practices.
- Implement and maintain monitoring and observability for systems, containers, and cloud resources; analyze logs and performance metrics to resolve bottlenecks.
- Document runbooks, test plans, and automation scripts; participate in DR/BCP and SOX testing and reporting.
- Lead or co-lead process improvement initiatives focused on automation, containerization, and AI adoption; collaborate with DevOps, Security, and Application teams.
Education Qualification:
- Bachelor's degree in Computer Science, Information Systems, or related field required, or equivalent experience.
- Certifications in AWS, Linux administration, or workload automation are a plus.
