Mandatory Skills
Databricks & Python
Job Description
Core Technical Skills
- Python - Advanced level (pandas, requests, boto3 libraries)
- API Integration - REST APIs, authentication (OAuth, API keys), error handling
- Cloud Platforms - AWS (S3, EC2, IAM) proficiency
- Databricks/Spark - PySpark, Delta Lake, SQL optimization
- Data Pipeline Tools - Airflow, Databricks Workflows, or equivalent orchestration
- SQL - Complex queries, window functions, performance tuning
Security & Enterprise Knowledge
- Understanding of security telemetry (logs, alerts, events, metrics)
- Familiarity with at least 2-3 security platforms (JAMF, Intune, Qualys, CrowdStrike,
Okta)
- Experience with AWS CloudTrail or similar cloud audit logs
- Knowledge of data security and compliance (encryption, access controls)
Additional Competencies
- Version control (Git)
- CI/CD pipeline familiarity
- Data quality and validation frameworks
- Performance tuning and optimization mindset
Education Qualificaiton
BE, BTech, MCA
Roles & Responsibilities
Data Integration & Connectivity
- Design and build connectors between enterprise security platforms (JAMF, Intune, Qualys,
CrowdStrike, Okta, AWS CloudTrail) and Databricks
- Establish secure API connections and authentication mechanisms with each source system
- Implement data extraction logic using REST APIs, SDKs, and native connectors
- Handle varying data formats and transformation requirements across heterogeneous sources
Data Pipeline Development
- Develop automated Python scripts for continuous data ingestion
- Build orchestration workflows using Apache Airflow, Databricks Workflows, or similar tools
- Implement error handling, retry mechanisms, and monitoring for pipeline reliability
- Ensure near real-time or scheduled data synchronization based on requirements
AWS S3 & Data Staging
- Configure S3 bucket architecture and data partitioning strategy
- Land raw security telemetry data in S3 with proper folder structures
- Implement lifecycle policies and data retention strategies
- Ensure proper encryption and access controls
Databricks Platform Configuration
- Create and optimize Databricks tables
