Big Data Engineer

Primary Skills: AWS, Bigdata, Spark, Scala, Hive

JOB RESPONSIBILITY

Design and implement predictive analytics models to identify data anomalies in Claims, Eligibility, and other critical datasets in Big Data Platform.

Collaborate with cross-functional teams to understand data sources and Data Quality objectives.

Develop and optimize machine learning algorithms to improve data accuracy and operational efficiency.

Deploy machine learning models and ensure their scalability and robustness in a production environment.

Continuously monitor and refine models to maintain data quality and performance.

Document the development process and produce comprehensive reports on analytics findings.

QUALIFICATION

B.Tech /B.E /MCA

EXPERIENCE

6+ year's experience in AWS Services: RDS, AWS Lambda, AWS Glue, Apache Spark, Kafka, Spark streaming, spark, Scala , Hive and AWS etc.

6+ year's experience SQL and NoSQL databases like MySQL, Postgres, Elasticsearch

6+ year's experience with Spark programming paradigms (batch and stream-processing)

6+ year's experience in Java, Scala. Familiarity with a scripting language like Python as well as Unix/Linux shells

6+ year's experience with Strong analytical skills and advanced SQL knowledge, indexing, query optimization techniques.

SKILLS AND COMPETENCIES

Strong proficiency in programming languages such as Python or R, and Strong SQL skills.

Experience with data processing tools and ML frameworks (e.g., TensorFlow, PyTorch, scikit-learn).

Experience in building predictive analytics solutions using time-series analysis, regression analysis, classification, and clustering.

Piping and processing massive data-streams in distributed Cloud Computing environments such as AWS EMR Spark,

Experience implementing end-to-end data science project

Excellent analytical, problem-solving, and communication skills.