Data Engineer
Job Description
- Design and develop data-ingestion frameworks, real-time processing solutions, and data processing and transformation frameworks
- Deploy and provide support for application codes and analytical models
- Create and enhance data solutions that enable seamless integration and flow of data across the data ecosystem
- Provide business analysis and develop ETL code and scripting to meet all technical specifications and business requirements according to the established designs
- Develop real-time data ingestion and stream-analytic solutions leveraging technologies such as Kafka, Apache Spark (SQL, Scala, Java), Python and Hadoop Platform and any Cloud Data Platform
- Utilize multiple development languages/tools such as Python, SPARK, Hive, Presto, Java to build prototypes and evaluate results for effectiveness and feasibility
- Develop application systems that comply with the standard system development methodology and concepts for design, programming, backup, and recovery to deliver solutions that have superior performance and integrity
- Offer system support as part of a support rotation with other team members
- Operationalize open-source data-analytic tools for enterprise use
- Ensure data governance policies are followed by implementing or validating data lineage, quality checks, and data classification
- Understand and follow the company development lifecycle to develop, deploy and deliver the solutions
- Designs and develops data ingestion frameworks leveraging open-source tools and data processing frameworks
- Designs and develops complex and critical data projects
- Develops, implements, and supports application codes and analytical models
- Creates and enhances data solutions that enable seamless integration and flow of data across the data ecosystem
- Designs and develops real time processing solutions
More Info
Job Type:
Industry:
Function:
Employment Type:
Key Skills
About Company
Skills:
Pyspark, AWS Glue, Data Modeling, Emr, Avro, Autosys, Sql, Cloudwatch, Terraform, Etl Tools, Databricks, Python, AWS, Airflow, Parquet, AWS Lake Formation, Iceberg
Skills:
data engineering , Data Governance, Data integration techniques and tools, Data Analysis Interpretation, Data Architecture Principles, Cloud-based data platforms, Distributed computing environments
Skills:
data engineering , AWS Glue, Data Modeling, Sql, Etl, Step Functions, Airflow, IAM roles and permissions
Skills:
Scala, Python, Spark
Skills:
Pyspark, Spark, Azure Databricks, Sql, Azure DevOps, Delta Lake, Event Hub