Who We Are
Artmac Soft is a technology consulting and service-oriented IT company dedicated to providing innovative technology solutions and services to Customers.
Job Description
Job Title : Junior Data Engineer
Job Type : Full-Time
Experience: 1 - 2 years
Location : Hyderabad, Telangana
Responsibilities
- Design and develop data ingestion pipelines to load CSV files, business flat files, and SAP Fusion data into Google BigQuery for analytics and reporting.
- Implement real-time data ingestion pipelines using Google Cloud Dataflow and Pub/Sub, and support streaming architectures using Delta Live Tables and Structured Streaming.
- Develop and manage batch ETL pipelines using Cloud Dataflow, Dataproc, and Apache Spark, ensuring scalable and reliable data processing.
- Build low-code ETL workflows using Google Cloud Data Fusion and integrate data using DataFactory and connector-based ingestion frameworks.
- Perform Spark-based data processing using Cloud Dataproc and Databricks Runtime (Photon) to handle large-scale data transformations.
- Design and maintain data storage and ingestion layers using BigQuery and Delta Lake for high-performance analytics workloads.
- Write and optimize complex SQL queries, views, and stored procedures to support transformation and reporting needs.
- Perform data cleansing, transformation, and validation using SQL and Spark to ensure data quality and integrity.
- Manage and maintain databases using DBeaver and SSMS, ensuring smooth database operations.
- Optimize BigQuery query performance, partitioning, and clustering for efficient data analytics.
- Troubleshoot and resolve data ingestion issues, ETL pipeline failures, and database performance problems.
- Ensure data accuracy, integrity, and consistency across ingestion and transformation pipelines.
- Work closely with analytics, reporting, and business teams to deliver reliable datasets and support analytical use cases.
- Support data modeling and schema design for efficient storage and query performance within Google BigQuery and Delta Lake environments.
Qualification
- Bachelor's degree or equivalent combination of education and experience
