N
Python , Pyspark
N
Python , Pyspark
N Consulting GlobalThis job is no longer accepting applications
Job Description
- Design and implement scalable data processing pipelines using PySpark for batch and real-time data processing.
- Develop Python scripts and libraries to integrate, cleanse, and process data from various data sources (databases, APIs, etc.).
- Optimize PySpark jobs for performance and resource efficiency.
- Collaborate with data engineers and analysts to define data transformation and aggregation requirements.
- Analyze and troubleshoot issues in large-scale distributed data systems.
- Develop custom functions for data manipulation and aggregation using Spark DataFrame API and RDD API .
- Perform data analysis and create insightful visualizations for internal stakeholders.
- Write unit tests, document code, and maintain code quality through peer reviews and best practices.
- Monitor and optimize the performance of Apache Spark clusters.
- Work with cloud platforms such as AWS , Azure , or GCP to deploy Spark jobs and other data processing services.
- Skills and Qualifications:Strong experience with Python programming language and libraries such as Pandas , NumPy , Pytest , etc.
- Hands-on experience with Apache Spark and PySpark for distributed data processing.
- Familiarity with data formats such as CSV , JSON , Parquet , Avro , and other big data file formats.
- Solid understanding of Hadoop ecosystem components (HDFS, YARN, etc.).
- Proficiency with data wrangling, transformation, and aggregation.
- Experience with performance tuning, resource management, and debugging Spark applications.
- Understanding of cloud-based data storage solutions (S3, GCS, etc.) and distributed file systems.
- Familiarity with data warehousing concepts and tools.
- Good knowledge of SQL and relational databases (e.g., PostgreSQL, MySQL).
- Strong problem-solving skills and ability to work with large datasets.
- Excellent communication and collaboration skills, with the ability to work effectively in a team environment.
- Preferred Qualifications:Bachelor s or Master s degree in Computer Science, Engineering, or a related field.
- Experience with additional big data frameworks like Flink , Kafka , Hive , etc.
- Knowledge of Docker and containerization for deployment.
- Experience with ETL tools like Airflow or Luigi .
- Familiarity with machine learning concepts and frameworks (e.g., MLlib , Scikit-learn ).
More Info
Job Type:
Industry:
Function:
Employment Type:
About Company
Skills:
Golang, Agile Development Methodologies, PostgreSQL, Kafka, API design, Regex, Microservices, Nosql, Open Source, Software Automation, Design Patterns, MySQL, Python, Version Control Systems, Database Management, Linux Troubleshooting, Data Structures, Algorithms, Restful Services, Containers, cloud databases, K8S, parsing, software design principles, AI coding tools, Generative AI, Agentic AI frameworks, workflow orchestration, LLM-based application development
RE : Lead Oracle Analytics, Python & Access Reporting Modernisation Architect - IndiaCareer Path Solutions Private Limited
C
Skills:
Oracle Analytics Server (OAS), Oracle Database, Bi Publisher, Python, Microsoft Access/VBA
Early Applicant
Skills:
Python Developer, Backend Developer, Python, Flask, Object Oriented Programming, Python Engineer, Software Enginee, Flask Framework, Object-Oriented Programming
Skills:
mendix , Github, CSS, Oracle Sql, PostgreSQL, HTML, Git, Javascript, Flask, Agile, FastAPI, Rest Apis, Python, Ai, CI CD
Skills:
Pyspark, Scala, AWS Glue, Kafka, Sql, Mapreduce, Tensorflow, Jenkins, Git, Hadoop Ecosystem, Hive, Pytorch, Unix Shell Scripting, Spark, Python, Spark Structured Streaming, Scikit-Learn, Machine Learning frameworks, Delta Lake, Data Governance and Data Lineage tools, Spark ML