Roles an responsibilities:
- Application Development: Design, develop, and maintain Python applications and scripts tailored to meet business requirements.
- Code Quality: Write clean, efficient, and well-documented code. Adhere to coding standards and best practices.
Data Processing and Analysis:
- Data Manipulation: Use Pandas and NumPy for data cleaning, transformation, and analysis.
- ETL Development: Develop and manage ETL (Extract, Transform, Load) processes using PySpark to handle and process large datasets.
Big Data Management:
- PySpark Development: Design and implement PySpark jobs for large-scale data processing. Optimize Spark applications for performance and resource efficiency.
- Performance Tuning: Identify and resolve performance bottlenecks in PySpark jobs and SQL queries.
Database Management:
- SQL Queries: Write and optimize SQL queries for data retrieval, manipulation, and reporting. Ensure data integrity and efficiency in database operations.
- Database Design: Assist in the design and maintenance of database schemas to support application and reporting needs.
Testing and Quality Assurance:
- Unit Testing: Develop and execute unit tests to ensure code functionality and reliability.
- Debugging: Identify, troubleshoot, and resolve issues or bugs in code and data processes.
Documentation and Reporting:
- Technical Documentation: Document code, processes, and workflows for clarity and future reference.
- Data Reporting: Generate reports and visualizations to present data insights to stakeholders effectively.
Collaboration and Communication:
- Cross-functional Teamwork: Collaborate with data scientists, analysts, and other developers to integrate and utilize data solutions effectively.
- Stakeholder Communication: Communicate technical concepts and project status to both technical and non-technical stakeholders.
Continuous Improvement:
- Learning and Development: Stay current with industry trends and emerging technologies to continuously improve technical skills and development practices.
- Process Improvement: Identify opportunities for process improvements and implement changes to enhance efficiency and effectiveness.
