Job Responsibilities

• Design, develop, and optimize complex ETL/ELT pipelines using tools like Apache NiFi, Apache Kafka, and Apache Airflow for data ingestion, transformation, and integration.
• Implement and maintain data architecture to ensure high availability and reliability of data systems.
• Work with MySQL or other RDBMS technologies to manage structured data, and optimize database performance for high-volume environme
• Partner with data scientists, business analysts, and product teams to understand data requirements and translate them into efficient data engineering solutions.
• Develop and maintain data systems to support AI/ML model training, validation, and deployment, ensuring optimal data quality and consistency.
• Assist in the development of data governance processes to ensure data quality, security, and integrity.
• Utilize Agile methodologies to collaborate on project timelines, sprints, and deliverables.
• Mentor junior engineers on best practices for data pipeline development, data architecture, and ETL processes.
• Provide technical expertise in data modeling, data integration, and performance tuning of large-scale data systems
• Troubleshoot and resolve issues related to data flows, ensuring the delivery of timely and accurate data.
• Work on real-time data streaming solutions to handle large data volumes with low latency.
• Stay current with emerging data technologies and trends to drive innovation in data engineering practices.

Required Skills & Qualifications

• 5+ years of experience in data engineering, with a focus on building scalable ETL pipelines and data integration solutions.
• Strong expertise in Apache NiFi, Apache Kafka, and Apache Airflow or other orchestration tools.
• Proficiency in MySQL or other relational databases like PostgreSQL, Oracle, or SQL Server.
• Strong understanding of AI/ML model pipelines and experience in working with data for training and deploying machine learning models.
• Solid understanding of data architecture and data modeling techniques.
• Hands-on experience in data integration, transformation, and working with large datasets.
• Familiarity with data governance practices, including data quality, security, and privacy.
• Agile experience in a fast-paced, iterative development environment.
• Expert knowledge in GIT Hub, Jenkins, GIT Hub Actions
• Strong problem-solving skills with attention to detail and an ability to work autonomously.
• Mentoring experience or a willingness to guide junior team members.
• Effective communication skills to collaborate with cross-functional teams and present technical information to non-technical stakeholders.

Keywords

​Apache NiFi, Apache Kafka, Apache Airflow, ETL Orchestration, ETL Pipelines, MySQL, Data Architecture, Data Modeling, Data Integration, Data Transformation, Agile Methodologies, Scrum, Real-Time Data Streaming, CI/CD Pipelines, Data Infrastructure, Scalability, Machine Learning Pipelines, DataOps, GIT, Jenkins, GIT Hub actions