AWS Data Engineer-Unity Catalog

KPI Partners is seeking a skilled Data Engineer with expertise in Databricks, Unity Catalogue, Pyspark, Python, and SQL. As a member of our dynamic team, you will be responsible for designing, developing, and maintaining scalable data pipelines and solutions that support our data-driven decision-making processes.

Key Responsibilities:

• Develop and maintain data pipelines using Databricks and Pyspark to ensure efficient data processing and transformation.
• Utilize Unity Catalogue to manage and govern data assets and ensure data quality and security.
• Collaborate with data analysts, data scientists, and other stakeholders to understand data requirements and deliver actionable insights.
• Write efficient and optimized queries using SQL to extract and manipulate data from various sources.
• Monitor and troubleshoot data pipeline performance, implement improvements, and provide support for data-related issues.
• Document data engineering processes, architecture, and workflows to ensure knowledge sharing within the team.

Qualifications:

• Bachelor’s degree in computer science, Data Science, or a related field.
• Proven experience as a Data Engineer or in a similar role with a strong understanding of data engineering principles
• Proficiency in Databricks, Unity Catalogue, Pyspark, Python, and SQL.
• Experience with data modelling, ETL processes, and data warehousing concepts.
• Strong problem-solving skills and the ability to work collaboratively in a team environment.
• Excellent communication skills to convey technical concepts to non-technical stakeholders.

…