Azure Data Engineer

Location: 2 days on site per week
Contract Type: 6=18 months

We are seeking an experienced Azure Data Engineer to join our dynamic team. As part of our data engineering division, you will work with cutting-edge tools and technologies to support data ingestion, integration, and transformation within the Azure ecosystem. The ideal candidate will have a strong background in data engineering, specifically with Azure Databricks, PySpark, and SparkSQL, as well as experience in testing and Azure DevOps.

Key Responsibilities:

• Data Ingestion & Transformation: Utilize PySpark and SparkSQL in Azure Databricks to ingest, process, and transform large datasets efficiently.
• Testing & Validation: Implement automated testing using Pytest to ensure the quality and integrity of the data pipelines.
• Data Integration: Work with Azure SQL Database to develop and optimize T-SQL queries and stored procedures for data integration and reporting.
• CI/CD Pipeline Management: Leverage Azure DevOps pipelines for automation of deployment, testing, and monitoring of data engineering workflows.
• Collaboration: Work closely with data analysts, data scientists, and other engineering teams to understand data needs and deliver data solutions.
• Optimization & Maintenance: Continuously improve data processes, ensuring high performance and reliability in the data ingestion and transformation workflows.

Primary Skills:

• Data Ingestion & Processing: Expertise in PySpark and SparkSQL for efficient data processing and transformation within Azure Databricks.
• Automated Testing: Hands-on experience with Pytest for testing data pipelines and ensuring their reliability and performance.

Secondary Skills:

• Azure SQL Database: Experience in developing T-SQL queries and stored procedures for data processing, integration, and reporting.
• DevOps Automation: Proficient in Azure DevOps pipelines to automate workflows, testing, and deployment of data solutions.

Qualifications:

• Bachelor's degree in Computer Science, Engineering, or a related field (or equivalent work experience).
• Proven experience in Azure Databricks, PySpark, SparkSQL, and automated testing.
• Familiarity with Azure SQL Database, T-SQL, and stored procedures.
• Solid understanding of Azure DevOps and CI/CD pipelines.
• Strong problem-solving skills and ability to work collaboratively in an agile environment.

People Source Consulting Ltd is acting as an Employment Business in relation to this vacancy. People Source specialise in technology recruitment across niche markets including Information Technology, Digital TV, Digital Marketing, Project and Programme Management, SAP, Digital and Consumer Electronics, Air Traffic Management, Management Consultancy, Business Intelligence, Manufacturing, Telecoms, Public Sector, Healthcare, Finance and Oil & Gas.