Key Responsibilities:

📍Design, develop, and maintain data pipelines that ingest, process, and transform data from various real-time sources (IoT devices, sensors, etc.) into Databricks delta lake. 📍Integrate and manage real-time streaming data from Azure IoT Hub into the data platform using Azure Stream Analytics or other real-time data processing services. 📍Build and manage real-time data pipelines for use cases requiring immediate analytics and processing. 📍Write complex and optimized Kusto Query Language (KQL) scripts for querying and analysing data in ADX. 📍Leverage Azure Databricks for data transformation, machine learning, and advanced analytics workflows. 📍Collaborate with cross-functional teams, including data scientists, software engineers, and product managers, to deliver data-driven solutions. 📍Ensure data quality, reliability, and consistency by implementing appropriate validation and monitoring tools. 📍Implement and manage Microsoft Fabric solutions to create unified and scalable data environments for efficient data exploration and analysis. 📍Optimize data storage, retrieval, and processing, ensuring that all pipelines are efficient, scalable, and cost-effective. 📍Work with Azure DevOps for version control, CI/CD pipeline setup, and automation for deployment of data solutions.

Required Skills and Qualifications:

📍Bachelor’s degree in computer science, Data Engineering. 📍6+ years of experience as a Data Engineer working with cloud-based data platforms. 📍Strong experience with Python, Azure Data Explorer (ADX) and proficiency in Kusto Query Language (KQL). 📍Experience in setting up and managing Azure IoT Hub for real-time streaming data ingestion. 📍Solid understanding of real-time streaming data architectures and tools like Azure Stream Analytics, Event Hubs, or Apache Kafka. 📍Hands-on experience with Azure Databricks for building scalable data transformation pipelines and advanced analytics workflows. 📍Familiarity with Microsoft Fabric for data integration, pipeline management, and optimization and real-time dashboards.