Data Engineer

Mandatory Skills: Hands-on experience in Azure Databricks & Medallion Architecture. 7-9 years of experience in Azure Databricks technologies. Experience in Medallion Architecture is mandatory.
• Good understanding of ETL, BI, and DW technologies.
• Hands-on experience with JSON or XML files as sources.
• Experience in autoloader for incremental loading and explode options for flattening data.
• Experience in loading Delta Live Tables.
• Experience in PySpark or Spark using data frames.
• Design, implement, and maintain data pipelines for data ingestion, processing, and transformation in Azure.
• Work together with data scientists and analysts to understand data needs and create effective data workflows.
• Create and maintain data storage solutions including Azure SQL Database, Azure Data Lake, and Azure Blob Storage.
• Utilize Azure Databricks to create and maintain ETL operations.
• Implement data validation and cleansing procedures to ensure data quality, integrity, and dependability.
• Improve scalability, efficiency, and cost-effectiveness of data pipelines.
• Monitor and resolve data pipeline problems to guarantee consistency and availability.
• Good communication skills – Oral & Written.