Gowtham M S

Gowtham M S

Azure Data Engineer

About Me

An Azure Data Engineer with 3 years of expertise in designing and implementing ETL pipelines using SQL, Azure Data Factory, Databricks, and Data Lake. Proficient in building end-to-end Azure pipelines with Azure DevOps and GitHub, recognized for delivering high-quality outputs and making significant contributions.

Technical Expertise

Key Projects

Real-time Order Transaction Processing Data Pipeline

Developed a real-time data ingestion and processing pipeline leveraging Confluent Kafka Cluster, Microsoft Fabric Eventstream, Python and PowerBI. This solution processes over 100,000 records per minute, providing real-time insights that improved operational efficiency by 40%.

Project Highlights:

  • Implemented a scalable solution processing 100,000+ records per minute
  • Achieved 40% improvement in operational efficiency
  • Built end-to-end real-time data processing ecosystem
Confluent Kafka Microsoft Fabric Python PowerBI Azure Event Hubs
Data Pipeline Architecture PowerBI Dashboard
GitHub Repository

Azure Databricks Data Processing Solution

Led the design and implementation of a scalable data lake solution using Azure Data Lake Storage Gen2 and Azure Databricks with support to process many types of data and utilized the factory pattern solution to generate insights of the data.

Project Highlights:

  • Created a flexible data processing framework using factory pattern
  • Implemented multi-format data ingestion (structured, semi-structured, unstructured)
  • Developed robust data governance and quality control mechanisms
Azure Databricks Azure Data Lake Storage Gen2 PySpark Delta Lake Azure Synapse
Data Processing Workflow
GitHub Repository

Professional Certifications

Microsoft Certified: Azure Data Engineer Associate (DP 203)

Demonstrate understanding of common data engineering tasks to implement and manage data engineering workloads on Microsoft Azure, using a number of Azure services.

View Credential

Microsoft Certified: Fabric Analytics Engineer Associate (DP 600)

Demonstrate foundational knowledge of microsoft fabric and Learned about implementing and managing common data engineering tasks in Fabric SAAS.

View Credential

Microsoft Certified: Azure Data Fundamentals (DP 900)

Demonstrate foundational knowledge of core data concepts related to Microsoft Azure data services.

View Credential

Microsoft Certified: Azure Data Fundamentals (AZ 900)

Demonstrate foundational knowledge of cloud concepts, core Azure services, plus Azure management and governance features and tools.

View Credential

Microsoft Certified: Azure AI Fundamentals (AI 900)

Demonstrate fundamental AI concepts related to the development of software and services of Microsoft Azure to create AI solutions.

View Credential

Connect With Me

GitHub Profile
LinkedIn Profile

Contact

↑