Role description
Tech Lead – Azure/Snowflake & AWS Migration
Key Responsibilities
- Design and develop scalable data pipelines using Snowflake as the primary data platform, integrating with tools like Azure Data Factory, Synapse Analytics, and AWS services.
- Build robust, efficient SQL and Python-based data transformations for cleansing, enrichment, and integration of large-scale datasets.
- Lead migration initiatives from AWS-based data platforms to a Snowflake-centered architecture, including:
o Rebuilding AWS Glue pipelines in Azure Data Factory or using Snowflake-native ELT approaches.
o Migrating EMR Spark jobs to Snowflake SQL or Python-based pipelines.
o Migrating Redshift workloads to Snowflake with schema conversion and performance optimization.
o Transitioning S3-based data lakes (Hudi, Hive) to Snowflake external tables via ADLS Gen2 or Azure Blob Storage.
o Redirecting Kinesis/MSK streaming data to Azure Event Hubs, followed by ingestion into Snowflake using Streams & Tasks or Snowpipe.
- Support database migrations from AWS RDS (Aurora PostgreSQL, MySQL, Oracle) to Snowflake, focusing on schema translation, compatibility handling, and data movement at scale.
- Design modern Snowflake lakehouse-style architectures that incorporate raw, staging, and curated zones, with support for time travel, cloning, zero-copy restore, and data sharing.
- Integrate Azure Functions or Logic Apps with Snowflake for orchestration and trigger-based automation.
- Implement security best practices, including Azure Key Vault integration and Snowflake role-based access control, data masking, and network policies.
- Optimize Snowflake performance and costs using clustering, multi-cluster warehouses, materialized views, and result caching.
- Support CI/CD processes for Snowflake pipelines using Git, Azure DevOps or GitHub Actions, and SQL code versioning.
- Maintain well-documented data engineering workflows, architecture diagrams, and technical documentation to support collaboration and long-term platform maintainability.
Required Qualifications
- 9+ years of data engineering experience, with 3+ years on Microsoft Azure stack and hands-on Snowflake expertise.
- Proficiency in:
o Python for scripting and ETL orchestration
o SQL for complex data transformation and performance tuning in Snowflake
o Azure Data Factory and Synapse Analytics (SQL Pools)
- Experience in migrating workloads from AWS to Azure/Snowflake, including services such as Glue, EMR, Redshift, Lambda, Kinesis, S3, and MSK.
- Strong understanding of cloud architecture and hybrid data environments across AWS and Azure.
- Hands-on experience with database migration, schema conversion, and tuning in PostgreSQL, MySQL, and Oracle RDS.
- Familiarity with Azure Event Hubs, Logic Apps, and Key Vault.
- Working knowledge of CI/CD, version control (Git), and DevOps principles applied to data engineering workloads.
Preferred Qualifications
- Extensive experience with Snowflake Streams, Tasks, Snowpipe, external tables, and data sharing.
- Exposure to MSK-to-Event Hubs migration and streaming data integration into Snowflake.
- Familiarity with Terraform or ARM templates for Infrastructure-as-Code (IaC) in Azure environments.
- Certification such as SnowPro Core, Azure Data Engineer Associate, or equivalent.
Skills
Azure,AWS REDSHIFT,Athena,Azure Data Lake