Job Description

Role Name: Azure Fabric - Data Engineer / Architect

Location: Newark, NJ 07102 (Onsite)

Contract

Job Description

• Deep expertise in modern data architecture, with specific experience in Microsoft's data platform and Delta Lake architecture.
• 6+ years of experience in data architecture and engineering.
• Required 2+ years hands-on experience with Azure Databricks / ADF and Spark.
• Required recent experience with Microsoft Fabric platform.

Key Responsibilities:

• Data Architecture:
• Design end-to-end data architecture leveraging Microsoft Fabric's capabilities.
• Design data flows within the Microsoft Fabric environment.
• Implement OneLake storage strategies.
• Configure Synapse Analytics workspaces.
• Establish Power BI integration patterns.
• Integration Design:
• Architect data integration patterns with analytics using Azure Data Factory and Microsoft Fabric.
• Implement medallion architecture (Bronze/Silver/Gold layers).
• Ability to configure real-time data ingestion patterns.
• Establish data quality frameworks.
• Lakehouse Architecture:
• Implement modern data lakehouse architecture using Delta Lake, ensuring data reliability and performance.

Data Governance:

• Establish data governance frameworks incorporating Microsoft Purview for data quality, lineage, and compliance.
• Microsoft Fabric Expertise:
• Data Integration: Combining and cleansing data from various sources.
• Data Pipeline Management: Creating, orchestrating, and troubleshooting data pipelines.
• Analytics Reporting: Building and delivering detailed reports and dashboards to derive meaningful insights from large datasets.
• Data Visualization Techniques: Representing data graphically in impactful and informative ways.
• Optimization and Security: Optimizing queries, improving performance, and securing data
• Azure Databricks Experience:
• Apache Spark Proficiency: Utilizing Spark for large-scale data processing and analytics.
• Data Engineering: Building and managing data pipelines, including ETL (Extract, Transform, Load) processes.
• Delta Lake: Implementing Delta Lake for data versioning, ACID transactions, and schema enforcement.
• Cluster Management: Configuring and managing Databricks clusters for optimized performance. (Ex: autoscaling and automatic termination)
• Integration with Azure Services: Integrating Databricks with other Azure services like Azure Data Lake, Azure SQL Database, and Azure Synapse Analytics.
• Data Governance: Implementing data governance practices using Unity Catalog and Microsoft Purview
• Security Framework:
• Design and implement security patterns aligned with federal and state requirements for sensitive data handling.
• Implement row-level security.
• Configure Microsoft Purview policies.
• Establish data masking for sensitive information.
• Design audit logging mechanisms.
• Pipeline Development:
• Design scalable data pipelines using Azure Databricks for ETL/ELT processes and real-time data integration.
• Performance Optimization:
• Implement performance tuning strategies for large-scale data processing and analytics workloads.
• Optimize Spark configurations.
• Implement partitioning strategies.
• Design caching mechanisms.
• Establish monitoring frameworks.

Regards

Rahul Bansiwal

Sr. Talent Acquisition Specialist

(2014792186) | Office: (201) 479 2186 EXT: 444

rahulb@net2source.com

270 Davidson Ave, Suite 704, Somerset, NJ 08873, USA

Knowledge is Power.

Job Tags

Similar Jobs