Azure Synapse link for Dataverse - Introduction

Microsoft has recently announced that the Data Export Service (DES) will no longer be supported after 1 year (Nov-22). This may come as a surprise to many of the Clients who are using and depending on DES for their data extraction, integration and analytics. DES was one of the main tool used to extract and store data from Dataverse for analytical processing and integration purposes with other systems. 

There is a new option to get data from your Dataverse environments, store the data in a data warehouse and perform analytical and machine learning processing using that data. This is the Azure Synapse link for Dataverse.

So what is Azure Synapse? - In Microsoft speak - "Azure Synapse is an enterprise analytics service that accelerates time to insight across data warehouses and big data systems. Azure Synapse brings together the best of SQL technologies used in enterprise data warehousing, Spark technologies used for big data, Data Explorer for log and time series analytics, Pipelines for data integration and ETL/ELT, and deep integration with other Azure services such as Power BI, CosmosDB, and AzureML."

The main benefit of using Azure Synapse link for Dataverse is to get near real time insights on your Dataverse data. 

The below diagram shows the architecture of Azure Synapse. 

Azure Synapse Architecture

There are other benefits of using Azure Synapse link for Dataverse - 

1. You can utilize Synapse SQL, a distributed query system for T-SQL that enables data warehousing and data virtualization scenarios and extends T-SQL to address streaming and machine learning scenarios.

2. Enable Big Data and ML using Apache Spark for Azure Synapse - deeply and seamlessly integrates Apache Spark--the most popular open source big data engine used for data preparation, data engineering, ETL, and machine learning.

3. Use SQL and Spark together with data in your Azure Data Lake 

4. Built-in Data integration engine to ingest and create ETL pipelines at scale

5. Perform near real-time log and IOT stream analytics using Azure Data Explorer services

6. Use Azure Synapse Studio to build solutions, maintain and secure your data with a unified experience

Some of the challenges that you may need to consider -

1. Existing integration with external systems will need to be re-designed

2. Investments in non-Microsoft analytics and ML solutions may need to be evaluated against the above benefits.

3. Change management - The data engineering and analysis team will need to update their skills to be able to move to Azure Synapse.

4. There could be some cost benefits analysis that you will need to make to consider moving to Azure Synapse. Planning and budgeting for the move to Azure Synapse link for Dataverse is going to be critical for enterprise clients. 

To overcome some of the pain points and ease the transition from DES to Azure Synapse link for Dataverse, Microsoft has also provided a playbook. (link is here)  

Hope the above gets you started in planning to move from Data Export Services to Azure Synapse and do let me know about your experiences.

Thanks for reading.

@mihircrm

365WithoutCode