Skip to main content

Tag: Azure ADF

New Azure DataFactory template makes Salesforce to Snowflake Pipelines fast and cost-effective

Posted on 2024-12-04 - 1:29 am by
Azure DataFactory for Snowflake and Salesforce

We built our free Azure DataFactory template to help you build your data pipelines from Salesforce to Snowflake in 5-30 minutes. The value of data is not realized by collecting and moving it. The value of data is realized when you transform it into information. Analytics insights and attributes for automation is the objective and reason why you invest in a data warehouse like Snowflake. That is why we have built our free data pipeline templates to reduce the level of effort to get your data ready for analysis up to 90%! View our documentation to lean how

In 2023, we launched the first version of our template tagging it as a “5 min data lake with Azure DataFactory”. Adoption and feedback led us to close 2024 with an upgraded version of our template alongside our new DataTools doctor service and our revised Snowflake rapid adoption service to help our customers extract value from data faster.

Download our Azure DataFactory Template Now

Name(Required)
This field is for validation purposes and should be left unchanged.

Why we build Salesforce to Snowflake pipelines with Azure DataFactory?

Fast, cheap and easy rarely happens in technology, but the Azure team, without a massive marketing blitz or fanfare created a very solid product for moving data between enterprise data sources in to Microsoft data platform and Snowflake.

We use Azure DataFactory pipelines to move massive volumes of data daily for customers who have invested in Azure. They save thousands of dollars per month while getting enterprise grade data extraction and migration.

When do we turn to other solutions than DataFactory?

We stick to technology tools that are flexible, practical, and well adopted. DataFactory in particular we like deploying with customers who are running MS SQL or Snowflake in Azure. When it comes to data transformation and ETL patterns, DataFactory does offer a no/low code Spark based Flow builder. However, depending on customer needs, scale, and team makeup, we do recommend alternative solutions that is catered to the existing processes, investments, team makeup and roadmap. We are always on the lookout for new, streamlined data pipeline solutions.

What’s Next for our Azure DataFactory template?

We are actively working on client projects for MySQL and MS SQL version of our template. Contact us for more details

Azure Data Factory Tutorials

Salesforce data pipelines to Snowflake free template

 Video Tutorial

1. Create Data Connections

 Connect Azure Data Factory to Salesforce.com (New 2024)

 Connect Azure Data Factory to Snowflake

2. Create Data Sources

 Create a Snowflake Data Source in Azure Data Factory

 Connect to Salesforce Lead and Account Object with Azure ADF

3. Move Data Between Applications

  Create a Data Pipeline to Connect Salesforce to Snowflake

4. Publishing and Administration

Publish your ADF Pipeline, Data Sets, and Triggers

Create an ADF Scheduled Trigger

What’s New in our Salesforce to Snowflake Pipelines

This week, we rolled out a long overdue update for our free Azure DataFactory template that makes extracting Salesforce data into Since 2023, there have been a lot of changes to Azure DataFactory, so we have rolled out a long overdue update and upgrade.

Salesforce to Snowflake ADF Data Lake

From our version 1.0.1 release notes:

  • New meta data staging process and table called “SFDC_METADATA_STAGE_TEMP” that feeds SFDC_METADATA_STAGE
  • Support for new field detection and addition (enable append fields)
  • New parameter AppendFields will insert new fields when detected
  • New parameter SnowErrorHandling allows for configurable error handling to skip error rows, skip file, or throw an error.
  • New MetaData field called “Status” that allows for “Disable” attribute that will ignore fields from being synchronized.
  • Update to Salesforce metadata request that supports compound fields by default like FirstName, LastName, Street, City, etc.
  • Pre-install check – End to end flow checks for existence of the metadata object
  • Added status variables to determine results for each pipeline for easier debugging
  • Schema insert and updates managed via merge by object.field ID
  • Changed field from ID to DurableId Salesforce field to Snowflake SFDC_METADATA_STAGE “ID”

Other Azure Related Articles

Read More on our Blog

Snowflake and Microsoft Expand their Data and AI Partnership

Posted on 2023-06-26 - 6:15 pm by
Microsoft and Snowflake Logos

Snowflake and Microsoft, announced a press release at Snowflake Summit 2023 that they are expanding their partnership promising substantial advancements for data scientists and developers. This enhanced collaboration is set to seamlessly merge Snowflake’s Data Cloud with Microsoft’s Azure ML, extending its capabilities through the potent combination of Azure OpenAI and Microsoft Cognitive Services.

This strategic alliance means that Snowflake and Microsoft Azure shared customers will gain access to the cutting-edge frameworks of Azure ML, a streamlined process for machine learning development right through to production, along with integrated continuous integration and continuous deployment (CI/CD) processes.

But this partnership doesn’t stop there. Snowflake is setting its sights on creating even more meaningful integrations with a host of Microsoft offerings, aiming to elevate the user experience even further. These plans include closer ties with Purview for advanced data governance, Power Apps & Automate for simplified, low code/no code application development, Azure Data Factory for efficient ELT processes, and Power BI for intuitive data visualization, among others.

The end goal? To foster a seamless ecosystem that capitalizes on the synergies between Snowflake and Microsoft’s product suites, unlocking new possibilities and delivering unparalleled value to users.

At DataTools Pro, we couldn’t be more excited to see our favorite data platform, Snowflake, with new enhancements that make data management easier. Azure balances powerful data management with scalable cost that makes sense for our clients. Additionally PowerBI continues to advance its dominance for Business Intelligence. We have been working with Snowflake and Microsoft together for years and have built a toolkit that can help you jumpstart Snowflake and Azure integration.

Learn how to use Azure Data Factory and Snowflake Together

We have created free interactive step by step tutorials to help you get started!

Create a Snowflake Data Source in Azure Data Factory

Create a Data Pipeline to Connect Salesforce to Snowflake

Publish your ADF Pipeline, Data Sets, and Triggers

Create an ADF Scheduled Trigger

VIEW ALL TUTORIALS

Azure Data Factory for Snowflake Articles

More Getting Started Tutorials

5 Min Snowflake Data Lake Powered by Azure Data Factory

Posted on 2023-05-14 - 8:11 pm by
Snowflake data lake powered by Azure Data Lake

In this video tutorial we build a Snowflake Data lake filled with Salesforce data using the power of Azure Data Factory. We built an ADF template that uses a few simple prompts to power a meta-data driven pipeline.

New Salesforce to Snowflake data pipeline template

View complete documentation

View more Tutorials

Download our Azure DataFactory Template Now

Name(Required)
This field is for validation purposes and should be left unchanged.

MVP Released in 2023

Learn more about the origins of our 5 Minute Snowflake data lake concept, where we automated our entire Salesforce data extraction process with a meta data driven approach. We used our own MVP to work with clients and refine our ELT solution with clients that invest in Azure for cloud infrastructure, Salesforce for sales, service, and operations, and Snowflake as a data platform for analytics.