Skip to main content

New Azure DataFactory template makes Salesforce to Snowflake Pipelines fast and cost-effective

Azure DataFactory for Snowflake and Salesforce

We built our free Azure DataFactory template to help you build your data pipelines from Salesforce to Snowflake in 5-30 minutes. The value of data is not realized by collecting and moving it. The value of data is realized when you transform it into information. Analytics insights and attributes for automation is the objective and reason why you invest in a data warehouse like Snowflake. That is why we have built our free data pipeline templates to reduce the level of effort to get your data ready for analysis up to 90%! View our documentation to lean how

In 2023, we launched the first version of our template tagging it as a “5 min data lake with Azure DataFactory”. Adoption and feedback led us to close 2024 with an upgraded version of our template alongside our new DataTools doctor service and our revised Snowflake rapid adoption service to help our customers extract value from data faster.

Download our Azure DataFactory Template Now

Name(Required)
This field is for validation purposes and should be left unchanged.

Why we build Salesforce to Snowflake pipelines with Azure DataFactory?

Fast, cheap and easy rarely happens in technology, but the Azure team, without a massive marketing blitz or fanfare created a very solid product for moving data between enterprise data sources in to Microsoft data platform and Snowflake.

We use Azure DataFactory pipelines to move massive volumes of data daily for customers who have invested in Azure. They save thousands of dollars per month while getting enterprise grade data extraction and migration.

When do we turn to other solutions than DataFactory?

We stick to technology tools that are flexible, practical, and well adopted. DataFactory in particular we like deploying with customers who are running MS SQL or Snowflake in Azure. When it comes to data transformation and ETL patterns, DataFactory does offer a no/low code Spark based Flow builder. However, depending on customer needs, scale, and team makeup, we do recommend alternative solutions that is catered to the existing processes, investments, team makeup and roadmap. We are always on the lookout for new, streamlined data pipeline solutions.

What’s Next for our Azure DataFactory template?

We are actively working on client projects for MySQL and MS SQL version of our template. Contact us for more details

What’s New in our Salesforce to Snowflake Pipelines

This week, we rolled out a long overdue update for our free Azure DataFactory template that makes extracting Salesforce data into Since 2023, there have been a lot of changes to Azure DataFactory, so we have rolled out a long overdue update and upgrade.

Salesforce to Snowflake ADF Data Lake

From our version 1.0.1 release notes:

  • New meta data staging process and table called “SFDC_METADATA_STAGE_TEMP” that feeds SFDC_METADATA_STAGE
  • Support for new field detection and addition (enable append fields)
  • New parameter AppendFields will insert new fields when detected
  • New parameter SnowErrorHandling allows for configurable error handling to skip error rows, skip file, or throw an error.
  • New MetaData field called “Status” that allows for “Disable” attribute that will ignore fields from being synchronized.
  • Update to Salesforce metadata request that supports compound fields by default like FirstName, LastName, Street, City, etc.
  • Pre-install check – End to end flow checks for existence of the metadata object
  • Added status variables to determine results for each pipeline for easier debugging
  • Schema insert and updates managed via merge by object.field ID
  • Changed field from ID to DurableId Salesforce field to Snowflake SFDC_METADATA_STAGE “ID”

Why taking a SQL Course still matters in a world with AI

SQL for Salesforce

This week, Salesforce Ben released new SQL course for Salesforce that aims to introduce a SQL learning path aimed at professionals who work in Salesforce. My goal for the course was to provide technical training from the perspective where data literacy and translating business questions is the driver to write SQL. In my course, I lean into LLMs, specifically ChatGPT, and even introduce how I use ChatGPT to assist debugging. A the end, anyone who takes the course will learn their way around Snowflake and have a lab built for funnel analytics.

Large Language Models (LLMs), are make it easier than ever to write SQL and Python. Some have made bold claims that learning how to code wont be necessary in the future. Despite advances in LLMs, SQL remains a vital skill in the data-driven world.

Writing SQL without LLM

  • Lots of time consumed troubleshooting and debugging SQL
  • Reverse engineering other’s SQL
  • Manually typing documentation
  • Understanding how functions work
  • Formatting data to use in expressions
  • Understanding data structure and meta data

Using LLMs while Writing SQL

  • Paste your code and the error and let LLMs point out syntax issues or how to correct errors
  • Break down and explain SQL structure and purpose
  • Auto-document SQL
  • Relate functions to your existing knowledge
  • Auto-prepare expressions with correct syntax
  • Explain meta data structure

What you get out of learning SQL course for Salesforce

1. General understanding – Fundamental SQL skills

AI tools like LLMs can write SQL queries, but without a solid grasp of SQL fundamentals, it’s challenging to evaluate or optimize those queries effectively. SQL is more than just a query language; it’s about understanding how data is structured, how relationships are built, and how to extract meaningful insights from databases. SQL gives you the foundation to translate questions into queries and ensures that you’re not just a passive consumer of AI-generated code.

2. Contextual Awareness

While LLMs are powerful, they might not fully grasp the nuances of your specific database environment or the business rules that govern your data. Learning SQL allows you to tailor queries to your unique context, ensuring the results are accurate and aligned with your business needs. This contextual understanding is something that AI, despite its advancements, can’t fully replicate.

3. Collaboration with Data Teams

SQL acts as a common language in the data world, bridging the gap between business professionals and technical teams. When you understand SQL, you can communicate more effectively with data engineers, analysts, and other stakeholders. Understanding the data structures needed for analytics also increases your awareness as you alter the Salesforce data model. At the end of the day, having SQL in your toolkit makes you a more valuable contributor.

4. Troubleshooting and Optimization

Even the best AI tools can generate inefficient queries that may impact system performance. By learning SQL, you gain the ability to troubleshoot, optimize, and refine these queries, ensuring they run efficiently and deliver the desired results.

5. Future-Proofing Your Career

SQL skills continue to be in high demand, with job opportunities in this field projected to grow significantly over the next decade. As DataCloud takes off, employers will value SQL proficiency, as it’s a core skill for data cloud related roles when you need to “bring your own data warehouse.”

More about SQL course for Salesforce

What You’ll Learn:

  • Data Query Language (DQL): Focus on querying and analyzing data.
  • Salesforce Integration: Learn how SQL concepts align with Salesforce SOQL.
  • Practical Skills: Hands-on exercises to build familiarity and proficiency.

Exciting New Native Salesforce Snowflake Integration

Salesforce and Snowflake Integration

When it comes to optimizing your business processes and data analytics, Salesforce and Snowflake stand as two potent platforms, each with its own ecosystem of developers, stakeholders, and users. The Salesforce Snowflake Integration is an essential conduit that amplifies the bond between these two cloud platforms.

Salesforce and Snowflake Integration

Native Salesforce Snowflake Integration: A Milestone in Native Data Sharing

Earlier this week, Salesforce and Snowflake made a groundbreaking announcement: the general availability of native Salesforce data sharing for Snowflake,  via what is colloquially referred to as “BYOL” (Bring Your Own License). This is a significant advancement, especially for Snowflake users familiar with the benefits of zero-copy sharing, a core Snowflake feature. With this integration, gone are the days when you needed layers of additional software, services, and complex processes to bridge the two platforms. This is where the Salesforce Snowflake Connector comes into play, simplifying data access and queries between Salesforce and Snowflake.

Skill Enhancement through Certification Paths

Salesforce Data Cloud serves as a data hub orchestrating a wide range of business activities—be it CRM, marketing, or or any web/mobile activities. To encourage this, Salesforce recently launched its Certified-Data-Cloud-Consultant learning path. This will help Salesforce organizations readily find skilled professionals adept in Salesforce Snowflake Integration.

Salesforce Runs on Snowflake: Following the Leader

In a revelation that should add credibility and assurance to the Salesforce Snowflake Integration, Salesforce’s internal data and analytics have migrated to run on Snowflake. This shows Salesforce is not just advocating for the technology but using it themselves, setting the stage for rapid advancements in Salesforce and Snowflake connectivity.

Transforming AI/ML Workloads

The Salesforce and Snowflake partnership holds tremendous promise for accelerating the time-to-value from your Salesforce data assets. From curating data to deploying ML models, the integration, facilitated by the Salesforce Snowflake Connector, will enable enterprises to leverage their data in novel ways, including the utilization of advanced AI features. There are many first and third party powered solutions to weave your model deployment efforts.

Need Help Navigating these Waters?

We have been in front of Salesforce and Snowflake integrating analytics apps for years. We recreantly wrote the  Salesforce data synchronization to Snowflake Guide and can’t wait to extend this into DataCloud. We have an incredible partner network that can help you implement any Salesforce or Snowflake Cloud components (CDP, MarketingCloud, Tableau).

Schedule a meeting to learn more

Unlocking the Full Potential of MLOps with Snowflake and Predactica

Machine Learning Network

The Evolution of Machine Learning Platforms

Snowflake has rapidly emerged as the go-to platform for data-centric enterprises. Its ability to centralize and harmonize diverse data types makes it an exceptional foundation for any data strategy. Machine Learning Operations (MLOps) have matured significantly over the years, thanks to Platform as a Service providers like Amazon, Google, and Microsoft, who have developed comprehensive solutions that span the entire model lifecycle. However, a plethora of platforms and tools exist in a very crowded and confusing marketplace. So, what should a customer with Snowflake, a data and analytics team, and a desire to get models to production quickly and practically do?

Predactica: The is a Glimpse of the Future of MLOps in a Snowflake-Centric World

One name that should be on your list is Predactica.. This innovative solution is engineered to fit seamlessly with Snowflake. Predactica elevates MLOps by offering a natively integrated, end-to-end machine learning solution within Snowflake. Unlike other platforms that require disparate workflows and additional data pipelines, Predactica unifies these operations, making it the ideal companion for Snowflake-centric enterprises.

The result is a unified, agile, and compliant system that dramatically reduces the time-to-market for new models while ensuring their long-term reliability. Risk modelers and data scientists can now focus on the nuances of data, feature engineering, explanation and fine tuning.

Snowflake as Center of Gravity for Enterprise Data

With the introduction of Snowpark, Snowflake has also paved the way for native model deployment, allowing organizations to manage the entire model lifecycle within the data platform. This is done using the same tools, Python libraries, and workflows that data scientists, data engineers, and DevOps professionals already use. However, the rapid evolution of MLOps calls for a more streamlined, low-code solution that can natively integrate with Snowflake. This is where Predactica comes in to compliment or potentially replace external ML platforms and expand aspects of your MLOps to more contributors.

The Competitive Edge: Agility, Compliance, and Real-Time Monitoring

Another often-overlooked aspect of the machine learning lifecycle is monitoring model performance over time. Models, especially in credit risk, are not “set and forget.” They require ongoing attention to ensure they do not degrade and continue to make accurate predictions as market conditions and customer behaviors change. Predactica addresses this crucial need by offering built-in performance monitoring features. These tools enable teams to catch performance drift early, allowing for timely model adjustments and ensuring that your decision-making remains both agile and accurate.

Conclusion

The collaboration between Snowflake and Predactica represents a leap forward for organizations looking to democratize model development and accelerate speed to value.. Don’t take our word for it, setup a meeting or sign up for a trial and let us know what you think! Sign up for a Predactica Trial

Ultimate Salesforce and Snowflake Guide on Salesforce Ben

Salesforce Ben

This week Ryan released a guide for Salesforce and Snowflake on Salesforce Ben. Salesforce Ben is the leading independent Salesforce.com community and authority on all things Salesforce.com.

Snowflake and Salesforce is a perfect marriage of cloud business applications and cloud data platform to turn data into information. Salesforce has built a powerful first-class integration within Salesforce Data Cloud that is the most advanced of any third party connectivity

If you are currently using Salesforce Data Cloud or Salesforce Tableau CRM this article is for you. Additionally, while at SnowflakeSummit2023, we saw some incredible Salesforce Data Cloud enhancements for Snowflake that will be game changing for enterprise cusetomers.

We can’t wait to write about upcoming zero copy feature from Salesforce to Snowflake. Included in our article is step by step tutorials how to integrate Salesforce with Snowflake to day. Should you have any questions how these capabilities apply to your enterprise or how Snowflake can advance your Salesforce analytics, we are here to help!

Snowflake and Microsoft Expand their Data and AI Partnership

Microsoft and Snowflake Logos

Snowflake and Microsoft, announced a press release at Snowflake Summit 2023 that they are expanding their partnership promising substantial advancements for data scientists and developers. This enhanced collaboration is set to seamlessly merge Snowflake’s Data Cloud with Microsoft’s Azure ML, extending its capabilities through the potent combination of Azure OpenAI and Microsoft Cognitive Services.

This strategic alliance means that Snowflake and Microsoft Azure shared customers will gain access to the cutting-edge frameworks of Azure ML, a streamlined process for machine learning development right through to production, along with integrated continuous integration and continuous deployment (CI/CD) processes.

But this partnership doesn’t stop there. Snowflake is setting its sights on creating even more meaningful integrations with a host of Microsoft offerings, aiming to elevate the user experience even further. These plans include closer ties with Purview for advanced data governance, Power Apps & Automate for simplified, low code/no code application development, Azure Data Factory for efficient ELT processes, and Power BI for intuitive data visualization, among others.

The end goal? To foster a seamless ecosystem that capitalizes on the synergies between Snowflake and Microsoft’s product suites, unlocking new possibilities and delivering unparalleled value to users.

At DataTools Pro, we couldn’t be more excited to see our favorite data platform, Snowflake, with new enhancements that make data management easier. Azure balances powerful data management with scalable cost that makes sense for our clients. Additionally PowerBI continues to advance its dominance for Business Intelligence. We have been working with Snowflake and Microsoft together for years and have built a toolkit that can help you jumpstart Snowflake and Azure integration.

Learn how to use Azure Data Factory and Snowflake Together

We have created free interactive step by step tutorials to help you get started!

Create a Snowflake Data Source in Azure Data Factory

Create a Data Pipeline to Connect Salesforce to Snowflake

Publish your ADF Pipeline, Data Sets, and Triggers

Create an ADF Scheduled Trigger

VIEW ALL TUTORIALS

Azure Data Factory for Snowflake Articles

More Getting Started Tutorials

5 Min Snowflake Data Lake Powered by Azure Data Factory

Snowflake data lake powered by Azure Data Lake

In this video tutorial we build a Snowflake Data lake filled with Salesforce data using the power of Azure Data Factory. We built an ADF template that uses a few simple prompts to power a meta-data driven pipeline.

New Salesforce to Snowflake data pipeline template

View complete documentation

View more Tutorials

Download our Azure DataFactory Template Now

Name(Required)
This field is for validation purposes and should be left unchanged.

MVP Released in 2023

Learn more about the origins of our 5 Minute Snowflake data lake concept, where we automated our entire Salesforce data extraction process with a meta data driven approach. We used our own MVP to work with clients and refine our ELT solution with clients that invest in Azure for cloud infrastructure, Salesforce for sales, service, and operations, and Snowflake as a data platform for analytics.

Snowflake Warehouse Management with ROI in Mind

Snowflake Cost Management

If you’re new to Snowflake, you might be confused by the term “Warehouse”. Don’t let it fool you, because in Snowflake’s context, Warehouse refers to virtual compute resources rather than a physical storage place. Snowflake Warehouse management for small BI and analytics teams is fairly straight forward if you start off on the right foot.

A majority of Snowflake’s cost is based on warehouse (compute) utilization. Therefore, it’s crucial to be thoughtful about how you design and deploy your Warehouses to optimize your usage and minimize your cost.

Segmentation of Warehouses

One of the key factors in optimizing your Snowflake Warehouse is segmentation by use case and spend categorization. For instance, our Snowflake instance currently consists of 5 warehouses, with each one serving a specific purpose. We started with X-Small or Small instances that can process thousands up to tens of millions of records, and gradually scaled up as needed.

However, over-segmenting and creating too many warehouses is not recommended. This can lead to unnecessary concurrent warehouse instances and significantly increase your spend. Additionally, detailed spend tracking can become very expensive and difficult to manage. Therefore, it’s important to strike a balance between segmentation and cost optimization to achieve the best outcome for your Snowflake usage.

Warehouse Segments and Lessons Learned…

Read more on our Medium Blog

How my Snowflake Powered Lead Distro Test Turned Out to be Reverse ETL

Snowflake Cloud Data Pipelines for Reverse ETL

A year ago, I worked on a small project to help us improve our data driven funnel. I learned what I called “Snowflake to Salesforce analytics sync” had a more buzzworthy term called “Reverse ETL.” This article shares some of the lessons learned along the way and some thoughts about where reverse ETL is headed.

Low Level of Effort Solution

All of the data and metrics were already available and calculated in Snowflake for reporting, so the process to push those measurements back into a Salesforce object using Azure Data Factory was quite simple.

The transformation work was prepared using Datameer on top of Snowflake which I had previously written about: Slice Through your Snowflake Data like a Buzzsaw with Datameer