How Suffolk Construction Leverages Procore Data in Real-Time with Toric
Having real-time data from Toric is allowing both our internal and external stakeholders to see their data updating live, which they are extremely excited about. In addition, the Toric tool has a great user experience and is an excellent addition to my engineering workflow.”
Amanda Moulaison, Data Engineer at Suffolk Construction
Key Facts
Amazon S3, Power BI, and Tableau
Author Note
You can imagine the shock I experienced when I joined the construction industry as a data engineer and discovered the deep complexity of construction data! Endless parameters, free text fields, and a seemingly infinite trickle of data points were coming in from so many sources.
Build Real-time Project Health Dashboards with Toric
Suffolk is prioritizing data and analytics at the forefront of our growth strategy. As our CEO, John Fish says, “if you can’t measure it, you can’t manage it.” As we continue to grow and expand our data strategy, we are always looking for the top technology and partners to level us up. We quickly identified Toric as a perfect fit for us to help.
Challenge
Getting Procore project data using our existing solution was slow and nowhere near real-time. Additionally, the dynamic nature of the Procore API made our solution very fragile - it broke frequently and these breaks were difficult and time-consuming to debug. In addition to giving us real-time data, Toric is also reducing the workload on our engineering team.
With our original Procore data ingestion pipeline, we frequently spent several hours per week debugging issues and trying to keep up with the changing Procore API. This forced us to be reactive instead of proactive, which made it difficult to be agile with this data.
Because our original ingestion solution was such a heavy lift on our engineering team, we had to prioritize and select which Procore endpoints to ingest. As a result of this, we weren’t able to pull in any tables that weren’t absolutely required due to time constraints. Now, because Toric has eased our engineering burden, we’re able to ingest and process all available Procore data tables. This in turn allows us to present more data points in our downstream dashboards and analytics, creating richer insights for our stakeholders. Lastly, my team and I love Toric because it’s data ingestion pipeline fits seamlessly into our existing data engineering workflow. Our pipeline leverages AWS S3 and Databricks to store and process our raw data before it lands in an AWS Redshift database to be used by our analysts and data scientists.
It was simple to set up our Toric pipelines to drop off our Procore data in S3 - just like our other existing data flows.
Solution
We use Toric to pull data from our Procore Projects to gain real-time project data using a solution that integrates with our existing data stack and automates data extraction.
Switching to Toric has also not only helped our engineering team - our data analysts, data scientists, and even our stakeholders see and appreciate its benefits.
In our Toric data ingestion pipeline, we were able to implement the Diff Node feature. This functionality instantly showcases the differences in a data set across time intervals by marking each record as “new”, “updated”, or “deleted.” This makes it much easier for our analysts to track and report on the ongoing progress of any project.
Our stakeholders are enjoying seeing real-time data updates in their Tableau and Power BI dashboards and other analytics tools. In fact, the seamless and instantaneous nature of Toric data ingestion pipelines has led our team to investigate introducing new data tools to our company, such as installing interactive touch-screen displays throughout our office to display analytics and trending data.
Result
Our team’s original Procore data ingestion solution met our minimal needs by processing and delivering our highest-priority data at daily intervals. Since switching to Toric, we can’t imagine going back. Toric has significantly eased the burden on our engineering team, which has increased our bandwidth and allowed us to focus on new projects and integrations. This process has saved our data team valuable time, and now our organization can make better-informed decisions faster, thanks to Toric.
Since completing our Procore data ingestion pipeline buildout, we have begun exploring the opportunity to leverage Toric for other data integrations as well, including Oracle Primavera and Revit.
