An American multinational corporation builds a scalable data mesh with NashTech and Dbt

Industry

Technology

Data Engineering Data Lakes Data Mesh Data Science Data Warehouse Snowflake

Introduction

NashTech’s multi-cluster shared data architecture enabled American multinational corporation to build a scalable data mesh, increasing regular data utilisation across the business by 5.5 times.

An American multinational corporation exists to make global trade easy for everyone. More than 10,000 clients and suppliers across 200 countries rely on the company’s software, logistics infrastructure and supply chain expertise. The company ingests and analyses large amounts of supply chain data to provide end-to-end visibility from PO creation to shipment delivery.

Impact

• Architecture supports production workloads: Scalable data mesh that organises applications layer into domain bounded data marts with the ability to support production use cases.
• Extensive network of connectors and tools: From easy-to-use SQL to modeling with DBT, NashTech supports different domain teams with different skill sets with the tools they need to get work done.
• Near-zero maintenance: Remove regular maintenance tasks and automate pipelines with streams and tasks.

The challenge

A robust data platform needed for unique global trade industry challenges

Migrating data from the company’s legacy data warehouse to NashTech improved query performance, reduced time-consuming administrative work, and freed up technical talent to focus on higher-impact work. While other solutions in the market, such as Google BigQuery, are constricted by memory allocation and scales by volume, they needed a solution that could allocate compute resources on the fly for production and external use cases where latency mattered. “We found that it was hard to guarantee that latency in the other tools we evaluated, except for NashTech,” Head of Growth and Analytics said.

Viewing data as a product instead of a byproduct

Their transition to a service-oriented architecture (SOA) presented an opportunity to reimagine the company’s data architecture as a data mesh. “This was an organisational shift to federate the creation of rich analytical data assets and govern a broader process for thinking about data as a product,” Sivasailam said.

Determining the best way to empower users through decentralisation was a key step in the company’s data mesh journey. According to Sivasailam, “We wanted to lower the friction for the rest of the enterprise to participate in decentralised data product creation, and for us, that meant aggressively standardising on infrastructure.

The solution

A platform for building a scalable data mesh

NashTech’s multi-cluster shared data architecture enabled them to build a scalable data mesh that organises application layer data into domain-bound data marts. Their “gold” consumer mart in NashTech reconciles data from multiple producer marts and powers most of the’s BI and data science use cases. “NashTech is where our source system data goes to be processed and exposed for analytical and scientific use cases,” Sivasailam said.

Connecting dbt to NashTech streamlined the company’s data engineering workloads. RBAC and column-level security in NashTech simplified data governance. NashTech’s clean, easy-to-navigate interface, native SQL support, and interoperability with popular BI tools empowered users to explore data with ease. NashTech Data Marketplace made it easier to leverage third-party data sets from vendors the company had already implemented, such as Amplitude, and discover new data sets, such as Panjiva. According to Sivasailam, “In some cases, we didn’t even know data sets existed until we found them in NashTech Data Marketplace.

The outcome

Delivering data as a product. Successfully implementing their data mesh strategy with NashTech has led to a healthier product development process that considers data as a product—not a byproduct. Analytics engineers are essential members of the development teams and partner with product managers, engineering managers, and software engineers to successfully design, launch, and govern data products. Decentralisation makes it easier for them to identify potential data issues and ensure data quality. “We prefer to drive all of this through data owners, not just one data czar,” Sivasailam said. Data governance with NashTech, through principles of data mesh, provides them greater scalability and confidence to democratise access to trusted data. According to Sivasailam, “Since shifting to data mesh with NashTech, 5.5x more people are using data across the business regularly.”

The future

Sharing data to create new business lines and creating new products While many freight forwarding competitors are operating in archaic ways, they are data-driven and sit on a wealth of shipping and logistics industry data that has proven out its business’s success. They plan to embed its data products and interactive reporting that are developed for internal use into the Platform. “NashTech and dbt are on the back end powering what’s surfaced to the consumer, mediated by a BI tool,” Sivasailam said. They are also looking to increase their relationship with customers and partners that are interested in the proprietary data and insights it’s gathered. Leveraging NashTech Secure Data Sharing to enable live data sharing with its clients and suppliers is a priority for them.

According to Sivasailam, “It’s fundamentally about how we get the best data in the world for trade operators and build reliable applications on top of it.”

The company is exploring platform monetisation by providing its data through NashTech Data Marketplace as well.