Patient analytics platform for Shoppers Drug Mart on Google Cloud

Industry

Technology

Apache Beam BigQuery DLP GCPS - OCR Kubernetes Pub/Sub SpringBoot

Introduction

We helped conceptualise, design and deliver a highly scalable and resilient GCP native Datalake solution employing GCP infrastructure and services such as Apigee, Kubernetes Engine, Pub/Sub, Dataflow, Data Loss Prevention and BigQuery.

About Shoppers Drug Mart

Shoppers Drug Mart is Canada’s leading pharmacy retailer, with more than 1,300 pharmacist-owned locations across the country. Founded in 1962, Shoppers Drug Mart is a subsidiary of Loblaw Companies Limited and has headquarters in Toronto, Ontario, Canada.

NashTech helped Shopper’ Drug Mart (SDM) build Google Cloud Platform (GCP) based scalable data pipelines to capture patient interactions in near real-time from a variety of proprietary SDM patient forms. Captured data is stored in BigQuery based DataLake and is available to run insights and analytics that can help improve patient outcome and increase operational efficiency. This solution has enabled SDM to capture and transform data from more than 30 patient and pharmacy forms, data is processed in GCP at a peak rate of 10K form submissions in an hour. Without this solution, the data captured through user interactions with these forms was not saved and hence not available for running any analytics.

Impact

Previously uncaptured patient user experience is now available to run insights and analytics
Patient and pharmacy user interactions captured from SDM’s 1000+ stores across Canada
Millions of records captured in first few months of operation
Seamless integration of new user experiences without any service disruption
Supports auto-discovery of dynamic data bindings
PII Data is masked using Cloud DLP and custom masking techniques
Scalable architecture that can support higher peak loads and addition of new forms without any manual intervention

Problem Statement

Shoppers Drug Mart utilises a set of web based forms that provide a variety of user experiences to both pharmacists and customers while screening and/or registering for various services like vaccines, medication review, health checkup, etc. The information captured by these user experiences were not persisted anywhere previously. The data captured is dynamically created based on user interactions. A solution was required that can capture data from these form submissions and store into a Datalake on GCP. The solution should be scalable and adhere to all security and privacy requirements.

Solution

NashTech’s team of skilled engineers and GCP experts designed a solution that was highly scalable and enterprise ready using Java and utilising several GCP native services. The solution was built to support the dynamic nature of the forms and requires no enhancements or service disruptions for introduction of any new form and/or updates to any existing form. The captured data is stored as events in an event store and also to the Datalake built on BigQuery. All PII data is de-identified before saving to the Datalake. Metric collections and monitoring was achieved using Stackdriver, Prometheus and Grafana.

Scalable Data Acquisition API: Real-time data is routed to the data pipelines via Springboot based Scalable Microservices hosted on Kubernetes Engine.
Event driven architecture with Pub/Sub: Encrypted data is pushed as events to a Pub/Sub queue.
Resilient Parallel Processing using Dataflow: Data is processed, transformed and stored into BigQuery Datalake using Apache Beam based Dataflow runners. Following tasks are performed by Dataflow:
- Store encrypted events in an event store
- Perform data quality checks
- De-identify PII data using Cloud DLP and custom masking
- Auto-discover schema and transform data to be stored in BigQuery Datalake
BigQuery Datalake: Captured data is made available for analytics and predictions via BigQuery tables and views.
Observability and monitoring using Prometheus and Grafana: All the components of the pipelines are integrated with Prometheus and Stackdriver. Metrics are monitored over Grafana dashboards

The outcome

All the form data is now ingested in BigQuery and SDM’s Data Science team is now able to access the customer and pharmacy data for further insights and analytics use cases – helping SDM improve patient outcomes, improved operational efficiency, etc. It supports the dynamic nature of forms and requires no enhancements or service disruptions for handling any new form or updates to any existing Forms.

“Our solution saves this information to BQ which can be utilised for any further processing like analytics, providing additional user experience based on previously captured data, having events/triggers where applicable and any other use cases on available data. We see this only as the beginning of our journey with SDM.”
Senior Director, Data Engineering – Shoppers Drug Mart