Today’s post is from Jamie Walters, CEO, QiH Group, an online digital marketing company, with its own affiliate networks, Tau Affiliate and Offer Tonic. Its goal is to attract new customers to sites and keep existing customers engaged. Founded in 2014, the company now works with a portfolio of more than 50 sites, including secretslots.com, and aims to launch a new brand or site once every quarter.
About 18 months ago, we launched an initiative to improve our reporting and data usage. Data has always been at the heart of our business and as the company has grown, so has our dependence on data.
Need for a single source of truth
Our main focus is to bring together data from multiple sources so that our “business owners” can devise campaigns for each of the specific brands they manage and evaluate their success. We also need to report earnings to our network of affiliates and provide daily KPI snapshots for the whole company.
To handle these reporting needs, we’d developed a system in house based around Google BigQuery. It pulls in data from social media platforms, along with data about keyword advertising, email and text campaigns, as well ingesting player activity streams from each of the sites we work with.
Over time, we began to see a growing number of performance issues, with at least one data pipeline failing in some way every month. Users started to lose trust in the results and were spending increasing amounts of time drilling down into the data to check results manually. Adding a new brand, a new marketing channel or any new data source was also slow and difficult because it involved combing through lines of SQL code to make the necessary changes.
Technical set up resulted in high costs
At the same time, our costs for Google BigQuery were spiralling because our data was held in a single table and accessed live with Tableau reports. Every time anyone used a report, the system was processing the whole table. It could also take up to ten minutes for a report to run and load, adding to our users’ frustration. On top of that, we were regularly charged a premium of up to 50% for exceeding the contracted monthly allowance for our third party data pipeline tool.
The turning point came when our in-house technical experts left and we made a business decision not to replace them. We recognised we needed outside help — and that led to us finding Ancoris.
Bringing simplicity and cost savings
From day one, I could tell the Ancoris team knew what they were doing and what works and doesn’t work. We’re a complex business and although some of our challenges were probably the first time they’d seen something exactly like that, they were still able to grasp the issues very quickly and immediately make several suggestions for how to proceed. And because they’re Google experts and very familiar with all the different products from Google and its partners, they were able to lay out the advantages and disadvantages of each option for us and handle the technical implementation once we’d made our choice.
For example, they suggested we use a combination of Fivetran and dbt (data build tool) in place of our existing data pipeline tool. Not only is this combination much less expensive, but it’s proving far more reliable, allowing us to build trust among our users that they’re looking at data that’s accurate and up-to-date. With these tools, it also takes just a few hours to add new data sources and create new data pipelines if we’re launching a new brand or working with a new marketing channel.
Another benefit of dbt is that it automates documentation of our dataflows. We were aware of this requirement in our old system, but we hadn’t invested the time to do it manually. Now we don’t need to: dbt has done it for us and gives us a nice visual representation — that’s easy to understand for someone who doesn’t write code — of where data is coming from, what the dependencies are and what transformations have been applied.
Using building blocks to create the new platform
With this kind of support from Ancoris, we’ve been able to use existing building blocks provided by Ancoris to rebuild our data reporting in just a few months. Taking this approach has eliminated issues such as inconsistent naming conventions and it’s enabled us to make better design choices in every area. Even simple suggestions from Ancoris about how to partition tables and structure our data into raw, clean and presentation layers have significantly improved efficiency and speed when running reports and digging into the detail.
Throughout the project, it’s been an absolute pleasure to work with Ancoris. We held formal weekly progress meetings online using Google Meet, but the team is very approachable. If we ever have any questions, they’re immediately available via Google Chat or jumping on a call. From the outset, it’s been a massive weight off my shoulders to always have that expertise available and to be working with people who are very good at translating from technical talk into layman’s terms.
Moving forwards, Ancoris will be running the system for us as a managed service, handling routine management and maintenance while helping us explore how we can get more value out of our data with technologies like machine learning. We’re the perfect business for that, because we’ve got all sorts of information about our players that will allow us to build prediction models based on really specific customer demographics. Now we have our data in order, plugging it into machine learning tools will open up no end of possibilities for us.
Working with Ancoris data analytics and AI team
Ancoris Data, Analytics and AI practice brings together a highly committed team of experienced data scientists, mathematicians and engineers. We pride ourselves in collaborating with and empowering client teams, such as QiH Group, to deliver leading-edge data analytics and machine learning solutions on the Google Cloud Platform.
We operate at the edge of modern data warehousing, machine learning and AI, regularly participating in Google Cloud alpha programs to trial new products and features and to future-proof our client solutions.
We have support from an in-house, award winning application development practice to deliver embedded analytics incorporating beautifully designed UIs. We are leaders in geospatial data and one of the first companies globally to achieve the Google Cloud Location-based Services specialisation.
If you'd like to find out more about how we can help you build your own modern data and analytics platform, why not take a look at some of our customer stories or browse our resources. Needless to say, please get in touch with our team if you'd like more practical support and guidance.
