Why is data engineering important for WFM and AI?

March 30, 2023

Suppose you are considering an enterprise-level Workforce Management platform with AI-powered demand forecasting and auto-scheduling. In that case, it is worth having a basic understanding of the role of data engineering in your proposed solution. Why? Because a big part of what your software vendor provides relies on data engineering to automate the data management required for fully AI-powered workforce management software. In this blog post, we outline the basics for a non-technical person to understand better why is data engineering important for WFM and AI and what it does.

What is data engineering?

Data engineering refers to the building of systems to enable the collection and usage of data. This data usually allows subsequent analysis and data science, which often involves machine learning.

What is the difference between data engineering and data science?

Data engineering is used to make sure all of the data needed to train an AI is in the right place and format through a fully automated process. Whilst data science takes that data and uses it to create an AI or machine learning model. Data engineering then extracts the output and loads it where it is needed. Data engineering becomes more important if you want bespoke AI solutions for your organisation.

Why is data engineering important for WFM and AI

What is a platform-wide AI compared to a bespoke customer AI?

Platform-wide AI solutions train an Ai model using all the data across a given software platform. In comparison, a bespoke solution will use data specific to the customer. 

To help us understand the concept, let us consider an example of a platform-wide AI. Alexa or Siri are both platform-wide AIs trained on millions of voices; they access one data set of every customer’s voice and learn from those voices. They are not trained on your voice or to recognise your accent and vocabulary. To be bespoke, they would have to store the voice data from you separately and train an AI to recognise your voice. Alexa and Siri would need to manage millions of data sets and millions of AIs. 

Now let’s consider a bespoke AI in a workforce management solution. Because the customer is a larger company with 100’s or 1,000’s of employees, it is possible to keep all of that company’s data separate and build an AI that only looks at that data. That is a bespoke AI. It could even use all the data from a single department or venue to build a model that forecasts the sales and demand curve for just that department or venue. As all retailers will know, every store is different so the ideal requirement would be a dedicated AI per store.

So why is data engineering so important for a workforce management system that uses AI?

To really meet an organisation’s requirements and deliver the best return on investment possible, workforce management requires a bespoke AI for each department for forecasting and a bespoke AI for the company for auto-scheduling. This can only be achieved with data engineering and a “pipeline” that manages that data.

To deliver the maximum benefit, a workforce management system needs inputs from at least three separate AIs:

  • Forecasting: creates an AI per store or department that forecasts the factors that correlate with staffing levels. This is often sales or footfall. In our experience, we often build three or four forecasts per store to get a very accurate demand forecast, for example, revenue, items, baskets, and footfall. These forecasts need to be per store per hour for a whole week to build a rota.
  • Demand: This uses the forecasts created above to generate a required staffing level for every role required for every store or department for every hour. If you have a complex business with five different roles, you will need 5 AIs, which need to be bespoke per venue. The layout of a kitchen and the distance required to deliver service can make two different demand requirements for two restaurants of the same chain.
  • Auto-scheduling: This is a complex AI which will build the optimal rota for a given set of criteria. We have a blog post dedicated to all those criteria, so we won’t go into them here. This AI can often work across every department in a company because companies tend to have set rules that apply across the whole organisation.

Hopefully, you are getting the idea that if you need nine different bespoke AIs per venue and one very complex AI per company and you want the process fully automated, then it can be quite complicated. Suppose we have 100 venues with 9 AIs per venue, and you want an up-to-date forecast every Monday at 09:00. To deliver this, you will need 900 data sets, train 900 AIs and move the data along the pipeline such that the Scheduling AI uses the results from the demand which in turn uses the results from the forecast AIs. 

This can only be done using some pretty advanced data engineering techniques.

What sort of tasks does data engineering do for workforce management software?

Let’s look at the example of a forecast AI.

The first task is to build a data set that can be trained for the forecast. This will involve taking a couple of years of historical transaction data right up until the time the forecast is run. You then need to add in third-party data such as weather and sporting events. The data needs cleaning and formatting and then is separated on a per-venue basis.

We then need to carry out “labelling” to support a supervised machine learning process. This involves taking the data we have and running a series of calculations, making it easier for the AI to spot patterns. For example, we might calculate the percentage of pre-prepared items and the percentage of live-prepared items such as coffee. There may be a requirement for hundreds of labels per dataset.

Once we have our labelled datasets, we split them into training and test data.

We then run 100’s of models against the training data and then use an algorithm to test all of the models against the test data and select the best model.

Next, we run the chosen model against the live data and produce our forecast.

Finally, we take our four forecasts per store and deliver them to the relevant demand AI and probably also to a business intelligence tool.

The whole process is automated and is done using data engineering techniques.


Hopefully, this wasn’t too technical and this blog post has given you an idea of just how many automated steps are involved in producing multiple AIs per venue or department and the role that data engineering plays. This level of bespoke AI build is necessary to generate great results with a workforce management platform and AI, and this can only be delivered using data engineering. That is why data engineering is so important if you want to use AI and WFM together.

Related Articles:

Solve Your Biggest Operational and HR Challenges Using Proven AI Solutions that Deliver ROI

You don’t need development skills or data science engineers to implement our solutions. They are quick and easy to integrate with your existing tech stack.

Latest blog posts

AI Consulting Glossary

Consulting has lots of jargon, acronyms, abbreviations and technical terms – this is particularly true when looking at specialised consulting in the fields of artificial intelligence and machine learning. In...

read more