Artificial Intelligence (AI) is everywhere. It recognises our faces to unlock our smartphones, it helps us to find our favourite product in an online shop, and it even is in our washing machines! It is no surprise then that AI has also found its way into the energy sector – it can forecast electricity demand, consumption, and cost, detect problems in energy systems, and much more.
AI seems like a magical creature, but in reality, it is a product of human intelligence and lots and lots of data. AI experts combine mathematic, statistic, programming, analytical, and many other skills to create AI models which are able to perform intelligent tasks. This is a complex process, where two key ingredients are necessary for an AI model to be able to forecast phenomena of interest.
The model needs to:
- gain experience – through data and interactions,
- know how to “think” and make decisions – which is defined by an algorithm.
At Endava, we have developed a solution for forecasting electricity demand, including getting the necessary data, developing an AI model that uses this data, and finally testing the whole solution.
DATA – OBTAIN IT, PREPARE IT
AI systems gain knowledge through data, and therefore, the data should be as good as possible in terms of volume, diversity, and quality. Simply put: the better the data, the better the prediction capability of the model. However, before arriving at the point where we can feed the model with the data, we have to take care of the following aspects:
Identify data sources: first, we need to identify the data sources which contain the data relevant to the problem we are trying to solve. Besides identifying obviously important sources, such as historical values of the variable we want to predict, in this phase we also create hypotheses and make assumptions about additional data sources that could contain valuable information. The practical usefulness of that additional data is evaluated later when we perform additional analyses and observe our AI model in action.
Obtain and process the data: next, we need to obtain access to the data sources, collect the data, parse, validate, transform, and store it. Since, in our case, the data was obtained from various sources, it was necessary to normalise it by transforming the data into a common format in terms of time zones, data frequency, etc. Moreover, since the real-life data is rarely ideal, we also took care of missing and anomalous data points. There are a lot of steps in data processing, and one needs to be very careful when carrying them out. Even the smallest mistake can shift or change the data in such a way that it stops representing a realistic scenario. Consequently, this can influence the forecasting power of the model trained on such data.
Understand your data and the domain: when working with data, it is important to explore and understand the specifics of our data and to know how the transformations affect it. Also, it is essential to learn about the particular domain we are working on and to apply this valuable domain knowledge as much as possible. For instance, weather can influence electricity demand, and this information should be taken into account when selecting data sources, for example. Only if we truly understand our data, the problem we are trying to solve, and the specifics of the domain, can we apply the right techniques and prepare the data for the next step: the algorithm.
THE ALGORITHM – THE BRAINS OF THE MODEL
Besides the data, the algorithm plays a crucial role in creating a predictive solution. The algorithm defines how the model processes data and makes predictions. From our experience, the implementation of an algorithm roughly incorporates the following aspects:
Select a machine learning (ML) algorithm: first, one needs to select a suitable ML algorithm. A lot of different algorithms exist, but it takes profound knowledge and experience to choose the best one (or several) for the problem at hand. The decision is made based on the type of problem we are trying to solve, the data characteristics, our resources, and previous experience. In our project, we aimed to forecast the electricity demand in a market, and for the basic ML algorithm, we chose the one called Random Forest due to its various advantages.
Implement the custom code: although the basic implementation of Random Forest exists in ML libraries, it was not sufficient to just call up a function from the library, pass the data as a parameter, and observe the final predictions. It was necessary to implement additional, carefully designed custom functions for the specific problem at hand. These combine various models for predicting demand in advance for different timeframes, process and aggregate the outputs, predict peaks, etc.
Again, understand your work: also in this phase, the domain knowledge and problem understanding are essential to creating a solution for predicting phenomena of interest in the best possible way.
DON'T FORGET TO TEST
Even if we carefully implement the code, patiently work with the data, incorporate expert knowledge into AI models, and truly understand every step on our way, one other key ingredient is necessary: testing. We wrote unit tests, component tests, integration tests, and estimated the predictive power of the developed models through carefully selected and designed AI validation procedures. Quality is important to us, and our goal is to deliver high-quality and robust AI solutions.
ON THE WHOLE...
Our journey hand in hand with artificial intelligence in the energy sector has been fantastic and inspiring – from collecting and processing the data to designing and integrating our predictive solution. When I think about it, I remember hard work, awesome ideas, and discussions which resulted in a whiteboard full of text and diagrams, but also all the time spent with a great team and their happy faces when, after all the hard work, we had achieved our goals.