What do we mean by quantitative and data visualisation skills?

The last decade has seen the advent of the near ubiquitous collection of high-quality data across industry. However, of interest is that high quality curated data is not a new phenomenon in the electricity sector. The very nature of the sector, with its need for continuous monitoring of supply and demand has meant that rich, curated data sets have been available for over twenty years.

The electricity sector has therefore developed a relatively advanced standard for the analysis and presentation of data. The ability to take large, many-faceted data sets and present them in new ways that can provide insights is a critical skill.

A more recent (the last 5 years) phenomenon is the advent of machine learning tools in the sector. With the open-sourcing of TensorFlow and other powerful deep-learning tools, new methods are emerging in the electricity sector that can:

  • dramatically improve forecast accuracy;
  • be used to estimate key optimisation parameters, and so inform prescriptive analytics; and
  • allow the creation of synthetic data that can be used to expand the range of possible outcomes beyond historical data sets.

What skills and knowledge does Endgame possess in these areas?

Our team members have been analysing and working with electricity market data virtually continuously for the last decade. We possess programming skills in Python and R and are comfortable working with either language to conduct analysis of large data sets either locally or, where security permits, on the cloud. We have worked with such large data sets as half-hourly customer data for hundreds of thousands of residential and business customers, and the 4 second SCADA data collected by AEMO (used to calculate frequency control ancillary service liabilities).
In terms of data visualisation, our team members are well known in the Australian electricity sector for the unique and distinctive charts that we produce for quarterly forums, and that are included in some of the our clients determinations. We are expert users of Tableau and have run courses teaching others how to use this tool to analyse electricity market data.
Finally, we have worked with both deterministic and probabilistic machine learning models, implemented in Python. We have used these models to create synthetic demand data using over a century of historical climate data, and to create short-term forecasts that improve upon those produced by other methods. We see the increasing use of machine learning as an inevitability in the sector and are working actively to build our own experience in this area, and to recruit experts with these skills.