In a list of the most influential phenomena in social and cultural history, the weather can stake a pretty good claim to a place near the top.
The weather touches almost every aspect of our lives, both on a micro and on a macro level. Think about those washed out family holidays back when you were a kid, and how those memories contrast with those of sunny summer days on the beach, of ice cream and sun tan lotion; think about the sporting events influenced by gusting winds or a sudden rain storm; think about architectural innovations designed solely to deal with extreme meteorological conditions; think about the ancient gods of thunder and lightning; think about all that social awkwardness eased by a timely remark on the likelihood that it may rain later in the day.
Weather is important, and so it makes sense that we should apply Big Data to our understanding of it. Computing giants IBM recognized this; which is why they acquired The Weather Company in 2016, in an effort to provide businesses with the weather data they need, and to position themselves strongly in the market.
However, IBM are far from the only industry body delving into the world of Big Data weather forecasting.
Business and Big Data Picking Up the Slack
In 2013, the National Weather Service’s budget was decreased across several key areas. Regional information technology collaboration units received a funding cut of almost $10m, the tsunami warning network funding was cut to the tune of $4.6m, while satellite profiler and conversion, and air quality forecasting departments encountered cuts of $4.1m and $3.1m respectively.
In other areas, the National Weather Service’s budget was raised, but, as specific budget portions are ring-fenced for specific purposes, the NWS found it increasingly difficult to provide the same levels of data and insight as before the cutbacks.
Four years on, and we have seen private businesses – armed with Big Data – stepping in to pick up the slack. One example of this in the USA comes from Raytheon – a defense hardware manufacturer – who have partnered with NASA and other governmental organizations to develop the Joint Polar Satellite Mission, plugging the gaps which have been left by reduced funding for weather services.
This private-public sector dynamic looks set to become a regular fixture of weather analysis in the near future. In April of this year, the Weather Research and Forecasting Innovation Act (2017) was signed into law by President Donald Trump. This bill is designed to facilitate the transfer of technology between public sector organizations – such as the National Weather Service – and their private sector counterparts.
This is perhaps not an ideal situation – it is always a little concerning to see direct public services like weather forecasting pushed into the private sector due to lack of funding – but it could spell very good news indeed for those who rely upon accurate weather reports. The private sector has the financial and meteorological resources required to provide such reports, as well as a history of effective innovation.
New Concepts in Weather Analytics
Humanity has been collecting weather data for generations, gradually developing an in-depth understanding of our planet and what makes it tick. The latest concepts which are driving the Big Data approach to weather forecasting revolve not so much around collecting this data as interpreting it and putting it to good use.
As we know, Big Data is expanding almost by the second, and the same applies to the Big Data stores we hold and manage in relation to the weather. One way in which organizations are hoping to achieve this is via High Performance Computing – or HPC – something which the team at Hewlett Packard Enterprises are working to develop.
Hewlett Packard have identified several unique challenges relating to Big Data analytics in the field of meteorology. Firstly, weather forecasting requires the analysis of patterns and models which have developed over decades, which means that software architecture must be able to draw upon and interpret historical datasets which stretch way, way back into storage archives.
What’s more, this data must be calibrated to allow for contemporary standards of quality. A study conducted by the National Hurricane Center found that a three day forecast in 2017 is more accurate than a two day forecast carried out in 2007, and of comparable accuracy to a one day forecast from 1997. Computing systems must be able to approach such historical data appropriately to factor these quality shifts into current forecasts.
Secondly, the nature of meteorology requires almost constant visualization. While other fields my utilize data visualization as an ad hoc tool to enhance understanding, weather forecasting relies on high quality visualization as standard when communicating complex data to those who need to use it. This, in turn, requires an extremely high level of computing power.
Thirdly, and perhaps most importantly, accurate weather forecasts require all weather centers and data collection points to be online simultaneously, transmitting their information to a central hub for interpretation. As weather forecasts – even local ones – tend to cover a statistically significant area, serious processing muscle is required if this is to be implemented effectively; something which only HPC can provide.
A Growing International Market
The importance of accurate weather forecasting in business, in governance and in society is not limited to the USA. Across the world, we are seeing meteorologists turn to Big Data analytics and private sector assistance as they try to craft genuine value for the millions of people who rely on them.
In Australia, government-backed corporate entities are using the power of Big Data weather forecasting to assist the work of agriculturalists in the country. In Japan, statistics show that increasing numbers of businesses are opting to spend money – in ever greater quantities – to secure the most accurate weather forecasting data.
This is a trend which is showing no sign of abating. As humans, our ties with the weather systems that affect our day to day existences are ingrained deep into our collective soul, into our mythologies and into our history. Big Data analytics merely represents the next step in the ongoing process of understanding our place on this planet.