Big Data
18.7. The Whole Story

Big data gives us an opportunity to gain a much deeper understanding of how many things in the world around us work. It is a new and exciting field, and even with the huge amounts of work and research happening at the moment we are barely scratching the surface of the possibilities. It has very close ties with Artificial Intelligence, as many AI systems use big data, and a lot of big data analysis is done using AI algorithms.

It is so new and influential in fact that there is now a huge demand for those with Data Science skills. Data Scientists are people with the skills and knowledge to appropriately analyse big data. With the amount of data we are collecting each day growing rapidly, it’s not surprising that this field is also rapidly growing.

But not only is there a huge demand for experts in this field, there is also demand for innovation. We are already pushing the limits of the infrastructure currently supporting big data. For example, once it is completed it is estimated that the Square Kilometre Array telescope will produce 160 terabytes of raw data every second, which is over 10 exabytes of data being produced everyday! Which is much, much more data than currently exists on the entire internet. The technology to store, let alone process, this amount of data simply does not exist right now. There is plenty of space for creative solutions to solve this problem, and other problems related to using big data.