Security Challenges in Distributed Big Data Systems Everyday, distributed big data systems take trillions of records and events. These systems store the most sensitive records of government, bank and hospitals. Everyday those systems are attacked by threat actors Can we avoid this? Security of big data system is most critical challenge of modern era.

Abstract

There is a harsh truth about modern big data systems good things that make a distributed system helpful are its scale and computational power of clusters, its decentralisation, and different data sources coming from different sources, like we know structured data, semi structured data, and unstructured data, which are extremely useful in some cases but challenges come with its security it is very difficult to secure them from threat actors. A single database of a company is easy to protect because it has a single access point and a smaller attack surface; we can use a firewall and other security measures. A distributed system with many nodes and many access points has a larger attack surface. A Kafka streaming pipeline takes data from thousands of different sources, and mostly the data is unfiltered and could be malicious.