By Jesús E. Gabaldón, analytics manager, Accenture Centre of Excellence.
In recent years, analytic databases have become crucial in many areas since they allow to provide the necessary workforce in sectors like social media, marketing, Internet of Things, among many others. For instance, everyday situations like sharing information in Twitter or Facebook would not be possible without this technology.
In this area, Massively Parallel Processing (MPP) as well as HDFS and MapReduce architectures have evolved separately most of the time, and both of them are still running on different tracks. However, each one is serving different purposes. Hadoop has a long record in high performance computing but it easily becomes an inaccurate and slow tool for certain tasks. There are some SQL-on-Hadoop databases but they are still immature and quite SQL unfriendly.
« Strengthening ties between universities and private sector is absolutely necessary for fulfilling the heavy demand of the business sector »
Several private and open source development lines are running in parallel. IBM, Pivotal and Apache Foundation are permanently extending the limits of performance in all directions: in-memory, MPP and Hadoop. However, MPP databases run a completely different strategy, probably the most powerful one for strongly demanding advanced analytic purposes. In any case, most of all are progressively relying on in-memory capabilities while caching more types of operations on progressively less expensive SSD supports.
MPP analytic databases are still less known than Hadoop but it is expected that they will turn soon into a strategic area, having a strong impact on highly process demanding tasks in sectors like digital marketing, banking and security. During the next years we will likely attend to a quick evolution of database and analytic technologies fuelled by the strong business demand.
However, this demand is not only requested on hardware performance but on technology development and integration, which often suffers from the scarcity of skilled and well trained professionals. Strengthening ties and establishing collaboration and partnership programs between universities and private sector is still absolutely necessary for fulfilling the heavy demand of the business sector.