According to Wikipedia,“Polyglot persistence is the concept of using different data storage technologies to handle different data storage needs within a given software application.” James Serra, in his blog writes, “Polyglot Persistence is a fancy term to mean that when storing data, it is best to use multiple data storage technologies, chosen based upon the way data is being used by individual applications or components of a single application. Different kinds of data are best dealt with different data stores. “
Recently at HarperDB, we have been performing some proof of concept projects which demonstrate the ability to consume, process and distribute sensor data directly on the edge and in many cases directly on these actual devices. Some of the projects are with clients while there are others that we undertake ourselves for R&D. In the past month we have integrated a variety of sensors with HarperDB including GPS and OBD2 sensors for vehicles, and high performance electrical relays for industrial plants. The projects all have the same workflows and similar outcomes - collect high frequency sensor data, perform filtering and analytics directly on the edge and provide real-time pertinent events directly to end users. The edge database capabilities of HarperDB provide a platform for edge computing and analytics - allowing companies to keep up with the explosion of sensors and data.
That said, if we are being honest, most organizations while hugely interested in IoT, are struggling with how to put it in place. This is due to the fact that they are hitting some common roadblocks when it comes to gaining actionable insights from their IoT sensor data.
Data is the foundation of all modern software. Without data, what’s the point? Now, what happens when we need to move that data from one application and/or database to another? That’s where ETL tools come into play. ETL stands from Extract, Transform, Load and essentially means data migration from one system to another. Sometimes these tools are in constant operation, sitting between various applications to manage continuous data migration. Many times these tools can be over-utilized and implemented in places they really don’t belong. My vision for ETL has always been a one-time thing, used exclusively to migrate data from one system to the other, not as an integration tool. However, there are plenty of use cases when ETL is the best option. This blog addresses some existing options out there and some suggestions of how to improve your landscape.