Recently at HarperDB, we have been performing some proof of concept projects which demonstrate the ability to consume, process and distribute sensor data directly on the edge and in many cases directly on these actual devices. Some of the projects are with clients while there are others that we undertake ourselves for R&D. In the past month we have integrated a variety of sensors with HarperDB including GPS and OBD2 sensors for vehicles, and high performance electrical relays for industrial plants. The projects all have the same workflows and similar outcomes - collect high frequency sensor data, perform filtering and analytics directly on the edge and provide real-time pertinent events directly to end users. The edge database capabilities of HarperDB provide a platform for edge computing and analytics - allowing companies to keep up with the explosion of sensors and data.
That said, if we are being honest, most organizations while hugely interested in IoT, are struggling with how to put it in place. This is due to the fact that they are hitting some common roadblocks when it comes to gaining actionable insights from their IoT sensor data.
Data is the foundation of all modern software. Without data, what’s the point? Now, what happens when we need to move that data from one application and/or database to another? That’s where ETL tools come into play. ETL stands from Extract, Transform, Load and essentially means data migration from one system to another. Sometimes these tools are in constant operation, sitting between various applications to manage continuous data migration. Many times these tools can be over-utilized and implemented in places they really don’t belong. My vision for ETL has always been a one-time thing, used exclusively to migrate data from one system to the other, not as an integration tool. However, there are plenty of use cases when ETL is the best option. This blog addresses some existing options out there and some suggestions of how to improve your landscape.
The easy way to solve a performance problem in technology seems to be to just get a bigger, better, faster computer. That’s what I told my mom when her Chromebook died. She hated that thing, so it was probably for the best. It’s also what everyone seems to do when their cloud service isn’t running fast enough. Get a bigger server, pay for a larger instance, etc. I get it, it’s simple and low effort, but it’s also expensive. Compute cost doesn’t scale linearly; it scales exponentially, so instead of buying a super computer you could buy tons of average servers. The solution is much cheaper when parallelization can be used to solve large computing problems.