With more and more devices running in every home, it is becoming increasingly important to collect and manage all of the data that is available. Most people have no idea just how much data is currently being collected in their homes. But as the future arrives, almost every home will need to aggregate and assess data in order to make informed decisions and take informed actions. When that time arrives for you, you will need a “plug and play” residential data hub. Such devices will become an instrumental part of transforming your household into an efficient information processing system.
Currently, data is collected on your utility usage (e.g., electricity, water, Internet data usage, thermostat settings, etc). But few people realize that future homes will be collecting enormous amounts of data. We (at the Olsen residence and at Lobo Strategies) have exploited many of the new technologies that are part of the Internet of Things (IoT). Through this experience, it is apparent just how much data is now available. We are collecting data about where out family and team members are located. We are collecting data on the physical environment throughout our buildings – including temperature and occupancy. We are collecting information on the internal and external network resources being used by “the team.” And the amount of data being collected today will be dwarfed by the amount data that will be collected in the next few years.
The Necessity Of Residential Data Hubs
Over the past six months, we have been assembling a huge portfolio of data sources.
- We use our DNS server logs and firewall logs to collects access-related data.
- The Home Assistant platform collects data about all of our IoT devices. [Note: In the past month, we’ve begun consolidating all of our IoT data into a TICK platform.]
- Starting this week, we are now using router data to optimize bandwidth consumption.
While it is possible to manage each of these sources, it is taking quite a bit of “integration” (measured in many labor hours) to assemble and analyze this data. But we are now taking steps to assemble all of this data for easy analysis and decision-making
Consolidating Router Data
Our ISP put us in a box: they offered us an Internet “data only” package at a seriously reduced price. But buried within the contract were express limits on bandwidth. [Note: Our recent experience has taught us that our current ISP is not a partner; they are simply a service provider. Indeed, we have learned that we will treat them as such in the future.] Due to their onerous actions, we are now on a needed content diet. And as of the beginning of the week, we have taken the needed steps to stay within the “hidden” limits that our ISP imposed.
Fortunately, our network architect (i.e., our beloved CTO) found the root cause of our excessive usage. He noted the recent changes approved by the premise CAB (i.e., our CTO’s beloved wife). And then he correlated this with the DNS log data that identified a likely source of our excess usage. This solved the immediate problem. But what about the irreversible corrective action?
And as of yesterday, we’ve also taken the steps needed for ongoing traffic analysis.
- We’ve exploited our premise network decisions. We normally use residential-grade equipment in our remote locations. In candor, the hardware is comparable to its pricier, enterprise brethren. But the software has always suffered. Fortunately, we’ve used DD-WRT in any premise location. By doing this, we had a platform that we could build upon.
- The network team deployed remote access tools (i.e., ssh and samba) to all of our premise routers.
- A solid-state disk drive was formatted and then added to the router’s USB 3.0 port. [Note: We decided to use a non-journaled filesystem to limit excessive read/writes of the journal itself.]
- Once the hardware was installed, we deployed YAMon on the premise router.
- After configuring the router and YAMon software, we began long-term data collection.
Next Steps
While the new network data collection is very necessary, it is not a solution to the larger problem. Specifically, it is adding yet another data source (i.e., YADS). So what is now needed is a real nexus for all of the disparate data sources. We truly need a residential data hub. I need to stitch together the DNS data, the router data, and the IoT data into a single, consolidated system with robust out-of-the-box analysis tools.
I wonder if it is time to build just such a tool – as well as launch the services that go along with the product.