Every group has their own collection of stories. In the Judeo-Christian world, the Tower of Babel is one such story. It has come to symbolize both the error of hubris and the reality of human disharmony. Within the open source community, the story of the Cathedral and the Bazaar (a.k.a., CatB) is another such story. It symbolizes the two competing schools of software development. These schools are: 1) the centralized management of software by a priestly class (i.e., the cathedral), and the decentralized competition found in the cacophonous bazaar. In the case of computer-based collaboration, it is hard to tell whether centralized overlords or a collaborative bazaar will eventually win.
Background
When I began my career, collaboration tools were intimate. You either discussed your thoughts over the telephone, you went to someone face-to-face, or you discussed the matter in a meeting . The sum total of tools available were the memorandum, the phone, and the meeting. Yes, the corporate world did have tools like PROFS and DISOSS. But both sets of tools were hamstrung either by their clumsiness (e.g., the computer “green screens”) or by the limitations of disconnected computer networks.
By the mid-eighties, there were dozens of corporate, academic, and public sector email systems. And there were just as many collaboration tools. Even the budding Internet had many different tools (e.g., sendmail, postfix, pine, elm).
The Riotous Babel
As my early career began to blossom (in the mid-nineties), I had the privilege of leading a bright team of professionals. Our fundamental mission was the elimination of corporate waste. And much of this waste came in the form of technological redundancy. So we consolidated from thirteen (13) different email clients to a single client. And we went from six (6) different email backbones to one backbone. At first, we chose to use proprietary tools to accomplish these consolidations. But over time, we moved towards more open protocols (like SMTP, X.500, and XMPP).
Since then, collaboration tools have moved from email and groupware tools (e.g., Lotus Notes) to web-based email and collaboration tools (e.g., Exchange and Confluence/Jira). Then the industry moved to “next-generation” web tools like Slack and even Discord. All of these “waves” of technology had one thing in common: they were managed by a centralized group of professionals who had arcane knowledge AND sufficient funding. Many of these tools relied upon open source components. But in almost every case, the total software solution had some “secret sauce” that ensured dominance through proprietary intellectual property.
The Times, They Are A Changing
Over the past few years, a new kind of collaboration tool has begun to emerge: the decentralized and loosely coupled system. The foremost tool of this kind is Matrix (and clients like Riot). In this model, messages flow between decentralized servers. Data sent between these servers is encrypted. And the set of data transferred between these servers is determined by the “interests” of local accounts/users. Currently, the directory for this network is centralized. There is a comprehensive ‘room’ directory at https://vector.im. But work is underway to build a truly decentralized authentication and directory system.
My Next Steps
One of the most exciting things about having a lab is that you get to experiment with new and innovative technologies. So when Franck Nijhof decided to add a Matrix server into the Hass.io Docker infrastructure, I leaped at the chance to experiment. So as of last night, I added a Matrix instance to my Home Assistant system. After a few hours, I am quite confident that we will see Matrix (or a similar tool) emerge as an important part of the next wave of IoT infrastructure. But until then, I am thrilled that I can blend my past and my future – and do it through a collaborative bazaar.
With more and more devices running in every home, it is becoming increasingly important to collect and manage all of the data that is available. Most people have no idea just how much data is currently being collected in their homes. But as the future arrives, almost every home will need to aggregate and assess data in order to make informed decisions and take informed actions. When that time arrives for you, you will need a “plug and play” residential data hub. Such devices will become an instrumental part of transforming your household into an efficient information processing system.
Currently, data is collected on your utility usage (e.g., electricity, water, Internet data usage, thermostat settings, etc). But few people realize that future homes will be collecting enormous amounts of data. We (at the Olsen residence and at Lobo Strategies) have exploited many of the new technologies that are part of the Internet of Things (IoT). Through this experience, it is apparent just how much data is now available. We are collecting data about where out family and team members are located. We are collecting data on the physical environment throughout our buildings – including temperature and occupancy. We are collecting information on the internal and external network resources being used by “the team.” And the amount of data being collected today will be dwarfed by the amount data that will be collected in the next few years.
The Necessity Of Residential Data Hubs
Over the past six months, we have been assembling a huge portfolio of data sources.
We use our DNS server logs and firewall logs to collects access-related data.
Starting this week, we are now using router data to optimize bandwidth consumption.
While it is possible to manage each of these sources, it is taking quite a bit of “integration” (measured in many labor hours) to assemble and analyze this data. But we are now taking steps to assemble all of this data for easy analysis and decision-making
Consolidating Router Data
Our ISP put us in a box: they offered us an Internet “data only” package at a seriously reduced price. But buried within the contract were express limits on bandwidth. [Note: Our recent experience has taught us that our current ISP is not a partner; they are simply a service provider. Indeed, we have learned that we will treat them as such in the future.] Due to their onerous actions, we are now on a needed content diet. And as of the beginning of the week, we have taken the needed steps to stay within the “hidden” limits that our ISP imposed.
Fortunately, our network architect (i.e., our beloved CTO) found the root cause of our excessive usage. He noted the recent changes approved by the premise CAB (i.e., our CTO’s beloved wife). And then he correlated this with the DNS log data that identified a likely source of our excess usage. This solved the immediate problem. But what about the irreversible corrective action?
And as of yesterday, we’ve also taken the steps needed for ongoing traffic analysis.
We’ve exploited our premise network decisions. We normally use residential-grade equipment in our remote locations. In candor, the hardware is comparable to its pricier, enterprise brethren. But the software has always suffered. Fortunately, we’ve used DD-WRT in any premise location. By doing this, we had a platform that we could build upon.
The network team deployed remote access tools (i.e., ssh and samba) to all of our premise routers.
A solid-state disk drive was formatted and then added to the router’s USB 3.0 port. [Note: We decided to use a non-journaled filesystem to limit excessive read/writes of the journal itself.]
Once the hardware was installed, we deployed YAMon on the premise router.
While the new network data collection is very necessary, it is not a solution to the larger problem. Specifically, it is adding yet another data source (i.e., YADS). So what is now needed is a real nexus for all of the disparate data sources. We truly need a residential data hub. I need to stitch together the DNS data, the router data, and the IoT data into a single, consolidated system with robust out-of-the-box analysis tools.
I wonder if it is time to build just such a tool – as well as launch the services that go along with the product.
I love it when I can blend my passion (for technology) and my training (in economics). Over the past six weeks, I’ve been doing just that – as I’ve tried to constrain household Internet usage. Six weeks ago, we began a voyage that has been years in the making: we’ve finally given ourselves a ‘broadband haircut’. And the keys to our (hopeful) success have been research, data collection, and data analysis.
Background
We have been paying far too much for broadband data services. And we’ve been doing this for far too many years. For us, our broadband voyage started with unlimited plans. Unlike most people, I’ve spent many years in the telecom business. And so I’ve been very fortunate to pay little (or nothing) for my wireless usage. At the same time, most household broadband was priced based upon bandwidth and not total usage. So we have always made our decisions based upon how much peak data we required at any given point in time.
But things are changing – for myself and for the industry.
First, I no longer work for a telecom. Instead, I work for myself as an independent consultant. So I must buy wireless usage in the “open” marketplace. [Note: The wireless market is only “open” because it is run by an oligopoly and not by a monopoly.]
Second, things have changed in the fixed broadband marketplace. Specifically, sanctioned, local access “monopolies” are losing market – and revenue. There is ample evidence to unequivocally state that cable companies charge too much for their services. For many years, they could charge whatever they wanted as long as they kept the local franchise in a particular municipality. But as competition has grown – mostly due to new technologies – so has the eventual downward pressure on cable revenues.
Starting a few years ago, cable companies started to treat their fixed broadband customers just as wireless operators have treated their mobile customers. Specifically, they started to impose data caps. But for many long-term customers, they just kept paying the old (and outrageously high) prices for “unlimited” services.
“But the times, they are a changin’.”
Cord Cutting Has Increased Pressure
As more and more content delivery channels are opening up, more customers are starting to see that they are paying far too much for things that they don’t really want or need. How many times have you wondered what each of the ESPN channels is costing you? Or have you ever wondered if the H&G DIY shows are worth the price that you pay for them?
Many people have been feeling the way that you must feel. And for some, the feelings of abuse are intolerable. Bundling and price duress have infuriated many customers. Some of those customers have been fortunate to switch operators – if others are available in their area. Some customers have just cut the cord to bundled TV altogether.
And this consumer dissatisfaction has led to dissatisfaction in the board rooms of most telecom companies. But instead of reaching out to under-served customers and developing new products and new markets (both domestic and overseas), most telecom executives are looking for increases in “wallet share”; they are trying to bundle more services to increase their revenue. Unfortunately, the domestic markets are pretty much tapped out. “Peak cable” is upon most operators.
Nevertheless, some boards think that punishing their customers is the best means of revenue retention. Rather than switching to new products and new services, some operators have put debilitating caps on their customers in the hopes that they can squeeze a few more dollars from people that are already sick and tired of being squeezed. The result will be an even further erosion of confidence and trust in these corporations.
Making It Personal
Six weeks ago, we decided that it was time to cut the cord. We’ve been planning this for eighteen months. However, we had a contract that we needed to honor. But the instant that we dropped off our set top devices at Comcast, they brought out their real deals. In a matter of moments, we had gone from $125 per month (w/o fees) to $50 per month (w/o fees). So we took that deal – for one year. After all, we would be getting almost the same bandwidth for a tremendously reduced price. Ain’t competition grand?
But like most people, we didn’t know how much data we used while we were on an ‘unlimited’ plan. And in fairness, we didn’t care – until we started to see just how much data we were using. Bottom line: Once we had to pay for total consumption (and not just for peak consumption), we started to look at everything that would spin the consumption ‘meter’. And when we got the first email from Comcast indicating that we had exceeded their artificial, one terabyte (per month) cap [that was buried somewhere deep within the new contract], we began a frantic search for ‘heavy hitters’.
Make Decisions Based Upon Data
DNS Data
Our hunt for high-bandwidth consumers began in earnest. And I had a pretty good idea about where to start. First, I upped my bet on ad blocking. Most ad blockers block content after it has arrived at your device. Fortunately, my Pi-hole was blocking ads before they were downloaded. At the same time, I was collecting information on DNS queries and blocked requests. So I could at least find some evidence of who was using our bandwidth.
Pi-hole Data: Biggest Ad Conveyors
After a few minutes of viewing reports, I noted that our new content streaming service might be the culprit. But when we cut the cord on cable TV, we had switched to YouTube TV (YTTV) on a new Roku device. And when I saw that device on the ‘big hitter’ list, I knew to dive deeper. I spent a few too many hours ensuring that my new Roku would not be downloading ad content. And after a few attempts, I’ve finally gotten the Pi-hole to block most of the new advertising sources. After all, why would I want to pay traffic fees for something that I didn’t even want!
The Price Of Freedom Is Eternal Vigilance
As is often the case, the first solution did not solve the real problem. Like President G.W. Bush in Gulf War II, I had prematurely declared success. So I started to look deeper. It would have helped if I had detailed data on just which devices (and clients) were using what amounts of bandwidth. But I didn’t have that data. At least, not then. Nevertheless, I had a sneaking suspicion that the real culprit was still the new content streamer.
DD-WRT Daily Usage
After a whole lot of digging through Reddit, I learned that my new Roku remote did not actually shut off the Roku. Rather, their ‘power’ button only turned off the television set. And in the case of YouTube TV, the app just kept running. Fundamentally, we were using the Roku remote to turn the TV off at night – while the Roku device itself kept merrily consuming our data on a 7×24 basis.
The solution was simple: we had to turn off YouTube TV when we turned off the TV. It isn’t hard to do. But remembering to do it would be a challenge. After all, old habits do die hard. So I took a piece of tech from the electrical monopoly (ConEd) to solve a problem with the rapacious Internet provider. A few months ago, we had an energy audit done. And as part of that audit, we got a couple of TrickleStar power strips. I re-purposed one of those strips so that when the TV was turned off, the Roku would be turned off as well.
What’s Next?
Now that we have solved that problem, I really do need to have better visibility on those things that can affect our monthly bill. Indeed, the self-imposed ‘broadband haircut’ is something that I must do all of the time. Consequently, I need to know which devices and applications are using just how much data. The stock firmware from Netgear provides no such information. Fortunately, I’m not running stock firmware. By using DD-WRT, I do have the ability to collect and save usage data.
Economics kicked off this effort. Data analysis informed and directed this effort. With a modest investment (i.e., Pi-hole, DD-WRT, an SSD drive, and a little ingenuity), I hope to save over a thousand dollars every year. And I am not alone. More and more people will demand a change from their operators – or they will abandon their operators altogether.
If you want to perform a similar ‘broadband haircut’, the steps are easier than they used to be. But they are still more difficult than they should be. But there is one clear piece of advice that I would offer: start planning your cable exit strategy.
When I graduated from college (over three-and-a-half decades ago), I had an eclectic mix of skills. I obtained degrees in economics and political sciences. But I spent a lot of my personal time working on building computers and writing computer programs. I also spent a lot of my class time learning about econometrics – that is, the representation of economic systems in mathematical/statistical models. While studying, I began using SPSS to analyze time series data.
Commercial Tools (and IP) Ruled
When I started my first job, I used SPSS for all sorts of statistical studies. In particular, I built financial models for the United States Air Force so that they could forecast future spending on the Joint Cruise Missile program. But within a few years, the SPSS tool was superseded by a new program out of Cary, NC. That program was the Statistical Analysis System (a.k.a., SAS). And I have used SAS ever since.
At first, I used the tool as a very fancy summation engine and report generator. It even served as the linchpin of a test-bed generation system that I built for a major telecommunications company. In the nineties, I began using SAS for time series data analysis. In particular, we piped CPU statistics (in the form of RMF and SMF data) into SAS-based performance tools.
Open Source Tools Enter The Fray
As the years progressed, my roles changed and my use of SAS (and time series data) began to wane. But in the past decade, I started using time series data analysis tools to once again conduct capacity and performance studies. At a major financial institution, we collected system data from both Windows and Unix systems throughout the company. And we used this data to build forecasts for future infrastructure acquisitions.
Yes, we continued to use SAS. But we also began to use tools like R. R became a pivotal tool in most universities. But many businesses still used SAS for their “big iron” systems. At the same time, many companies moved from SAS to Microsoft-based tools (including MS Excel and its pivot tables).
TICK Seizes Time Series Data Crown
Over the past few years, “stack-oriented” tools have emerged as the next “new thing” in data centers. [Note: Stacks are like clouds; they are everywhere and they are impossible to define simply.] Most corporations have someone’s “stack” running their business – whether it be Amazon AWS, Microsoft Azure, Docker, Kubernetes, or a plethora of other tools. And most commercial ventures are choosing hybrid stacks (with commercial and open source components).
And the migration towards “stacks” for execution is encouraging the migration to “stacks” for analysis. Indeed, the entire shift towards NoSQL databases is being paired with a shift towards time series databases. Today, one of the hottest “stacks” for analysis is TICK (i.e., Telegraf, InfluxDB, Chronograf, and Kapacitor).
TICK Stack @ Home
Like most projects, I stumbled onto the TICK stack. I use Home Assistant to manage a plethora of IoT devices. And as the device portfolio has grown, my need for monitoring these devices has also increased. A few months ago, I noted that an InfluxDB add-on could be found for HassIO. So I installed the add-on and started collecting information about my Home Assistant installation.
Unfortunately, the data that I collected began to exceed my capacity to store the data on the SD card that I had in my Raspberry Pi. So after running the system for a few weeks, I decided to turn the data collection off – at least until I solved some architectural problems. And so the TICK stack went on the back burner.
I had solved a bunch of other IoT issues last week. So this week, I decided to focus on getting the TICK stack operational within the office. After careful consideration, I concluded that the test cases for monitoring would be a Windows/Intel server, a Windows laptop, my Pi-hole server, and my Home Assistant instance.
Since I was working with my existing asset inventory, I decided to host the key services (or daemons) on my Windows server. So I installed Chronograf, InfluxDB, and Kapicitor onto that system. Since there was no native support for a Windows service install, I used the Non-Sucking Service Manager (NSSM) to create the relevant Windows services. At the same time, I installed Telegraf onto a variety of desktops, laptops, and Linux systems. After only a few hiccups, I finally got everything deployed and functioning automatically. Phew!
Bottom Line
I implemented the TICK components onto a large number of systems. And I am now collecting all sorts of time series data from across the network. As I think about what I’ve done in the past few days, I realize just how important it is to stand on the shoulders of others. A few decades ago, I would have paid thousands of dollars to collect and analyze this data. Today, I can do it with only a minimal investment of time and materials. And given these minimal costs, it will be possible to use these findings for almost every DevOps engagement that arises.
In addition to the first stage of the malware, the threat actors included the following “plugins”:
‘htpx’ – a module that redirects and inspects the contents of unencrypted Web traffic passing through compromised devices.
‘ndbr’ – a multifunctional secure shell (SSH) utility that allows
remote access to the device. It can act as an SSH client or server and
transfer files using the SCP protocol. A “dropbear” command turns the
device into an SSH server. The module can also run the nmap network port
scanning utility.
‘nm’ – a network mapping module used to perform reconnaissance from
the compromised devices. It performs a port scan and then uses the
Mikrotik Network Discovery Protocol to search for other Mikrotik devices
that could be compromised.
‘netfilter’ – a firewall management utility that can be used to block sets of network addresses.
‘portforwarding’ – a module that allows network traffic from the device to be redirected to a network specified by the attacker.
‘socks5proxy’ – a module that turns the compromised device into a
SOCKS5 virtual private network proxy server, allowing the attacker to
use it as a front for network activity. It uses no authentication and is
hardcoded to listen on TCP port 5380. There were several bugs in the
implementation of this module.
‘tcpvpn’ – a module that allows the attacker to create a Reverse-TCP
VPN on compromised devices, connecting them back to the attacker over a
virtual private network for export of data and remote command and
control.
Disaster Averted?
Fortunately, the impact of VPNFilter was blunted by the Federal Bureau of Investigation (FBI). The FBI recommended that every home user reboot their router. The FBI hoped that this would slow down infection and exploitation. It did. But it did not eliminate the threat.
In order to be reasonably safe, you must also ensure that you are on a version of router firmware that protects against VPNFilter. While many people heeded this advice, many did not. Consequently, there are thousands of routers that remain compromised. And threat actors are now using these springboards to compromise all sorts of devices within the home. This includes hubs, switches, servers, video players, lights, sensors, cameras, etc.
Long-Term Implications
Given the ubiquity of devices within the home, the need for ubiquitous (and standardized) software update mechanisms is escalating. You should absolutely protect your router as the first line of defense. But you also need to routinely update every type of device in your home.
Bottom Line
Update your router! And update it whenever there are new security patches. Period.
Only buy devices that have automatic updating capabilities. The only exception to this rule should be if/when you are an accomplished technician and you have established a plan for performing the updates manually.
Schedule periodic audits of device firmware. Years ago, I did annual battery maintenance on smoke detectors. Today, I check every device at least once a month.
Retain software backups so that you can “roll back” updates if they fail. Again, this is a good reason to spend additional money on devices that support backup/restore capabilities. The very last thing you want is a black box that you cannot control.
As the VPNFilter scope and capabilities have expanded, the importance of remediation has also increased. Don’t wait. Don’t be the slowest antelope on the savanna.
Amazon Echo devices now have a foothold in most American homes.
Voice control is the ‘holy grail’ of UI interaction. You need only look at old movies and television to see that voice is indeed king. [For example, the Robinson family used voice commands to control their robot. And Heywood Floyd used voice as his means of teaching and communicating with HAL.] Today, there are many voice assistants available on the market. These include: Amazon Alexa, Apple Siri, Google Assistant (aka Google Home), Microsoft Cortana, Nuance Nina, Samsung Bixby, and even the Voxagent Silvia. But the real leaders are only now starting to emerge from this crowded market. And as of this moment, Alexa dominance in third-party voice integration is apparent.
Apple Creates The Market
Apple was the first out-of-the-gate with the Apple Siri assistant. Siri first arrived on the iPhone and later on the iPad. But since its introduction, it is now available as part of the entire Apple i-cosystem. If you are an Apple enthusiast, Siri is on your wrist (with the watch). Siri is on your computer. And Siri is on your HomePod speaker. It is even on your earbuds. And in the past six months, we are finally starting to see some third-party integration with Siri.
Amazon Seizes The Market
Amazon used an entirely different approach to entrench its voice assistant. Rather than launch the service across all Amazon-branded products, Amazon chose to first launch a voice assistant inside a speaker. This was a clever strategy. With a fairly small investment, you could have an assistant in the room with you. Wherever you spent time, your assistant would probably be close enough for routine interactions.
This strategy did not rely upon your phone always being in your pocket. Unlike Apple, the table stakes for getting a voice assistant were relatively trivial. And more importantly, your investment was not limited to one and only one ecosystem. When the Echo Dot was released at a trivial price point (including heavy discounts), Alexa started showing up everywhere.
From the very outset, an Amazon voice assistant investment required funds for a simple speaker (and not an expensive smartphone). You could put the speaker in a room with a Samsung TV. Or you could set it in your kitchen. So as you listened to music (while cooking), you could add items to your next shopping list. And you could set the timers for all of your cooking. In short, you had a hands-free method of augmenting routine tasks. In fact, it was this integration between normal household chores coupled with the lower entry price that helped to spur consumer purchases of the Amazon Echo (and Echo Dot).
A second key feature of Amazon’s success was its open architecture. Alexa dominance was amplified as additional hardware vendors adopted the Alexa ecosystem. And the young Internet-of-Things (IoT) marketplace adopted Alexa as its first integration platform. Yes, many companies also provided Siri and Google Assistant integration. But Alexa was their first ‘target’ platform.
The reason for Alexa integration was (and is) simple: most vendors sell their products through Amazon. So vendors gained synergies with their main supplier. Unlike the Apple model, you didn’t have to go to a brick and mortar store (whether it be the Apple Store, the carriers’ stores, or even BestBuy/Target/Walmart). Nor did a vendor need to use another company’s supply chain. Instead, they could bundle the whole experience through an established sales/supply channel.
Google Arrives Late To The Party
While Apple and Amazon sparred with one another, Google jumped into the market. They doubled-down on ‘openness’ and interoperability. And at this moment, the general consensus is that the Google offering is the most open. But to date, they have not gained traction because their entry price was much higher than Amazon’s. We find this to be tremendously interesting. Google got the low price part down when they offered a $20-$30 video streamer.
But with the broader household assistant, Google focused first upon the phone (choosing to fight with Apple) rather than a hands-free device that everyone could use throughout the house. And rather than follow the pricing model that they adopted with the Chromecast, Google chose to offer a more capable (and more expensive) speaker product. So while they used one part of the Amazon formula (i.e., interoperability), they avoided the price-sensitive part of the formula.
Furthermore, Google could not offer synergies with the supply chain. Consequently, Google still remains a third-place contender. For them to leap back into a more prominent position, they will either have to beat ‘all-comers’ on price or they will have to offer something really innovative that the other vendors haven’t yet delivered.
Alexa Dominance
Amazon dominance in third-party voice integration is apparent. Not only can you use Alexa on your Amazon ‘speakers’, you can use it on third-party speakers (like Sonos). You can launch actions on your phone and on your computer. And these days, you can use it with your thermostat, your light bulbs, your power sockets, your garage door, your blinds, and even your oven. In my case, I just finished integrating Alexa with Hue lights and with an ecobee thermostat.
Bottom Line
Market dominance is very fleeting. I remember when IBM was the dominant technology provider. After IBM, Microsoft dominated the computer market. At that time, companies like IBM, HP, and Sun dominated the server market. And dominance in the software market is just as fleeting. Without continually focusing on new and emerging trends, leadership can devolve back into a competitive melee, followed by the obsolescence of the leader. Indeed, this has been the rule as dominant players have struggled to maintain existing revenue streams while trying to remain innovative.
Apple is approaching the same point of transition. Their dominance of the phone market is slowly coming to an end. Unless they can pivot to something truly innovative, they may suffer the same fate as IBM, Sun, HP, Dell, Microsoft, and a host of others.
Google may be facing the same fate – though this is far less certain. Since Google’s main source of revenue is ‘search-related’ adverstising, they may see some sniping around the edges (e.g., Bing, DuckDuckGo, etc). But there is no serious challenge to their core business – at this time.
And Amazon is in a similar position: their core revenue is the supply chain ‘tax’ that they impose upon retail sales. So they may not see the same impact on their voice-related offerings. But they dare not rest upon their laurels. In candor, the Amazon position is far more appealing than the Google position. The Amazon model relies upon other companies building products that Amazon can sell. So interoperability will always be a part of any product that Amazon brands – including voice assistants.
Only time will sort out the winners and losers. And I daresay that there is room enough for multiple ‘winners’ in this space. But for me, I am now making all of my personal and business investments based upon the continued dominance of Alexa.
This weekend, we took another step in our home automation quest. We have used smart switches (for lamps), smart thermostats, smart music, smart cars, and even smart timers. But until Saturday, we did not have any smart lights, per se. On Saturday, we bought some Philips Hue lights (and the associated hub). That means that we now have Ethernet (i.e., wired) devices, Wifi devices, and now Zigbee devices.
Is this a big deal? The answer to that is somewhat nuanced. We’ve had smart home puzzle pieces for a while. And we almost bought a Z-Wave infrastructure to put smart switches in place. But the age of our house makes this impractical. [We don’t have neutral wires on any switches in the house. And the price to refurbish these switches would be prohibitive.] So our home automation quest stalled. But on Saturday, I could take it no more. When we went out on errands, we stopped and picked up five (5) Hue lights.
Just Add Lights
The installation and setup was simple. It took almost no time to get everything installed and paired. And within a little more than an hour, we had functioning lights in the second floor hallway and in our master bedroom. Over the next year, we can start to populate the various ceiling fans in the house. I figure that we can do this whenever we need to replace the incandescent bulbs that are currently installed. Given our current pace of replacement, I’m figuring that it will take a year or so to retrofit the house.
After getting everything installed, I started to make an inventory of our various smart home investments. As of today, we have the following pieces:
Current “On-Premises” Infrastructure
Today, we have so many physical (and logical) pieces in our home automation puzzle:
Network: Cisco network switch, Cisco VPN appliance, Netgear router, NordVPN proxy, Raspberry Pi ad blocking, Raspberry Pi DNS
Print: Hewlett-Packard printer
Entertainment: Plex media server (on PC desktop), Roku media player, Samsung TV, Silicon Dust HDHomeRun player
Storage: Synology storage, WD MyCloud storage
IoT: Amazon Echo Dot speakers, Huawei sensor/camera (on surplus phone), Kia Soul, Personal location / presence (on personal phones), Philips Hue lights, Raspberry Pi home automation appliance, TP-Link Kasa switches, WeightGURUS scale
Current “Off-Premises” Services
While we have lots of smart pieces in the house, we also have more than a few external cloud services providers. In most of these cases, these services allow us to extend “access” beyond the confines of our network. Our current list of services includes:
Lobostrategies Business: Bluehost, GoDaddy
Olsen Personal: Amazon Alexa, Dropbox, Google Drive, Google GMail, Home Assistant cloud, IFTTT cloud, Plex cloud, Pushbullet cloud, TP-Link Kasa cloud, WD MyCloud
So after adding yet another home automation “category” to the premises, we learned an important lesson: external access requires a measure of trust – and diligence. If you aren’t willing to secure your devices, then you must accept the consequences of an electronic intrusion.
In my third installment of this series about DNS security, I am focusing on how we can audit all DNS requests made in our network. In the second installment, I focused on secure communications between DNS servers. And I highlighted the value of DNSSEC and the potential of DNS-Over-HTTPS (DOH). I outlined some of the changes that we’ve made – including the deployment of DNSMasqd on our router and on our Pi-hole. Today, I will focus upon ensuring that end-to-end auditing is possible – if desired.
As noted previously, we upgraded our router (i.e., a Netgear R8000 X6) from stock firmware to DD-WRT. We did this for numerous reasons. Chief among these reasons was the ability to set up an OpenVPN tunnel from our network out to our VPN providers’ VPN endpoints. But a close second was a desire to run DNSMasqd on the router. Since we haven’t chosen to move DHCP functions to the Pi, we wanted a DHCP service that would have better capabilities. More importantly, we wanted to update DHCP option #6 so that we could control what name servers our clients would use. In our case, we knew that all clients would use our Pi-hole as their name server.
After we realized that DNS options on the admin panel didn’t apply to DHCP clients, we figured out how to set name servers for all of our DHCP clients. Once done, all clients began direct interaction with the Pi-hole. [They had previously used the router as a DNS proxy to our Pi-hole system.]
But as is often the case, there were unforeseen consequences. Specifically, reverse lookups (for static address) failed. This meant that DNS security would suffer because we couldn’t correlate elements across the entire request chain. We could have moved dhcpd to the Pi-hole. But we wanted to have a DNS fall-back – just in case the Pi-hole failed. So we changed our processes for assigning static addresses. Now, we will add them to the router as well as to the /etc/hosts file on the Pi-hole. Once implemented, we had clear visibility between request origination and request fulfillment. [Note: We will probably turn this off as it defeats the very anonymity that we are trying to establish. But that is for another day.]
So what’s left? In the first two articles, we set up DNS authentication wherever possible. We also established DNS payload encryption – using DNS-Over-HTTPS. Now we have a means of authenticating DNS server communications. And we have an encrypted payload. But there is one last need: we have to limit the ‘data at rest’ stored by each DNS resolver.
Consider this: If we validate our connection to Google, and we encrypt the traffic between us, then Google can still look at all of the DNS queries that it receives from us. That is fine if you trust your principal DNS resolver. But what if you don’t? A more secure process would be to ask for name resolution directly, rather than through a trusted (or un-trusted) intermediary. In my next article, I will discuss how we implemented a recursive DNS resolver alongside our Pi-hole. Specifically, I’ll talk about using Unbound to limit the amount of ‘data at rest’ that we leave with any single DNS service.
Yesterday, I wrote about how Mozilla is updating Firefox to improve DNS security. But my conclusion was that Mozilla may have designed a system with a single point of failure. In fairness, this assessment was far too simplistic. Today, I want to amplify on my thoughts.
The Firefox implementation assumes something that is probably false. It assumes that most people are using Firefox. It also assumes that all Firefox users can choose an appropriate resolver/referrer to meet their specific needs. I would argue that the first assumption is patently wrong while the second assumption is altogether naive. As noted yesterday, I would have much preferred that Mozilla be more transparent about their proposed changes. Also, rather than assume that the code is open and thus reviewed, Mozilla could have asked for more extensive input. [Note: I suspect that Mozilla was transparent with a somewhat limited community. I just wish that their design had been explicitly shared.]
The real problem that I have with their approach is that it is a Firefox-only solution. No, I don’t expect Mozilla to solve this problem for their competitors. But most organizations need a broader solution that will meet everyone’s needs. Enter dnsmasq. In my case, I have implemented DNS over HTTPS (DoH) as my mechanism to improve DNS security.
I am quite fortunate: I am using a DNS server that was just updated. The Pi-hole team has just released Pi-Hole 4.0. And a DNS service provider (Cloudflare) has released an AMD and ARM DNS-over-HTTPS implementation. [Note: Their solution is in binary form. So I will add my voice to the chorus asking for the software to be published under a free/open license. Until that happens, I’m testing the system to see how it performs.]
So what am I seeing thus far?
After switching to cloudflared on Pi-hole 4.0, I ran some benchmarks. And as expected, there was quite a bit more activity on my DNS server. But the overall DNS response time (i.e., the server at 10.42.222.22) was quite acceptable. I want to get about a week’s worth of data. But at this moment, I am very hopeful that the software will maintain acceptable performance levels.
So what should you do? If you’re bold, then use a test system and try it out for yourself. Or keep close tabs on folks who are testing this technology. At the current time, I am thrilled to close down yet another vector of surveillance.
I recently gave the keynote address at a Kansas City symposium on cloud computing. So when I saw this article, my interest was piqued. The author believes that cloud computing is devouring itself. Um, alright. I guess I can buy the premise. But I think the author may be too absorbed with the technical layers of the infrastructure rather than focusing on the real transformation.
From my vantage point (as a consumer and as a technologist), I see this wave of transformation as being more about business and less about technology. Maybe even more important is the fact that tech-centric companies are moving from a capex-centric budgeting model to an opex-centric budgeting model.
What does this imply? First, it means that businesses are seeing IT as fundamental to their future. But while it is fundamental, it is no longer a strategic differentiator. Indeed, most companies are frustrated after spending thousands of dollars (or millions or billions) on IT capital without any real strategic benefit. The cloud allows a company to abandon capex investments in favor of opex outlays. This means that companies can be nimble – assuming that the cloud vendors are nimble enough to meet their customers’ needs.
But the larger and more fundamental implication of this capex/opex migration is that businesses are moving to an even more short-term focus. With a capex focus, companies were thinking about tax advantages and long-term viability. With an opex focus, they are looking at short-term return to the owners/shareholders. This does ensure that a company is competitive as long as it is agile.
But are we abandoning the future by focusing even more acutely on the short term? I am not certain. It does mean that on a micro-level, individual companies are trusting their future to their partners. On a macro-level, we may not be losing future investment streams to other countries. But we are pushing capital investment into the hands of a smaller number of companies that will be managing public or private clouds on behalf of other companies.
If this is indeed what is happening, then the savvy investor will need to look at hosting providers within the tech sector as these companies will be the real capital aggregation points. I’m looking at HP, Rackspace and Amazon being some of the biggest players in this space. And I would throw Salesforce.com into the mix as a software aggregation point.
