Home Automation Dashboard – Version 3

Over the past two years, I’ve had a few iterations on my home dashboard project. All of the integrations for a “smart home” have been rather dumb in the sense that they’re just handling static transactions or act only as a new channel for taking actions. I wanted to change this and start bringing actual intelligence into my “smart” devices.

A major problem in the current smart device landscape is the amount of proprietary software and devices that are suffocating innovation and stifling the convenience and luxury that a truly “smart home” can bring to consumers/homes of the future — this means improving my standard of living without effort, not just being a novelty device (a “smart” lightbulb that can be controlled through another novelty device like Amazon Alexa).

In this vein, I’ve been connecting my devices (not just my smart devices) into a single product that enables devices to interact with each other without my intervention. This project has slowly morphed from a UI that simply displayed information and allowed on/off toggling to an actual dashboard that will take actions automatically. There’s not much special behind many of these actions at the moment but it’s a starting point.

Home UI: Version 3

In the prior two iterations of my Home UI product, I focused on two static aspects: device functionality and data collection. With V3, I’ve shifted focus to merging those two and bringing in proactive, intelligent actions and notifications.

Key features

  • Building Habits and Accomplishing More: Using my calendar, weather forecast, my entertainment preferences, and my to-do lists, the system will make scheduling suggestions to help me build positive habits or remind me to take take care of household tasks in a more timely manner. For example, the system knows that I enjoy going to the movies but also knows I enjoy doing things outdoors. The system will encourage an outdoor task if the weather is nice and suggest a movie when it’s raining/I have nothing else scheduled. Similarly, the system will suggest items from my to-do list based on their due date and priority.
  • Commute Planning: the system collects real-time traffic information from Google Maps; toll, traffic alerts (crashes, special events, construction, etc), and camera feeds from WashDOT; and road condition information, including subsurface temperatures from WashDOT, to compare against my calendar for the day and recommend a time for travelling to/from work. For example, if there’s a SeaHawks game in the evening, the system will recognize that and recommend an earlier or later departure to avoid sitting in traffic. Similarly, if I have an early meeting, the system will send me a push notification the night before to recommend setting an earlier alarm.
  • Device Event Bundling: a common use case in home automation, the system will take multiple actions across multiple devices based on a single trigger. For example: before leaving the house, I’m able to reduce my thermostat, turn off all lights, and set my security alarm with having to take each of those actions individually. This isn’t a new concept but it’s a nice implementation despite the various product types supported.
  • Neighborhood Awareness: police events around my home are pushed to me so I know when there was a burglary, car theft, or other concerning event near me. Others are stored and available in a map view.

Full List of Features

  • Pipes RTSP feeds from security cameras and save them to AWS S3 (30 days of storage for ~$1.50)
  • Detect motion in video feeds and triggers notifications
  • Push notifications for:
    • Motion detection from security cameras
    • Police events near my house
    • Traffic alerts that can impact my commute
    • To-Do list reminders and calendar reminders
  • SimpliSafe Security System integration
  • Nest thermostat API integration
  • Nest Hello doorbell camera integration
  • Police events, restaurant health inspection scores, building permit applications, and traffic information for my community are captured/plotted
  • YeeLight integration/control
  • Google Calendar integration
  • Stock price integration (for stock in my portfolio)
  • Amazon Echo Music integration (history only)
  • And a few other things I’ve shared before (such as my movie collection UI)

Hardware in Use

  • Nest thermostat
  • Nest Hello
  • Hikvision security cameras
  • SimpliSafe Alarm System
  • YeeLight light bulbs (I highly recommend these)
  • Raspberry Pi (handles some LAN things)

Software Used

The Underlying Logic for Expansion

The foundation of the system has three core components: 1) building and flattening a timeline for my persona so it knows what to recommend/do and when to recommend/do it, 2) data collection and transformation from a number of different sources, and 3) API/event handling for the devices I use (cell phone, Nest, security stuff, etc).

In order for the system to be most effective, it needs to know a bit about me – it needs data for intelligence. To enable this, I’ve integrated a ton of my day-to-day apps (calendar, note app, commute times, data from my android phone, etc.) so that it’s aware of what I need/want/plan to do. Using this, I can build a sufficient schedule on-the-fly and the system can accompany me by bringing relevant meta-data along the way.

When the persona and supplemental data are merged, higher-quality and intelligent recommendation are the result.

1984

The downside to this approach is the obvious self-inflicted 1984 “big-brother” effect. I’m putting a lot of meta-data about my routine and my lifestyle into the system to effort to encourage the system to reduce the number of small decisions I’m burdened with day-to-day. It sounds crazy just writing that out…I know this.

I see this as inevitable, though. In order for us to achieve the next level of immediacy and convenience, we’ll have to get used to the idea that the next generation of smart devices (ie the next generation of Google AI, Alexa, Siri, etc) will begin using more of the information they already know about us to improve the quality and effectiveness of the convenience we told ourselves we’d get when we purchased the current generation of these devices. Accepting this, I’m okay with sharing a small amount of additional detail alongside what I already share today into a system I control end-to-end.

What’s Next?

I’m working towards extension of the personas concept through deeper integration. I want to focus on making the outputs surfaced to me higher value (ie more intelligent alerting and suggesting) while also concerning myself with less information.

In parallel, I want to continue shifting the system from primarily smart home to an intelligent assistance and entertainment console. I also see this evolving into hardware integrated into the house.

Logging router traffic and other changes

It’s been a slow couple months between the holidays, travelling, and work.  I did manage to accomplish a few things with the Home Dashboard project, though.  I redesigned the UI to move away from an exclusively mobile interface as the amount of data and type of data I’m including in the project now simply don’t all make sense to squeeze into a mobile UI.  Sometime in early January, the system broke the 2 millionth record milestone — I’m unsure what I’ll do with some of the data I’m collecting at this point but I’ve learned a lot through collecting it and I’m sure I’ll learn more analyzing it at some point in the future.
_statsThis brings the list of events I’m collecting to:

  1. Indoor temperature and humidity
  2. Amazon Echo music events
  3. DirecTV program and DVR information
  4. Cell phone location and status details
  5. Local fire and police emergency events
  6. Home lights and other Wink hub events
  7. …and now home network information

Analyzing home network information

The biggest change was the addition of network event logging.  After seeing that a foreign IP was accessing my LAN, I started logging each request to or from my home network until I was sure I had fixed the vulnerability.  After that, I found the information interesting so I just kept logging it.  For example, I was able to discover that an app on my phone was making repeated calls (~2,000 per day) to an app monitoring service (New Relic) which wasn’t doing my phone’s battery life any favors.
_routerlogs
_routerlogsbydevice

Collecting additional phone data

After launching Location to HTTP, I’ve been tinkering with additional data collection from my cellphone such as Bluetooth, WiFi, GPS, battery, and screen status.  After collecting this information for a month or so, here are some useless data points from the most recent 30 days:

  • I’ve actively used my phone for 104.5 hours (3.5hrs per day – I need to cut back on the work email…)
  • Average battery level: 61%
  • Average free memory: 516MB (12.6%)
  • Average location accuracy: +/-31FT
  • Average altitude: 154FT
  • Average speed: 1MPH
  • Average uptime: 153.3HRs
  • Maximum uptime: 437.4HRs

_phonestats
I also improved the location history mapping to show different color map markers depending on the age of the record and phone details at the time the record was made:


 

Improved home climate visuals

I added some simple graphs of MoM temperature and humidity and also updated the heat-mappings for the daily climate information.  These are a bit easier to read that those I had in the previous UI.  It’s interesting to see the effectiveness of thermostat automation and our daily routines.

More detailed emergency event

Lastly, I expanded on the emergency event information to surface the top event types and the total number of events by type:


 

Collecting and Handling 911 Event Data

Seattle has a pretty awesome approach to data availability and transparency through data.Seattle.gov.  The city has thousands of data sets available (from in-car police video records to land zoning to real-time emergency feeds) and Socrata, a Seattle-based company, has worked with the city (and many other cities) to allow developers to engage this data however they like.  I spent some time playing around with some of the data sets and decided it’d be nice to know when police and fire events occurred near my apartment.
I setup a script to pull the fire and police calls for events occurring within 500 meters of my apartment and started storing them into a local database (Socrata makes it so simple – amazing work by that team).  While reading it from the API, I check the proximity of the event to my address and also the type of event (burglary, suspicious person, traffic stop, etc) and trigger emails for the ones I really want to know about (such as a near by rape, burglary, shooting, vehicle theft, etc).  I decided to store all events, even traffic stops, just because.  I may find a use for it later – who knows…
After I’ve scrubbed through and sent any notifications for events I care about, I display the data in a simple table in my existing home dashboard and highlight red any rows for events which are within certain square area of my apartment.
911table.png
To add a nice visual, I also plot the most recent events on a map using the Google Maps API.  Police events are noted with blue pins, fire events are noted with red pins:
911map
Clicking the pins will give us some details about the event:
911detail.png
All told, it was a pretty simple project which helped me gain some experience with the Google Maps API and also poke around with some of the data the city provides.  I’m sure I’ll be doing a bit more of that in the future.  These two projects have been integrated back into my home automation dashboard so I can continue to build on them in the future.

Data Visualization and Demo

As mentioned previously, my goal wasn’t to just create a home controller/dashboard but to also collect as much data as possible while doing so.  So tonight, I started playing around with a few different visualizations of the data I’ve collected thus far.  It took a few hours but I’m satisfied with the current state.
I’m doing simple dumps of the most recent music played by my Amazon Echo; most recent programming watched via DirecTv; visualizing the daily average, minimum, and maximum temperature and humidity levels in my apartment; visualizing by hour of day the average, min, and max temperature for the current month vs the previous month; breaking down the amount of time I spend at home by day of week (and telling on myself that I like to leave work early on Fridays :)); and visualizing my TV watching habits by hour of day and day of week.
I recorded a video of this all and also included the DirecTv control demo at the end.

A month of tinkering

New Design

The original design wasn’t “clean” feeling and didn’t function too well on mobile or even tablet displays.  I changed that up a bit and the new design has a lot of transparent divs, bokeh background images, and some jquery to make actions a bit smoother.
new ui

Wink Integration Improvements

The initial integration of the Wink API wasn’t that great.  I was using PHP to trigger shell scripts which would then make the API call – quite messy and had several opportunities for failure.  This method also made a new request for a bearer token each time an action was taken so if I turned on three lights, I requested three unique tokens from the API.  I’ve since cleaned that up and now use a single token per session and the API calls are all made in a single PHP file.  This still isn’t the cleanest or safest way to do this but it works for my usecase.
While doing this, I also added the ability to dim some lights (such as the kitchen light which we leave on during the night).  The next step is to fetch the current state of the lights so that we can eliminate the on/off option and simply toggle it.  The problem with that is that the Wink Hub struggles to maintain accurate states for its devices.
dim.PNG
Gaining my Wink bearer token for the session:

<?php
$ch_token = curl_init();
curl_setopt($ch_token, CURLOPT_URL, "https://api.wink.com/oauth2/token");
curl_setopt($ch_token, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch_token, CURLOPT_HEADER, FALSE);
curl_setopt($ch_token, CURLOPT_POST, TRUE);
curl_setopt($ch_token, CURLOPT_POSTFIELDS, "{
  \"client_id\": \"<insert_here>\",
  \"client_secret\": \"<insert_here>\",
  \"username\": \"<insert_here>\",
  \"password\": \"<insert_here>\",
  \"grant_type\": \"password\"
}");
curl_setopt($ch_token, CURLOPT_HTTPHEADER, array(
  "Content-Type: application/json"
));
$ch_token_response = curl_exec($ch_token);
curl_close($ch_token);
$ch_token_json = json_decode($ch_token_response, true);
$bearer_token=$ch_token_json['access_token'];
?>

Wink Control:

<?php
$device_id=$_GET["device_id"];
$new_state=$_GET["new_state"];
$bearer_token=$_GET["bearer_token"];
if(is_numeric($new_state)){$action="brightness";} else {$action="powered";}
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "https://api.wink.com/light_bulbs/".$device_id."/desired_state");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_HEADER, FALSE);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "PUT");
curl_setopt($ch, CURLOPT_POSTFIELDS, "{
  \"desired_state\": {
    \"".$action."\": ".$new_state."
  }
}");
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
  "Content-Type: application/json",
  "Authorization: Bearer ".$bearer_token.""
));
$response = curl_exec($ch);
curl_close($ch);
if($new_state=="true"){echo "Turned On"; } else if($new_state=="false") { echo "Turned Off"; } else { echo "Light Dimmed: $new_state"; }
?>

DirecTv Changes

I integrated TheMovieDB.org‘s API to pull images of the movies or shows that are currently on.  This currently works very well for movies but it often fails to find images for shows so I’ll loop back to fix that at some point in the future.  I also added in a link to view the title on IMDB for easy access.  An alarm was added to trigger a notification if the DVR is nearly full.

Mapping GPS Coordinates

Out of sheer curiosity, I decided to push my phone’s GPS coordinates to my server and plot them on a map.  I’m using the SendLocation app to push the coordinates to a script I have setup to listen to the app.  This may, perhaps, help me locate my phone one day if I ever lose it.  For now, though, it’s merely something for me to play with.  I’m also capturing things like speed so I can view when I’m in transit.  Essentially, this traces my steps.
gps

MQ2 Sensor (Gas/Smoke)

I added an MQ2 sensor to the PI and set it up to store the current state as well as send me a text message and email if it detects something.  Overall, the setup is pretty simple and certainly isn’t life-saving but does serve the goal of being able to monitor home while away.
Here’s my Python file I use which I simply schedule a cron job to ensure it’s monitoring frequently enough to be useful (note the loop of 20 and sleep of 3 seconds so it runs for the entire minute before the cron fires again):

import time, sys
import RPi.GPIO as GPIO
import MySQLdb
import smtplib
def sendemail(from_addr, to_addr_list, cc_addr_list,
              subject, message,
              login, password,
              smtpserver='smtp.gmail.com:587'):
    header  = 'From: %s\n' % from_addr
    header += 'To: %s\n' % ','.join(to_addr_list)
    header += 'Cc: %s\n' % ','.join(cc_addr_list)
    header += 'Subject: %s\n\n' % subject
    message = header + message

    server = smtplib.SMTP(smtpserver)
    server.starttls()
    server.login(login,password)
    problems = server.sendmail(from_addr, to_addr_list, message)
    server.quit()
GPIO.setmode(GPIO.BOARD)
GPIO.setup(11, GPIO.IN, pull_up_down=GPIO.PUD_DOWN)
n = 20
def action(pin):
    print 'Sensor detected action!'
    db = MySQLdb.connect("","","","" )
    cursor = db.cursor()
    cursor.execute( 'insert into mq2(event) values("Gas detected")')
    db.commit()
    db.close()
    sendemail(from_addr    = '',
          to_addr_list = ['att phone email address'],
          cc_addr_list = [''],
          subject      = 'Gas Detected',
          message      = 'Elevated levels of gas or smoke detected.',
          login        = '',
          password     = '')
    return
GPIO.add_event_detect(11, GPIO.RISING)
GPIO.add_event_callback(11, action)
try:
    while n > 0:
#        print 'alive'
#        print n
        n = n-1
        time.sleep(3)
except KeyboardInterrupt:
    GPIO.cleanup()
    sys.exit()

Other Changes

  • I’ve changed the interior camera to disable itself if anyone is home.  There’s no need for it to run if during that time and it also ensures an additional layer of security/privacy.
  • The date formats were updated to provide “friendlier” outputs.  I also added a time of day greeting for the active user – “Good morning/afternoon/evening/night”…
  • My favorite Google News RSS feeds were added to the left and a weather forecast link was added.  I also included a calendar with the eventual goal of integrating the Google Calendar API (I use Google Calendar quite heavily).  These are hidden by default in order to maintain the “clean” UI I’m going for.news.png

The Foundation

Purpose

Let’s just get it out of the way now — there’s no true practical purpose or value in doing this.  I took this on as an experiment and opportunity to learn something new.

What is it?

Using a Raspberry Pi, some sensors, and a lot of Googling with trial and error, I took my first step into custom home automation (Wikipedia).  I can control lights, DirecTv receivers, some appliances, measure indoor temperature and humidity, determine who is home, and view indoor/outdoor webcams through a single UI.

Materials and Cost

Screenshots

image

 

image

Control

Lighting Control
Each tailed light uses a GE Link bulb which is connected to a Wink hub.  This allows for on/off control, dimming control, on/off scheduling, and dimming scheduling (such as gradual increases in brightness in the mornings).  Wink comes with a nice app but I opted to use their API so I could incorporate it into the custom UI/dashboard along with everything else.
Cameras
I’m using an old D-Link camera to gain outdoor views and the RPI camera for inside the apartment, I setup scripts to take snapshots once every minute and dump them into a MySQL db running on the Pi and also update the snapshot to include in the UI.
Weather Reporting
Outdoor weather (temperature, “feels like” temperature, humidity, pressure, and windspeed) is pulled from Yahoo! XML weather feeds.
Indoor temperature and humidity is polled every minute using a DHT11 sensor attached to the RPi.  Historicals for all of these are stored in a MySQL database with the intention of graphing these some time in the future.  I’d like to incorporate a Nest-style thermostat for indoor climate control but, alas, I’m a renter and don’t want to deal with that.
Who’s Home?
Using a Bluetooth dongle attached to the RPi, I poll for cell phones to determine who is home and who is away.  Every minute, I log the status of all detected Bluetooth devices so we can see who’s around.  This is also stored in a MySQL db so I can go back in time.
DirecTV Control
Using the DirecTV SHEF API, I currently poll the current program title, the channel number, the station ID (ie NBC or HBO), the program rating, whether or not the DVR is recording.  The API allows you to take full control of the receiver and do all actions you can with the remote but I don’t see much value in that as I can’t watch it while I’m away so why have the functionality…
Appliance Control
Using WEMO plugs, I can power on/off appliances.  This came in handy at Christmas when the outlet was located directly behind the Christmas tree.  At this point, though, there aren’t many appliances I want to control with the Wemo so I have a few of these sitting idle.

Automation and Availability

Amazon Echo Integration
All of these devices have been integrated with the Amazon Echo device either via Echo skills or via IFTT integration.  This allows all of the functionality above to be controlled via voice recognition.  There’s some trial and error getting these setup correctly but I think that’s mostly with the Echo’s voice recognition quality.
IFTT
With integration of IFFT, I can do any number of things if desired.  One of the more useful IFFT setups I’ve found is simply turning on the bedroom lamp ~8 minutes after my alarm goes off and gradually increasing the bulb’s brightness every minute.  Another possible option is to turn some lights on when the Pi’s bluetooth dongle detects that I’m nearby.
Web Server
In order to make this valuable, I installed Apache on the Pi and used ngrok to tunnel to localhost so that I don’t have to worry about the vulnerabilities of port forwarding on my router.  I have this forwarded over to a domain name I wasn’t using and added some .htaccess protection (among other things) to keep it private.

Future Plans

Living in a small apartment limits the value and the opportunities of home automation.  Things like adding reed switches to windows and door don’t make sense in my scenario as I doubt anyone will be climbing through my 7th floor window or trying to break into my door.  Some more practical things I’ll be doing, though, is adding a gas, CO2, and smoke sensor to the Pi so that I’m alerted via text message and push notification if the Pi detects any of those levels becoming elevated…better than waiting on the neighbors to call the fire department, no?  I’d also like to add a PIR motion detector to trigger the Pi Cam to start capturing video instead of still snapshots if motion is detected during hours that I’m normally away from home.  I’ve had some troubles getting the motion detector to work but I’ll loop back to that eventually.