More on the LightwaveRF UDP Interface and COSM

Previously I've blogged on the LightwaveRF UDP interface and logging measurements to COSM.  I've tinkered with the scripts to make them more reliable so I thought I would share a full code listing with some notes as to how to do it yourself.

Before I get into the code, here's a chart from COSM showing recent electricity costs for my home:

This shows how Christmas Eve has been the most expensive day this week, (actually the second most expensive since I started logging).  This was down to us doing stacks of cooking that day in preparation for the Christmas festivities.

The method consists of two Python scripts running on an Android handset.  One to send the command to the LightwaveRF “WiFi Link”, one to receive the response and post it to COSM.

The code listing for the sender script is shown below, (I'll then go on to describe the longer receiver script).  The only thing you'll need to change is the constant "UDP_IP" to represent the IP address that your WiFi Link is on.  I have a feature on my ADSL router to always assign this address to the WiFi link.

The script then stays in and endless loop, sending the command, pausing 60 seconds then starting again.  Change time.sleep(60) if you want it to send more / less frequently.

When you run this script, you’ll have to press a button on the WiFi Link to authorise the handset to send commands to it.  After this you’ll never have to do this again.  The try: and except: constructs allow for error capture and handling.  Until I put these in the script was quite unreliable but with them the script has been running non-stop for 2 months.

import socket
import time

UDP_IP = '192.168.0.2'
UDP_PORT = 9760
INET_ADDR = (UDP_IP,UDP_PORT)

MESSAGE = "123,@?\0"

print "UDP target IP:", UDP_IP 
print "UDP target port:", UDP_PORT 
print "message:", MESSAGE

sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

#sock.sendto(MESSAGE, (UDP_IP, UDP_PORT))

while True:
  try:
    sock.sendto(MESSAGE, INET_ADDR)
    print "done sending udp"
    time.sleep(60)
  except Exception, err:
    #Write a log with the error 

print "Got us an exception: " + str(err)   


The receiving script is a lot longer and is shown below.You'll have to set up your COSM account
accordingly and edit a number of number of things to get it working.

Here are the things to edit and set up:

1)The IP address assigned to the Android handset, (UDP_IP).  Again I use my ADSL router feature to make sure this IP address is always assigned.

2)Change the COSM parameters:

• TimeZoneOffset - This is used to manage timezones and daylight saving time.  I had this as "+01:00" during UK daylight saving time and changed this to "+00:00" when we moved back to Greenwich Mean Time.  Change this to reflect the timezone you're in, (COSM uses UTC).
• MyFeedID - This is the feed ID created in COSM
• MyWattsNowAPIKey - The API key set up for this feed and the associated datastreams. Make sure you have full priviledges for the key.
• The six datastream names.  Four are for the distinct LightwaveRF measurements.  The cost ones are derived from the Wh values and my personal energy costs.

3)The UnitCost and DailyCost parameters represent my electricity costs which are calculate as DailyCost + (UnitCost * KWh) in the CalculateCosts sub-routine.

4)Naughty, naughty - I've embedded what could be a constant quite low down in the code with this line "chdir('/mnt/sdcard/webserver/energy_measurements')".   Change the directory to one you've got set up on your handset, (or delete / comment the line out and the script will write to the /scripts directory).  In this directory I write two log files:

• energy_measurements.csv - This is a local copy of all the measurements I take.
• energy_measurements_log_file.txt - At various points in the code I call a sub-routine called WriteDebugLog which simply write a line of text to a file associated with key points of code execution.  I used this when I set the script up to give me information as to where the code was getting "stuck".

The main body of code then just stays in a continuous loop, waits for a UDP segment to be received, parses it and writes the measurements to COSM.

To run the code simply create 2 .py files with each of the scripts listed, copy them to the /sl4a/scripts directory on the handset and run them.  If it works like mine it will be very reliable!

#v1=Just Watts measurement. V2=Added other4 values. V3=Added GBP values. V4=Added errorlogging

#Import statements
import socket
import datetime
from os import chdir 
import httplib
import sys

#Some constants for this, the server (192.168.0.3)
UDP_IP = "192.168.0.3"
UDP_PORT = 9761            #Responses always sent to this port

#These are constants related to the COSM feed
TimeZoneOffset = "+00:00"
MyFeedID = ""
MyWattsNowAPIKey = ""
MyWattsNowDataStream = "WattsNow"
MyMaxWattsDataStream = "WattsNow_Max"
MyCumulativeWattsDataStream = "WattsNow_Cumulative"
MyYesterdayTotalDataStream = "WattsNow_TotalYesterday"
MyCumulativeCostDataStream = "WattsNow_CostToday"
MyCostYesterdayDataStream = "WattsNow_CostYesterday"

#Constants related to costs
UnitCost =13.42
DailyCost = 16.45

#This is a Python function that writes a log file.  Used for debugging purposes
def WriteDebugLog(StrToLog):
  #Form a date and time for this
  #Get the date and time
  DateToday = datetime.date.today()
  TimeNow = datetime.datetime.now()
    
  #Form the string we will write to screen and local file
  LogFileString = str(DateToday) + "," + str(TimeNow) + "," + StrToLog  


  #And log to file.  "a" means append if necessary
  logfile = open("energy_measurements_log_file.txt", "a")
  logfile.write(LogFileString + "\n")
  logfile.close()  

  return

#This is a Python function to log to COSM
def SendToCOSM(ValToSend,KeyToUse,FeedToUse,DataStreamToUse):
  #Use this try statement to capture errors  
  try:
    #Write to our debug log file
    WriteDebugLog("Start of write to COSM Function. " + DataStreamToUse)  

    #First form the string to send.  Here be an example '2012-09-30T22:00:00.676045+01:00,66'
    #So we need some date geekery for this  
    #Get a variable to hold the date
    today = datetime.datetime.now()

    #Create an overall string with the story so far
    MyDateTimeString = today.strftime("%Y-%m-%d") + "T"

    #Now for the time bit - First the format string
    FormattedTime = today.strftime("%H:%M:%S")    #Get the formatted time

    #Now form the full monty string
    MyDateTimeString = MyDateTimeString + FormattedTime + TimeZoneOffset + "," + ValToSend
  
    #And get it's length
    MyStrLen = str(len(MyDateTimeString))

    #Print what we got so far
    print 'FullString:', MyDateTimeString
  
    #Now do the HTTP magic - Connect to the server
    h = httplib.HTTP('api.cosm.com')
  
    # build url we want to request
    FullURL = 'http://api.cosm.com/v2/feeds/'+ FeedToUse + '/datastreams/' + DataStreamToUse + '/datapoints.csv'

    #Print the URI string we will use
    print "Full URL: " + FullURL
  
    # POST our data.  
    h.putrequest('POST',FullURL)    
 
    # setup the user agent
    h.putheader('X-ApiKey',KeyToUse)
    h.putheader('Content-Length',MyStrLen)   

    # we're done with the headers....
    h.endheaders()
  
    #Send the data
    h.send(MyDateTimeString)

    #Get the response from the request
    returncode, returnmsg,headers = h.getreply()
   
    #display whatever the results are....
    f = h.getfile()
    MyData = f.read()
    print f.read()
  
    #Write to our debug log file
    WriteDebugLog("End of write to COSM Function")
    
    #Now just return
    return 
  #Catch an exception
  except Exception, err:
    #Write a log with the error
    print "Got us an exception: " + str(err)
    #WriteDebugLog("Caught this error in log to COSM function: " + str(err)     
           
#This function calculates the cost in pounds for the electricity used.
#The formula is ((WattHours/ 1000) * (UnitCost / 100)) + (DailyCharge / 100)
def CalculateCosts(InWattHours):
  #WattHours comes in as a string so need to turn to a number
  
  #do the calculation
  CostInPoundsFloat = ((float(InWattHours) / 1000) * (UnitCost / 100)) + (DailyCost / 100)

  #Round it to 2 decimal places
  CostInPoundsFloat = round(CostInPoundsFloat,2)
  
  #return a string
  return str(CostInPoundsFloat)

########################################
#Now we start the main part of the code
########################################


#Change directory that we will write to
chdir('/mnt/sdcard/webserver/energy_measurements') 

#Tell the user we've started
print "UDP server started.  Waiting for response...."

#Bind a socket 
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) # UDP
sock.bind((UDP_IP, UDP_PORT))

#Now just loop until you receive a response
while True:
    #Read data from the buffer
    data, addr = sock.recvfrom(1024) #buffer size is 1024               
    
    #Write to our debug log file
    WriteDebugLog("What we read from the buffer: " + data)
    
    #Get rid of the initial part of the result string as this
    #Is just a static command portion.  First get the length
    DataLength = len(data) - 1
    #Now extract everything from character 7 to the end           
    MeasurementCSV = data[7:DataLength]

    #Write to our debug log file
    WriteDebugLog("Just the measurements after removing the command prefix: " + MeasurementCSV)

    #Get the date and time
    today = datetime.date.today()
    TheTime = datetime.datetime.now()
    
    #Form the string we will write to screen and local file
    OutString = str(today) + "," + str(TheTime) + "," + MeasurementCSV
    
    #Print the result...
    print OutString
    
    #Write to our debug log file
    WriteDebugLog("The string that we will log to the log file: " + OutString)
     
    #And log to file.  "a" means append if necessary
    logfile = open("energy_measurements.csv", "a")
    logfile.write(OutString)
    logfile.close()
     
    #Write to our debug log file
    WriteDebugLog("Have just written the log file CSV") 

    #Split the string and assign to variables  
    SplitMeasurement = MeasurementCSV.split(',')    
    WattsNow = SplitMeasurement[0]            #The power value for now (Watts)
    MaxWatts = SplitMeasurement[1]            #The max power today (Watts)
    CumToday = SplitMeasurement[2]            #Cumulative today (Watt Hours)
    TotalYesterday = SplitMeasurement[3]      #Total yesterday (Watt Hours)

    #Write to our debug log file
    WriteDebugLog("Have just split the string in 4") 

    #Print the output 
    print "Watts Now [W]:" + WattsNow
    print "Max Watts Today [W]:" + MaxWatts
    print "Cumulative Today [Wh]:" + CumToday
    print "Total Yesterday [Wh]:" + TotalYesterday    

    #Write to our debug log file
    WriteDebugLog("Have just printed the measurements to screen") 

    #Log to COSM dude!!! First check it's not 0 as that looks rubbish!
    if WattsNow == "0":
      print "Not sending as it's 0 Watts"
      
      #Write to our debug log file
      WriteDebugLog("Saw that the Watts measurement was 0 so didn't log to COSM") 
    else:
      SendToCOSM(WattsNow,MyWattsNowAPIKey,MyFeedID,MyWattsNowDataStream)
      SendToCOSM(MaxWatts,MyWattsNowAPIKey,MyFeedID,MyMaxWattsDataStream)
      SendToCOSM(CumToday,MyWattsNowAPIKey,MyFeedID,MyCumulativeWattsDataStream)
      SendToCOSM(TotalYesterday,MyWattsNowAPIKey,MyFeedID,MyYesterdayTotalDataStream)

      #Write to our debug log file
      WriteDebugLog("Have just sent the 4 measurements to COSM.  Now calculate costs.") 

      #Now calculate the costs
      CumulativeCost = CalculateCosts(CumToday)
      TotalYesterdayCost = CalculateCosts(TotalYesterday) 
      
      print "Cumulative Cost GBP" + CumulativeCost
      print "TotalCost GBP" + TotalYesterdayCost
      
      #Write to our debug log file
      WriteDebugLog("Have calculated costs. Was cumulative GBP" + CumulativeCost + "and yesterday GBP" + TotalYesterdayCost + ". Now send to COSM") 

      #Send them to COSM
      SendToCOSM(CumulativeCost,MyWattsNowAPIKey,MyFeedID,MyCumulativeCostDataStream)
      SendToCOSM(TotalYesterdayCost,MyWattsNowAPIKey,MyFeedID,MyCostYesterdayDataStream)

      #Write to our debug log file
      WriteDebugLog("Sent costs to COSM.") 

on{X} Traffic Incident Monitoring

A couple of weeks ago I blogged how I'd used on{X} to automate a couple of common tasks based on geo-location (text my Wife when I left work and turn on WiFi at home).  I've also used on{X} to solve another problem in my life, traffic jams!

After suffering a spate of traffic jams (including one that caused my to go on a country lane rally drive) I vowed that I'd use technology to avoid this in future.  You can always check websites before you travel and listen to radio bulletins but this is hit and miss, (i.e. remembering to check the website before you leave).

In the UK, the highways agency publish a bunch of XML feeds showing things like current roadworks, what's showing on the overhead matrix signs and planned roadworks.  See here for the full list.  I chose to use the "unplanned events" feed which uses their road sensor network to warn of incidents.

I've put the full code at the bottom of this posting but in simple terms it just:

1)Detects that I've unlocked the screen of my handset (which I'm doing all the time).
2)Checks that the time is either 6,7,8 in the morning or 4,5,6 in the evening (when I commute).
3)Does a HTTP GET to download the XML feed
4)Parses the feed to pick out a set of roads and place names I care about.  This is controlled by the array:
var MyTrafficArray = ["Birmingham","Wolverhampton",">A34<",">M6<",">M5<"];

So you can just edit the terms in the array to change the set of roads.  The "> <" for the road names means you get an exact match as this is how they're composed in the XML tags, e.g. avoid getting "M6" and "M60" mixed up.

Here's the tool in action.  Firstly, after a screen unlock it warned me that there was a problem on one of the roads:

Then I could just look at the Highways Agency website to get more information:

This warns of the "A34 southbound..." so I could make a judgement as to whether it would impact me.  You can event look at the raw XML, (what the on{X} app parsed} to see more information:

So quite a simple example but there's plenty more tinkering to be had here:

• Adding a timer rather than relying on a screen unlock.
• Mixing location/geo-fencing the XML feed to always be warned wherever I am.
• Playing with some of the other traffic feeds, (variable message signs looks fun).
• Using some of the other UK Government APIs (e.g. weather and crime). 

Here's the full code listing:

// Every day from 0600 until 0900 and 1600 until 1900, when the screen is unlocked, parse the highways agency XML feed for key words.  Then 

// if you find this, show a notification together with which keyword you found. I then use this to trigger lookign at the AA or similar.

//Variables, here's an array of terms to search for

var HighwaysXMLURL = "http://hatrafficinfo.dft.gov.uk/feeds/datex/England/UnplannedEvent/content.xml";

var MyTrafficArray = ["Birmingham","Wolverhampton",">A34<",">M6<",">M5<"];

var HTTPErrorString = "Error in HTTP";

// Triggers the function when the screen unlocks

device.screen.on("unlock", function(){

//console.log('I have been unlocked');

    //Check whether to call the function.  Based upon time of day

        

    //Call the function to get the XML.  This also further calls a function to parse the XML due to the interesting way this all works...

    if (DetermineWhetherToCheck()) 

      {

      GetTheXML();        

      }

    else

      {

      //Just in for test.  Comment out if all is dandy

      //var MyNotification = device.notifications.createNotification('Not time to check now dude');

      //MyNotification.show();      

      }

});

//Sends for the XML

function GetTheXML(){

    device.ajax(

    {    

      url: HighwaysXMLURL,

      type: 'GET',

      headers: {

         //'Content-Type': 'application/xml'

      }  //End of headers

    },  //End of Ajax  

    function onSuccess(body, textStatus, response) {

      //Set up the response XML

      ParseXMLResponse(body);

      //Log to console that all was good      

      //console.info('successfully received http response!');

      },  //onSuccess

    function onError(textStatus, response) {

      var error = {};

      error.message = textStatus;

      error.statusCode = response.status;

      console.error('error: ',error);

      var MeNotification = device.notifications.createNotification('Highway XML Check had HTTP Error');

        MeNotification.show();

    });    

}  //End of GetTheXML

//Parses the XML response

function ParseXMLResponse(InText){

    //Variables

    var SearchPos;         //Position in the string we are searching

    var MyNotification;    //For when we show a notification

    var HitCount = 0;      //Increases if we find a search term

    

    //Loop through our search term array, seeking a key term

    for (var i = 0; i < MyTrafficArray.length; i++) {

      //See if the search string exists in the XML

      SearchPos = InText.search(MyTrafficArray[i]);

      if (SearchPos > -1){   //-1 means not found, otherwise it's the position

        MyNotification = device.notifications.createNotification('Highway XML Check has spotted: ' + MyTrafficArray[i]);

        MyNotification.show();

        //console.log('Highway XML Check has spotted: ' + MyTrafficArray[i]);

        //Increment our hit count counter

        HitCount++;

      } //End of If 

    }  //End of for loop

    

    //See if we had no hits.  Notify just to be sure

    if (HitCount === 0){

        MyNotification = device.notifications.createNotification('Highway XML checked but found nowt');

        MyNotification.show();

    }

}  //End of function

function DetermineWhetherToCheck()

  {

  //Get the current hour value.  Only check if it's 6,7 or 8 or 16, 17 or 18'          

  //The variable to use for the time

  var currentTime = new Date();

    

  //Form the hours part and check it

  var hours = currentTime.getHours();

  

  if (hours == 6 || hours == 7 || hours == 8 || hours == 16 || hours == 17 || hours == 18)

    {

    return true;    

    }

  else

    {

    return false;    

    }

  }  //End of function