Sunday, 1 March 2015

Ultrasonic Elephant Cheese-puff Game using Raspberry Pi and Scratch

In two previous posts I've written how my kids and I used an ultra sonic sensor, Raspberry Pi and Scratch to make a press up counter and a cuddly toy theft alarm.  One deficiency in those projects was that the Scratch/GPIO capability could only take an ultra-sonic measurement every one second.  In particular, for the press up counter I had to do the press ups REALLY slowly to get them to count. Too much like hard work!

A bit more investigation on Simple Si's epic Scratch/GPIO site showed that he has created a capability using an "UltraDelay" variable which means you can take a reading every 0.2 seconds.  So this means you can many more measurements and so, racking my brains to think what we could do with this, I had the idea of making the Scratch cat levitate by moving my hand up and down over the ultra sonic sensor.  A extension of this was to create a Scratch game, using the sensor to control a sprite's position.

Here's what we came up with:

  • Have a sprite moving across the screen from right to left at random heights.  When it reaches the left hand side of the screen it returns to the right hand side and re-starts at a new height.
  • Have a sprite that moves up and down the screen based upon the position of your hand over the sensor.  If this sprite touches the right to left sprite then you score points.
  • Have this happening for a fixed period of time and then have the final score read out at the end.

Here's the code for the right to left sprite:

So a very simple block of code:
  • Start when g (for game) is pressed.
  • Set the initial position of the sprite as x=200, y=-100.  This is on the right edge of the screen towards the bottom.
  • Loop 600 times.  
  • For each loop, move the x position of the sprite by -13 steps. 
  • If the sprite gets to an x position of -180 (left hand side of the screen) then move it back to the right hand side of the screen but at a random height that is somewhere between -200 (bottom of screen) and 200 (top).
There was some excellent learning to be had just in this short block of code:
  • x and y coordinates on the screen.
  • Loops, and changing the loop variable to get the game time we needed.
  • Experimenting with the number of steps to move the sprite by for each loop.  The final value of -13 was found to be just the right number to make the game challenging but not too challenging.
  • An if statement and an action taken should the statement be true.

Which leaves the (slightly more complex) code for the second sprite:
Here we:
  • Again start by pressing g.
  • Set the UltraDelay variable to 0.  This means a reading is available every 0.2 seconds.
  • Set a HitCount variable to 0, this was incremented when the sprites touch.
  • Set the sprite to it's initial position.  x = -100 is left of centre meaning there's time to move the sprite into position to hit the right to left sprite.
  • Go into a loop with 600 iterations, the same as for the other sprite.
  • Broadcast for an ultrasonic measurement.
  • Get the response and then set the y position of the sprite to be (value * 10) - 200.  Having -200 meant that when the value is very low (or zero) the sprite moves to the bottom of the screen.  Having * 10 means that 40cm worth of hand movement over the sensor can translate to 400 steps of movement of the sprite.
  • A check to see of the sprite is touching the right-left sprite.  If it is then increment a variable to count the number of touches and play a pop sound.
  • When the main 600 iterations are done, go into a loop that reads out the final score.

So in doing this we learnt:
  • About setting and incrementing variables
  • About maths statements, taking one value and transposing it to another.
  • The simple concept of taking a real world value and turning this into an on-screen analogue.

Which left the important task of choosing the sprite icons.  For this we used:
  • A bowl of cheese puffs for the right to left sprite.
  • An elephant for the up and down sprite.

You ask why?  I say why not!!!

Here's the game in action:


Future improvements:
  • A high score feature.
  • Having only one point scored when the sprites touch (should be able to use a simple boolean for this).





Sunday, 22 February 2015

Ultra Sonic Pinky Pig Anti-Theft Device

Meet Pinky Pig:



Pinky Pig is the most loved soft toy in the Geek household.  He's been owned by my eldest daughter since birth, is a a bit worse for wear these days but is loved all the same.  In fact Pinky Pig rarely leaves the house,such would be the trauma of losing him.

In a search for something new to do with our ultra sonic sensor (see last week's post), we came up with the idea of using the ultra sonic sensor to make a device to detect if Pinky has been stolen.  In simple terms, Pinky's resting place would be on the ultra sonic sensor.  If he was taken then this could be detected and an alarm raised.  

We had a lot of fun (and a lot of learning) coming up with the Scratch code below on one of our Raspberry Pi's.  It was a really good way to focus on the role of a loop and an if statement.



So if the measurement from the sensor is small, all is well and no alarm is raised.  However if the measurement is not small then the (audible) alarm is raised.

We we did learn which was interesting was that Pink Pig (and other cuddly toys) are transparent to ultra sonic sound waves.  Very interesting, this added an extra layer of investigation to this project.  Please comment below if you can tell me why this is the case.  Hence to make the alarm work, Pinky had to be wrapped in an extra layer of paper.  Here's a picture showing this in action.


So last thing, here's a video of it in action (wish the volume on the TV was louder - listen carefully and you can hear the alarm going off).



Monday, 16 February 2015

Raspberry Pi Ultra Sonic Press Up (Push Up) Counting Machine using Scratch

Looking at back at recent posts I've noticed:
  • I've not done any electronics related projects recently.
  • I've not done much programming with my kids recently.
When doing some press ups* recently I lost count and couldn't log them on my Jerks exercise logging system.  Hence this gave me the idea to build a machine to count my press ups.

(*Push ups for you readers in North America)

I saw you can buy one from Amazon and there are some available as smart phone apps but this isn't the Geek Dad way; I wanted to build one with a Raspberry Pi, program it in Scratch and use some electronics.  Ideas:
  • Mount a light dependent resistor - as I go down it blocks out light from the LDR, as I go up it allows light again and counts a press up.
  • Use a camera and a QR code.  As I go down it reads the QR code and counts a press up.
  • Use an ultra sonic sensor to measure distance as I go up and down.
I decided to use an ultra sonic sensor - purely because I'd hadn't used one since a University project about 20 years ago.  Here's the one I bought, an HC-SR04 module which has both a transmitter and a receiver.  They're widely available on the interweb.


To integrate the sensor to my Raspberry Pi I used the utterly amazing, epic and awesome Cymplecy Scratch GPIO system (did I mention that this site is brilliant).  This chap has developed a series of Scratch releases that allow you to integrate with many and various third party boards and bits of electronic kit that you can connect to the GPIO.  There's a specific page on Ultra Sonic module integration; it's available here and the solution only uses one GPIO input/output which is ace.

The resistors shown on the breadboard diagram on Cymplecy's site are essential to stop you blowing up your Raspberry Pi; they create a potential divider to reduce the +5V output from the HC-SR04 to a GPIO friendly 3.3V.  The resistors are:
  • Brown-Black-Red = 1kOhms
  • Red-Red-Red = 2.2kOhms
I won`t do the maths here, but see this site for an explanation of potential dividers.

The system works brilliantly and with this code block you can get Scratch to show the measurement from the ultra sonic sensor and move the Scratch Cat around.



So this worked beautifully, next I needed a robust housing for the press up counting machine to avoid me breaking it.  Enter Lego, the tool of tinkerers everywhere.  Here's a series of before, during and after shots:





So the breadboard is mounted perpendicular in the Lego housing, meaning the sensor points straight up.  Hence as I'm doing press ups I'm moving up and down vertically over the sensor.  The Raspberry Pi sits separate to it.

Here's a quick video:



The basic algorithm for the press up code is as follows:

boolean = false
forever
  Take Measurement
  if Measurement < Low AND boolean = false
    boolean = true
  if Measurement > High and boolean = true
    increment press up count
    boolean = false

So this uses a simple boolean to log when my body is low down (say < 10cm from the sensor).  The spots the bottom part of the press up.  Then when I move up, if my body is high up (say > 30cm) AND I've previously been low down then a press up is counted (and the boolean is reset).

The basic Scratch code is shown on the screen shot below (put your email address in a comment below and I'll send you the file).  This does a good job of counting press ups (although you have to go quite slow as the sampling period is only one second) and getting the Scratch Cat to say them.



What I wanted to do with my daughters is jazz it up a bit.  In preparation for this I recorded a number of different audio messages for Scratch to play out.  I used this site which takes a text string input and provides a downloadable MP3 file.  These files can then be put in the /usr/share/scratch/Media/Sounds directory on your Raspberry Pi.  Armed with this I spent an enjoyable afternoon with my daughters making the press up counter more interesting.

We:
  • Changed the background to make it look better.
  • Added a sound at the start to tell you that "It's press up time!".
  • Added If statements to detect each Press Up (up to 10) and play an audio message with the number ("1" for the first press up, "5" for the fifth etc).
  • Added some motivational (semi-abusive) messages to keep you going.
  • Added a "Well Done" message at the end.

Here's a video of it in action!



Here's a screenshot of the Scratch code (top part). A but hard to see so, again, put your email address in comments below and I'll send you the Scratch file.


...and bottom part:







    

Saturday, 7 February 2015

Fixing LightwaveRF UDP Power Measurements

The thing that got me into this Geek blogging lark was a challenge from an old boss to look into how home automation equipment works.  This culminated in a posting on how to take power measurements on a Raspberry Pi and post them to an online measurements logging site (COSM, now called Xively).

Those measurements were happily up, working and logging for over 3 years until they abruptly stopped working towards the end of last year.  Looking at the LightwaveRF control unit (the so called WiFi link) the firmware version is now 2.91Q.  A bit of interweb searching tells me that this version was introduced at some point in late 2014 and I'm guessing it's that which has caused the problem.  (In an old posting I noticed that I had version 2.28).

It's taken me a while to get round to it  but I've finally investigated and fixed the problem that stopped the measurements working.  Here's what I did...

Step 1 was to check the IP address used for the LightwaveRF control unit.  Reminder: the way to get measurements is to send the string "123,@?\0" in a UDP segment to port 9760 on  the IP address that the LightwaveRF control unit sits on.  The response comes back on port 9761.

The IP address of the LightwaveRF control unit is defined as a constant in my script so I had to make sure that the one I had defined was still the correct one.  A quick check of my router "LAN IP Address" settings (which means the same IP address is always allocated to the same MAC address) showed that it was all OK.

Step 2 was to use tcpdump to work out what was going on on my LAN between my Raspberry Pi and the LightwaveRF control unit.

The command sudo tcpdump -n -i any -s0 -vv -X host 192.168.0.2 means:

"-n" means don't resolve addresses, "-i any" means any interface, "-s0" means all the packet, "-vv" means very verbose, "-X" means give us the full packet.

The log I got was:

15:31:24.446184 IP (tos 0x0, ttl 64, id 3918, offset 0, flags [DF], proto UDP (17), length 35)
    192.168.0.5.46246 > 192.168.0.2.9760: [udp sum ok] UDP, length 7
        0x0000:  4500 0023 0f4e 4000 4011 aa24 c0a8 0005  E..#.N@.@..$....
        0x0010:  c0a8 0002 b4a6 2620 000f ff13 3132 332c  ......&.....123,
        0x0020:  403f 00                                  @?.
15:31:24.651793 IP (tos 0x0, ttl 100, id 63682, offset 0, flags [none], proto UDP (17), length 56)
    192.168.0.2.9760 > 255.255.255.255.9761: [udp sum ok] UDP, length 28
        0x0000:  4500 0038 f8c2 0000 6411 9d48 c0a8 0002  E..8....d..H....
        0x0010:  ffff ffff 2620 2621 0024 38ed 3132 332c  ....&.&!.$8.123,
        0x0020:  3f57 3d35 3739 2c39 3536 312c 3632 3531  ?W=579,9561,6251
        0x0030:  2c35 3630 343b 0d0a                      ,5604;..
15:31:27.110984 IP (tos 0x0, ttl 100, id 63683, offset 0, flags [none], proto UDP (17), length 196)
    192.168.0.2.4096 > 255.255.255.255.9761: [udp sum ok] UDP, length 168
        0x0000:  4500 00c4 f8c3 0000 6411 9cbb c0a8 0002  E.......d.......
        0x0010:  ffff ffff 1000 2621 00b0 a208 2a21 7b22  ......&!....*!{"
        0x0020:  7472 616e 7322 3a32 3631 3335 362c 226d  trans":261356,"m
        0x0030:  6163 223a 2230 333a 3031 3a35 4222 2c22  ac":"03:01:5B","
        0x0040:  7469 6d65 223a 3134 3232 3830 3436 3836  time":1422804686
        0x0050:  2c22 7072 6f64 223a 2270 7772 4d74 7222  ,"prod":"pwrMtr"
        0x0060:  2c22 7365 7269 616c 223a 2242 4332 4246  ,"serial":"BC2BF
        0x0070:  4522 2c22 7369 676e 616c 223a 3535 2c22  E","signal":55,"
        0x0080:  7479 7065 223a 2265 6e65 7267 7922 2c22  type":"energy","
        0x0090:  6355 7365 223a 3537 372c 226d 6178 5573  cUse":577,"maxUs
        0x00a0:  6522 3a39 3536 312c 2274 6f64 5573 6522  e":9561,"todUse"
        0x00b0:  3a36 3235 332c 2279 6573 5573 6522 3a35  :6253,"yesUse":5
        0x00c0:  3630 347d                                604}

The first packet is the command sent from my Python script.

The second and third packets are from the LightwaveRF control unit.  There are two points to notice:

1)The response is now sent to 255.255.255.255, the zero network broadcast address.  I assume that this means you can have several hosts on the same network logging and broadcasting their results.

2)There is now two responses from the control unit.  One looks to be in the same format as before (four comma separated values) and the other looks to be some longer format that needs investigating.

However what I also spotted was that the responses in the new format seem to come without me actually sending the "123,@?\0" string.  This was puzzling until I realised that the control unit was broadcasting out measurements without being prompted, i.e. without me sending anything in the first place.  A simpler tcpdump shows:

pi@raspberrypi ~/lightwaverf $ sudo tcpdump -n -i any -s0 host 192.168.0.2
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
18:26:36.135825 IP 192.168.0.2.4096 > 255.255.255.255.9761: UDP, length 169
18:26:51.291677 IP 192.168.0.2.4096 > 255.255.255.255.9761: UDP, length 169
18:27:06.344437 IP 192.168.0.2.4096 > 255.255.255.255.9761: UDP, length 169
18:27:21.501676 IP 192.168.0.2.4096 > 255.255.255.255.9761: UDP, length 169
18:27:36.553111 IP 192.168.0.2.4096 > 255.255.255.255.9761: UDP, length 169
18:27:51.708639 IP 192.168.0.2.4096 > 255.255.255.255.9761: UDP, length 169
18:28:06.761778 IP 192.168.0.2.4096 > 255.255.255.255.9761: UDP, length 169
18:28:21.917158 IP 192.168.0.2.4096 > 255.255.255.255.9761: UDP, length 169

Here you can see a measurement being broadcast every 15 seconds.

So these are the problems that need fixing....

Problem 1 was easy to fix.  Changing a constant to:

UDP_IP = "255.255.255.255"

Means that when these commands are run:

sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) # UDP
sock.bind((UDP_IP, UDP_PORT))

...the script binds to 255.255.255.255 to capture the response.

Problem 2 was a bit harder to fix.  Capturing and printing the new string showed it to be of this format:

*!"trans":261534,"mac":"03:01:5B","time":1422813309,"prod":"pwrMtr","serial":"BC2BFE","signal":55,"type":"energy","cUse":1504,"maxUse":9561,"todUse":7696,"yesUse":5604}

So a series of comma and colon separated values.  The initial values look to be useful metadata (e.g. MAC address, power meter etc) which would be useful in a multi-measurement, multi-unit environment.  However for me it's just the last 4 elements I need (current power, max power used today, cumulative today and cumulative yesterday).

I updated the code to remove the old measurement handling statements and to interpret the new measurement types.

The code below is the key new element added.  I also removed the code handling the old response format.  As the control unit just broadcasts energy measurements I no longer have to run a script to poll for information.  This should make the overall system more reliable...

#Put result into an array and get rid of the superfluous bits
      ResultArray = data.split(',')
      MeasurementCSV = TrimAndTidy(ResultArray[7]) + ',' + TrimAndTidy(ResultArray[8]) + ',' + TrimAndTidy(ResultArray[9]) + ',' + TrimAndTidy(ResultArray[10])
      print 'Measurement CSV >>> ' + MeasurementCSV

Here I simple split the string into parts (based upon the comma) and then send it to a tidy up function.  This is a simple piece of code:

def TrimAndTidy(InputString):
  #First split based upon the : and then remove a }
  try:
    ArrayOfVals = InputString.split(':')
    return ArrayOfVals[1].replace('}','')
  except:  #Just return a zero
    return '0'

That simply splits based upon the colon and cleans up the extra } at the end of the response string.

As an aside, one of the thing the code does is log to a local log file (in CSV format).  The log file is 32Meg in size and will have several hundred thousand rows of data it.  I smell some excellent geek action analysing those files to look for trends....

Here it is working again, huzzah!



...and here's all the code:

#v1=Just Watts measurement.  V2=Added other4 values.  V3=Added  GBP values.  V4=Added errorlogging.  V5=Handled lightwaverf response format changes
#V6=Removed old methodology

#Change IP address un comment chmod before putting on the phone "server"

#Import statements
import socket
import datetime
from os import chdir
import httplib
import sys

#Some constants for this, the server (192.168.0.5)
UDP_IP = "255.255.255.255" #Changed 31/01/2015 when Lightwaverf box started sending responses to broadcast address
UDP_PORT = 9761            #Responses always sent to this port

#These are constants related to the COSM feed
TimeZoneOffset = "+00:00"
MyFeedID = "YOUR_ID_HERE"
MyWattsNowAPIKey = "YOUR_KEY_HERE"
MyWattsCostAPIKey ="YOUR_KEY_HERE"
MyWattsNowDataStream = "WattsNow"
MyMaxWattsDataStream = "WattsNow_Max"
MyCumulativeWattsDataStream = "WattsNow_Cumulative"
MyYesterdayTotalDataStream = "WattsNow_TotalYesterday"
MyCumulativeCostDataStream = "WattsNow_CostToday"
MyCostYesterdayDataStream = "WattsNow_CostYesterday"


#Constants related to costs
UnitCost =11.68
DailyCost = 18.00

#This is a Python function that writes a log file.  Used for debugging purposes
def WriteDebugLog(StrToLog):
  #Form a date and time for this
  #Get the date and time
  DateToday = datetime.date.today()
  TimeNow = datetime.datetime.now()

  #Form the string we will write to screen and local file
  LogFileString = str(DateToday) + "," + str(TimeNow) + "," + StrToLog


  #And log to file.  "a" means append if necessary
  #COMMENT OUT THE 3 ROWS BELOW ELSE UNLESS YOU NEED THE LOG
  logfile = open("energy_measurements_log_file.txt", "a")
  logfile.write(LogFileString + "\n")
  logfile.close()
  return

#This is a Python function to log to COSM
def SendToCOSM(ValToSend,KeyToUse,FeedToUse,DataStreamToUse):
  #Use this try statement to capture errors
  try:
    #Write to our debug log file
    WriteDebugLog("Start of write to COSM Function. " + DataStreamToUse)

    #First form the string to send.  Here be an example '2012-09-30T22:00:00.676045+01:00,66'
    #So we need some date geekery for this
    #Get a variable to hold the date
    today = datetime.datetime.now()

    #Create an overall string with the story so far
    MyDateTimeString = today.strftime("%Y-%m-%d") + "T"

    #Now for the time bit - First the format string
    FormattedTime = today.strftime("%H:%M:%S")    #Get the formatted time

    #Now form the full monty string
    MyDateTimeString = MyDateTimeString + FormattedTime + TimeZoneOffset + "," + ValToSend

    #And get it's length
    MyStrLen = str(len(MyDateTimeString))

    #Print what we got so far
    print 'FullString:', MyDateTimeString

    #Now do the HTTP magic - Connect to the server
    h = httplib.HTTP('api.cosm.com')

    # build url we want to request
    FullURL = 'http://api.cosm.com/v2/feeds/'+ FeedToUse + '/datastreams/' + DataStreamToUse + '/datapoints.csv'

    #Print the URI string we will use
    print "Full URL: " + FullURL

    # POST our data.
    h.putrequest('POST',FullURL)

    # setup the user agent
    h.putheader('X-ApiKey',KeyToUse)
    h.putheader('Content-Length',MyStrLen)

    # we're done with the headers....
    h.endheaders()

    #Send the data
    h.send(MyDateTimeString)

    #Get the response from the request
    returncode, returnmsg,headers = h.getreply()

    #display whatever the results are....
    f = h.getfile()
    MyData = f.read()
    print f.read()

    #Write to our debug log file
    WriteDebugLog("End of write to COSM Function")

    #Now just return
    return
  #Catch an exception
  except Exception, err:
    #Write a log with the error
    print "Got us an exception: " + str(err)
    #WriteDebugLog("Caught this error in log to COSM function: " + str(err)

#This function calculates the cost in pounds for the electricity used.
#The formula is ((WattHours/ 1000) * (UnitCost / 100)) + (DailyCharge / 100)
def CalculateCosts(InWattHours):
  #WattHours comes in as a string so need to turn to a number

  #Added 26/10/2014 - First check InWattHours
  print "InWattHours: " + InWattHours

  if (";" in InWattHours):
    print "Caught a rogue InWattHours: " + InWattHours
    CostInPoundsFloat = "0"
  else:
    #do the calculation
    CostInPoundsFloat = ((float(InWattHours) / 1000) * (UnitCost / 100)) + (DailyCost / 100)

    #Round it to 2 decimal places
    CostInPoundsFloat = round(CostInPoundsFloat,2)

  #return a string
  return str(CostInPoundsFloat)

#New def added 01/02/2015 to handle the new format of command response.  Example
#*!{"trans":261395,"mac":"03:01:5B","time":1422806123,"prod":"pwrMtr","serial":"BC2BFE","signal":57,"type":"energy","cUse":809,"maxUse":9561,"todUse":6349,"yesUse":5604}
#Will get one of the pieces from within the commas.  Need to get the part after the colon and if required trim off a }
def TrimAndTidy(InputString):
  #First split based upon the : and then remove a }
  try:
    ArrayOfVals = InputString.split(':')
    return ArrayOfVals[1].replace('}','')
  except:  #Just return a zero
    return '0'

########################################
#Now we start the main part of the code
########################################


#Change directory that we will write to
chdir('/home/pi/lightwaverf')

#Tell the user we've started
print "UDP server started.  Waiting for response...."

#Bind a socket
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) # UDP
sock.bind((UDP_IP, UDP_PORT))

#Now just loop unit you receive a response
while True:
    #Read data from the buffer
    data, addr = sock.recvfrom(1024) #buffer size is 1024

    #Write to our debug log file
    WriteDebugLog("What we read from the buffer: " + data)

    #A debug print
    print 'I just read this >>> ' + data

    try:
      #Put result into an array and get rid of the superfluous bits
      ResultArray = data.split(',')
      MeasurementCSV = TrimAndTidy(ResultArray[7]) + ',' + TrimAndTidy(ResultArray[8]) + ',' + TrimAndTidy(ResultArray[9]) + ',' + TrimAndTidy(ResultArray[10])
      print 'Measurement CSV >>> ' + MeasurementCSV

      #Write to our debug log file
      WriteDebugLog("Just the measurements after removing the command prefix: " + MeasurementCSV)

      #Get the date and time
      today = datetime.date.today()
      TheTime = datetime.datetime.now()

      #Form the string we will write to screen and local file
      OutString = str(today) + "," + str(TheTime) + "," + MeasurementCSV

      #Print the result...
      print OutString

      #Write to our debug log file
      WriteDebugLog("The string that we will log to the log file: " + OutString)

      #And log to file.  "a" means append if necessary
      logfile = open("energy_measurements.csv", "a")
      logfile.write(OutString)
      logfile.close()

      #Write to our debug log file
      WriteDebugLog("Have just written the log file CSV")

      #Split the string and assign to variables
      SplitMeasurement = MeasurementCSV.split(',')
      WattsNow = SplitMeasurement[0]            #The power value for now (Watts)
      MaxWatts = SplitMeasurement[1]            #The max power today (Watts)
      CumToday = SplitMeasurement[2]            #Cumulative today (Watt Hours)
      TotalYesterday = SplitMeasurement[3]      #Total yesterday (Watt Hours)

      #Write to our debug log file
      WriteDebugLog("Have just split the string in 4")

      #Print the output
      print "Watts Now [W]:" + WattsNow
      print "Max Watts Today [W]:" + MaxWatts
      print "Cumulative Today [Wh]:" + CumToday
      print "Total Yesterday [Wh]:" + TotalYesterday

      #Write to our debug log file
      WriteDebugLog("Have just printed the measurements to screen")

      #Log to COSM dude!!! First check it's not 0 as that looks rubbish!
      if WattsNow == "0":
        print "Not sending as it's 0 Watts"

        #Write to our debug log file
        WriteDebugLog("Saw that the Watts measurement was 0 so didn't log to COSM")
      elif (WattsNow.isdigit() == False):
        print "Not sending as it's not a numeric Watts value"

        #Write to our debug log file
        WriteDebugLog("Saw that the Watts measurement was not a number so didn't log to COSM.  Measurement >> " + WattsNow)
      else:
        SendToCOSM(WattsNow,MyWattsNowAPIKey,MyFeedID,MyWattsNowDataStream)
        SendToCOSM(MaxWatts,MyWattsNowAPIKey,MyFeedID,MyMaxWattsDataStream)
        SendToCOSM(CumToday,MyWattsNowAPIKey,MyFeedID,MyCumulativeWattsDataStream)
        SendToCOSM(TotalYesterday,MyWattsNowAPIKey,MyFeedID,MyYesterdayTotalDataStream)

        #Write to our debug log file
        WriteDebugLog("Have just sent the 4 measurements to COSM.  Now calculate costs.")

        #Now calculate the costs
        CumulativeCost = CalculateCosts(CumToday)
        TotalYesterdayCost = CalculateCosts(TotalYesterday)

        print "Cumulative Cost GBP" + CumulativeCost
        print "TotalCost GBP" + TotalYesterdayCost

        #Write to our debug log file
        WriteDebugLog("Have calculated costs. Was cumulative GBP" + CumulativeCost + "and yesterday GBP" + TotalYesterdayCost + ". Now send to COSM")

        #Send them to COSM
        SendToCOSM(CumulativeCost,MyWattsCostAPIKey,MyFeedID,MyCumulativeCostDataStream)
        SendToCOSM(TotalYesterdayCost,MyWattsCostAPIKey,MyFeedID,MyCostYesterdayDataStream)

        #Write to our debug log file
        WriteDebugLog("Sent costs to COSM.")
    except Exception, err:
      #Write a log with the error
      print "Got us an exception: " + str(err)
      #WriteDebugLog("Caught this error in the main area of code: " + str(err)

Saturday, 31 January 2015

Terrific Times Table Test Tool with SL4A

School test practice for my daughter has again given me inspiration for a Geek project (after my previous project to create a spellings test tool).  At my daughter's school they're currently conducting quick-fire times oral table tests where the teacher reads out 50-odd times table questions in quick succession.  In helping my eldest practice for these tests we found:

  1. It's hard to be really random in the questions you ask and you often end up focusing on one times table (e.g. the twelves).
  2. It's hard to keep track of the questions you ask and the associated answers.
  3. It's hard to make sure you are keeping to an appropriate pace.

So I solved this challenge using SL4A, my favourite scripting language for Android handsets.

Using Python and the Android API binding I could implement the following features:

Feature 1: Generate random times table questions using the randint method from the random module:

from random import randint
FirstVal = randint (3,12)
SecondVal = randint (3,12) 

...the 3 and 12 attributes means the random numbers are always within the range 3 to 12 (inclusive).  This means no super easy 1 and 2 times tables.

Feature 2: Form strings to speak and to print to screen and then say the times table:

#Form strings to print and say
SayString = str (FirstVal) + ' times ' + str (SecondVal)
PrintString = str (i) + ': ' + SayString + ' = ' + str (FirstVal * SecondVal)
print PrintString

#Speak the times table
droid.ttsSpeak(SayString)

Feature 3: Compute a variable delay after speaking ones times table and then speaking the next. Easier calculations get a shorter delay, harder calculations get a longer delay:

  #Compute a different delay depending on the complexity of the times table
  TheAnswer = FirstVal * SecondVal
  if TheAnswer < 60:
    TheDelay = 3
  elif TheAnswer > 90:
    TheDelay = 5
  else:
    TheDelay = 4
  time.sleep (TheDelay)

...it's arguable that the complexity of a times table is not 100% correlated with how large the number is (e.g. 7 * 8 is harder than 10 * 10) but it's a good general rule of thumb.

Feature 4: Having an on-screen record of all the questions.  These can be used for post-test marking of my daughter's answers:



Here's a video of it in action:



...and here's all the code.  The great things for me is that this tool, which makes life just a little bit easier, took a mere 20 minutes to write, test and debug.  SL4A is epic!

import android
from random import randint
import time

#Create a Droid object and speak a welcome message
droid = android.Android()
droid.ttsSpeak('Hi. get ready for a terrific times table test')

#Sleep - gives time for the TTS application to initiate
time.sleep (10)

#Loop for as many times as required
for i in range (1,51):
  #Form the two values for the times table
  FirstVal = randint (3,12)
  SecondVal = randint (3,12)
    
  #Form strings to print and say
  SayString = str (FirstVal) + ' times ' + str (SecondVal)
  PrintString = str (i) + ': ' + SayString + ' = ' + str (FirstVal * SecondVal)
  print PrintString
    
  #Speak the times table
  droid.ttsSpeak(SayString)
    
  #Compute a different delay depending on the complexity of the times table
  TheAnswer = FirstVal * SecondVal
  
  if TheAnswer < 60:
    TheDelay = 3
  elif TheAnswer > 90:
    TheDelay = 5
  else:
    TheDelay = 4
  time.sleep (TheDelay)

Saturday, 24 January 2015

Raspberry Pi - Python - MySQL - Cron Jobs and Physical Jerks #2

In a previous posting I described how I used Twitter, Python and MySQL to capture and log my exercise habits (what I like to call physical jerks).

What I've learnt about myself over the years is that to keep exercising regularly I need:

  • Gratification, i.e. something to say "well done" when I've done some exercise (a la Strava Kudos).
  • Having some fun data to play with.
  • Nagging, i.e. something to keep telling me to do my exercises. 
  • Targets

I talked about gratification in my last Jerks post.  When I do exercise, I tweet, my Raspberry Pi picks this up and sends me a Twitter DM to say "well done".  Example:


When it comes to fun data, it's a virtuous circle.  I exercise more and get fitter, I get more fun data. I want more fun data, I exercise more and get fitter.  At the time of writing the database looks something like this:

mysql> SELECT exercise, SUM(count) FROM exercise GROUP BY exercise order by sum(count);
+--------------------+------------+
| exercise           | SUM(count) |
+--------------------+------------+
| Yoga               |          7 |
| Pilates            |          7 |
| Leg Weights        |         10 |
| Hundred Ups        |         11 |
| Foam Rolling       |         16 |
| General Stretching |         16 |
| Squatting          |         55 |
| Side Raises        |        169 |
| Arm Raises         |        169 |
| Bicep Curls        |        176 |
| Shoulder Press     |        182 |
| Tricep Curls       |        239 |
| Clap Press Ups     |        263 |
| Sit Ups            |        335 |
| Abdominal Crunches |        578 |
| Press Ups          |        872 |
| Calf Raises        |       1384 |
+--------------------+------------+
17 rows in set (0.13 sec)

When it comes to nagging, this is what I've been working on recently.  I decided to create a Python script that would periodically email with details of:
  • Jerks I've done today
  • Jerks I did yesterday
  • Jerks I've done this week
  • Jerks I've done this month
  • Jerks I've done this year
  • All time Jerks
The SQL for this is pretty basic, (similar to that laid out above but with date parameters).  The first thing I needed to do was be able to look at today's date and calculate some other dates as offsets to it (i.e. date yesterday, date of start of week, date of start of year).  Here's an example for start of week from the GetADate function (full code below):

 elif (DateType == DateFirstDayOfMonth):
      now = datetime.now()
      DayOfMonth =  int(now.strftime("%d"))
      DayDelta = DayOfMonth - 1
      FirstDateOfMonth = now - timedelta(days=DayDelta)
      return FirstDateOfMonth.strftime("%Y-%m-%d")

This uses the "%d" attribute for strftime to return the number associated with the day of month.  e.g. would return 24 for today, the 24th of January.  It then uses the timedelta method (imported from datetime) with an offset of the day number minus 1 (so 23 in my example) to calculate the date of the first day of the month.  This is then returned to be used in the SQL.

The email I create is formed from HTML so there's a function (CreateHTMLTable) that takes an SQL cursor as an attribute and forms a heading plus HTML table.  It does this no matter how many columns or rows in the SQL response.  This results in a HTML segment, albeit with no indentation.

I send the email using methods from the smtplib module.  There's plenty of examples of how to do this on the interweb.  I used this one from Stack Overflow that shows how to create HTML and text emails.  The full code is shown below in the SendEmail function and is pretty self-explanatory.  What I did find is that when I tried to use my Gmail and Outlook.com accounts to send the email, these providers did not "like" me using this method to send.  Gmail blocked it out-right, telling me to lower  my privacy settings to continue (which I didn't).  Outlook.com kept asking me to re-authenticate my account which was a pain.  I ended up using an old, unused email account from my ISP which seems to less restrictions.  (It's re-assuring that Google and Microsoft have implemented these feature).

So a cron job runs the script every hour (from 8am to 10pm at weekends and 6pm to 10pm on weekdays).  The email comes to my smartphone and,  as I'm basically addicted to it, I pick it up pretty quickly.  The first email of the day is often something like this which is a big insulting nag to do something:



...but then I get emails like this which is like "get in, did more today than yesterday":


Then I get a series of interesting summaries like these:


...and these:



Followed by a reminder of the short codes for the TUI:

So that just leaves the targets.  I set these at the start of the year:


...but currently have to manually compare actuals with targets.  Sounds like another Geek Dad project to create a nagging capability that includes targets...

Full code listing:

#V1 - First version with exercise table summary
#V2 - Second version with HTML tables and lookup summary
#V3 - Added more summaries and a def to create tables
#V4 - Finished the summaries and formatting changes

#Sends a summary email of Jerks exercise
from datetime import datetime, timedelta
import smtplib
import MySQLdb

#MIME multipart stuff
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText

#Email Constants
smtpserver = 'SMTP Server Here'
AUTHREQUIRED = 1 # if you need to use SMTP AUTH set to 1
smtpuser = 'SMTP Username Here'  # for SMTP AUTH, set SMTP username here
smtppass = 'SMTP Password Here'  # for SMTP AUTH, set SMTP password here
RECIPIENTS = 'Receiving address Here'
SENDER = 'Sending address Here'

#Database related contents
dbMainTable = "exercise"
dbLookupTable = "lookup"

#Date manipulation related constants
DateToday = 'DateToday'
DateYesterday = 'DateYesterday'
DateFirstDayOfWeek = 'DateFirstDayOfWeek'
DateFirstDayOfMonth = 'DateFirstDayOfMonth'
DateFirstDayOfYear = 'DateFirstDayOfYear'

#Gets a initial timestampt for emails
def GetDateTime():
  #Get the time, format it and return it
  now = datetime.now()
  return str(now.date()) + ' ' + str(now.strftime("%H:%M:%S"))

#Send an email
def SendEmail(TheSubject,TheMessage):
  #Form the message
  #msg = "To:%s\nFrom:%s\nSubject: %s\n\n%s" % (RECIPIENTS, SENDER, TheSubject, TheMessage)

  #Start forming the MIME Multipart message
  msg = MIMEMultipart('alternative')
  msg['Subject'] = TheSubject
  msg['From'] = SENDER
  msg['To'] = RECIPIENTS

  # Record the MIME types of both parts - text/plain and text/html.
  #part1 = MIMEText(text, 'plain')
  part2 = MIMEText(TheMessage, 'html')

  # Attach parts into message container.
  # According to RFC 2046, the last part of a multipart message, in this case
  # the HTML message, is best and preferred.
  #msg.attach(part1)
  msg.attach(part2)

  #print msg

  #Do the stuff to send the message
  server = smtplib.SMTP(smtpserver,587)
  server.ehlo()
  #server.starttls()
  server.ehlo()
  server.login(smtpuser,smtppass)
  server.set_debuglevel(1)
  server.sendmail(SENDER, [RECIPIENTS], msg.as_string())
  server.quit()

#Creates a string with a HTML table and a heading based upon parameters sent
def CreateHTMLTable(InDBResult, InTitle):
  try:
    #Start with the heading
    OutString = '<H2>' + InTitle + '</H2>\r\n'

    #See if there is anything to write
    if (InDBResult.rowcount > 0):
      #Add the table opening tag
      OutString = OutString + '<table border="1">\n\r'

      #Loop through each of the database rows, adding table rows
      for row in InDBResult:
        #New table row
        OutString = OutString + '<tr>\r\n'
        #Add the table elements
        for DBElement in row:
          OutString = OutString + '<td>' + str(DBElement) + '</td>\r\n'

        #Close the table row
        OutString = OutString + '</tr>\r\n'

      #Close the table tag
      OutString = OutString + '</table>\r\n'
      #Return the result
      return OutString
    else:
      OutString = OutString + '<p>No database results for this time period.  Come on Jerk!</p>\r\n'
      return OutString
  except:
    return 'Error creating HTML table.\r\n'

#Returns a date based upon the parameter supplied
def GetADate(DateType):
  #try:
    if (DateType == DateToday):    #Just get and return todays date
      #Get the time, format it and return it
      now = datetime.now()
      return now.strftime("%Y-%m-%d")
    elif (DateType == DateYesterday):
      now = datetime.now()
      TheDateYesterday = now - timedelta(days=1)
      return TheDateYesterday.strftime("%Y-%m-%d")
    elif (DateType == DateFirstDayOfWeek):   #The first day of the current week.  Sunday is 0. Monday is 1 etc.  We want to know how many days from Monday it is
      #Find what day of the week it is
      now = datetime.now()
      DayOfWeek = int(now.strftime("%w"))   #Get the number of the day of the week
      print 'Day of week ->>' + str(DayOfWeek)
      #See what to subtract.  Sunday is a special case
      if (DayOfWeek == 0):
        DayDelta = 6         #Monday was always 6 days ago on a Sunday!
      else:
        DayDelta = DayOfWeek - 1
      print 'Day delta ->>' + str(DayDelta)
      DateOfMonday = now - timedelta(days=DayDelta)
      print 'Monday was ->>' + str(DateOfMonday)
      return DateOfMonday.strftime("%Y-%m-%d")
    elif (DateType == DateFirstDayOfMonth):
      now = datetime.now()
      DayOfMonth =  int(now.strftime("%d"))
      DayDelta = DayOfMonth - 1
      FirstDateOfMonth = now - timedelta(days=DayDelta)
      return FirstDateOfMonth.strftime("%Y-%m-%d")
    elif (DateType == DateFirstDayOfYear):
      now = datetime.now()
      DayOfYear =  int(now.strftime("%j"))
      DayDelta = DayOfYear - 1
      FirstDateOfYear = now - timedelta(days=DayDelta)
      return FirstDateOfYear.strftime("%Y-%m-%d")

  #except:
   #return '2014-01-01'    #Just returns a default date for before I was a jerk
#%j     Day of the year as a zero-padded decimal number.
#%d     Day of the month as a zero-padded decimal number.

####################################################################
#Main part of the code
#Database stuff
db = MySQLdb.connect("localhost", "username", "password", "database")   #host,user,password,database name
curs=db.cursor()

#Run a query for today
DateForQuery = GetADate(DateToday)
JerksQuery =  'SELECT exercise, SUM(count) FROM exercise WHERE tdate = "' + DateForQuery + '" GROUP BY exercise order by sum(count);'
print JerksQuery
curs.execute (JerksQuery)

#Form the HTML Table for today
OutString = CreateHTMLTable(curs, 'JERKS EXERCISE SUMMARY - TODAY')

#Run a query for yesterday
DateForQuery = GetADate(DateYesterday)
JerksQuery =  'SELECT exercise, SUM(count) FROM exercise WHERE tdate = "' + DateForQuery + '" GROUP BY exercise order by sum(count);'
print JerksQuery
curs.execute (JerksQuery)

#Form the HTML Table for today
OutString = OutString + CreateHTMLTable(curs, 'JERKS EXERCISE SUMMARY - YESTERDAY')

#Run a query for first day of this week
DateForQuery = GetADate(DateFirstDayOfWeek)
JerksQuery =  'SELECT exercise, SUM(count) FROM exercise WHERE tdate >= "' + DateForQuery + '" GROUP BY exercise order by sum(count);'
print JerksQuery
curs.execute (JerksQuery)

#Form the HTML table for this week
OutString = OutString + CreateHTMLTable(curs, 'JERKS EXERCISE SUMMARY - THIS WEEK (Since ' + DateForQuery + ')')

#Run a query for first day of this Month
DateForQuery = GetADate(DateFirstDayOfMonth)
JerksQuery =  'SELECT exercise, SUM(count) FROM exercise WHERE tdate >= "' + DateForQuery + '" GROUP BY exercise order by sum(count);'
print JerksQuery
curs.execute (JerksQuery)

#Form the HTML table for this week
OutString = OutString + CreateHTMLTable(curs, 'JERKS EXERCISE SUMMARY - THIS MONTH (Since ' + DateForQuery + ')')

#Run a query for the first day of this year
DateForQuery = GetADate(DateFirstDayOfYear)
JerksQuery =  'SELECT exercise, SUM(count) FROM exercise WHERE tdate >= "' + DateForQuery + '" GROUP BY exercise order by sum(count);'
print JerksQuery
curs.execute (JerksQuery)

#Form the HTML table for this week
OutString = OutString + CreateHTMLTable(curs, 'JERKS EXERCISE SUMMARY - THIS YEAR (Since ' + DateForQuery + ')')

#Form and run the query for the exercise table - all time
JerksQuery = 'SELECT exercise, SUM(count) FROM exercise GROUP BY exercise order by sum(count);'
curs.execute (JerksQuery)

#Form the HTML Table
OutString = OutString + CreateHTMLTable(curs, 'JERKS EXERCISE SUMMARY - ALL TIME')

#Form and run the query for the lookup table
JerksQuery = 'select * from ' + dbLookupTable + ' order by twoletters;'
#print JerksQuery
curs.execute (JerksQuery)

#Call the def to create a table
OutString = OutString + CreateHTMLTable(curs, 'JERKS EXERCISE SHORT CODES')

#Send the email
SendEmail('Jerks Summary at ' + GetDateTime(), OutString)