Tuesday, 24 March 2015

Sleep Infographic Using Raspberry Pi and Fitbit API

Infographics.  They're everywhere.  Here's one I did (it's my first one so be kind!):


So why did I do this?  In a previous post I spotted how the Fitbit API contains lots of delicious Geek data for me to analyse.  Hence I wanted to extract this data, analyse it and present it.  My inclination is to present  detailed charts and tables of data but I decided to present it as an infographic as they seem to be everywhere these days.

In terms of my Fitbit Charge HR and sleep measurement, I think it's reasonably accurate.  If I've had a good night sleep then this is generally registered.  Similar for a bad nights sleep.  When I get up in the night  to answer nature's call it's always logged.  Needs more analysis to be sure though...

So the process was:
  • Write a Python script on my Raspberry Pi to access the Fitbit API.
  • Have the script output data in a raw format.
  • Use Excel to process and present the data.  (Excel is not very Geek Dad but I didn't have time to learn a different method).

Full code is at the bottom but highlights from it are summarised below.

Accessing the API
Uses methods described in this previous post.  Snippet:

#Get a client
authd_client = fitbit.Fitbit(CLIENT_KEY, CLIENT_SECRET, resource_owner_key=USER_KEY, resource_owner_secret=USER_SECRET)

Calculating How Many Days to Process For
The Fitbit API takes a single date as a parameter.  I wanted to process for every day that I'd had the Fitbit.  Hence I wrote a function called CountTheDays() to count how many days had passed between the day the script is run and the day I got my Fitbit.

Looping through Each Day
Having the number of days to process for, I could then create a for loop to look at each day in turn.  

This snippet:

for i in range(DayCount,-1,-1):
  #Get the date to process
  DateForAPI = ComputeADate(i)

Means you loop with the loop var i decrementing each time.  It starts off back at the first day I got the Fitbit and finishes on today's date.

The code then gets an actual date using ComputeADate(i) which means it computes a date that is i days different from today.

Getting the API Data
This method gets the API data for you:
fitbit_sleep = authd_client._COLLECTION_RESOURCE('sleep',DateForAPI)

...and the resulting JSON structure starts like this:
{u'sleep': [{u'logId': 778793259, u'isMainSleep': True, u'minutesToFallAsleep': 0, u'awakeningsCount': 26, u'minutesAwake': 45, u'timeInBed': 483, u'minutesAsleep': 417, u'awakeDuration': 11, u'efficiency': 90, u'startTime': u'2015-03-16T22:18:00.000', u'restlessCount': 29, u'duration': 28980000, u'restlessDuration': 52, u'minuteData': 

....has a bunch of "per minute" records like this....

{u'value': u'1', u'dateTime': u'22:38:00'}, {u'value': u'1', u'dateTime': u'22:39:00'}, {u'value': u'1', u'dateTime': u'22:40:00'}

...then finishes with more summary data like this.

u'awakeCount': 3, u'minutesAfterWakeup': 1}], u'summary': {u'totalTimeInBed': 483, u'totalMinutesAsleep': 417, u'totalSleepRecords': 1}}

Extracting Elements of the JSON
Using my patented trial and error method I managed to work out you pick out different parts of the JSON like this:

#Get the total minutes in bed value.  This will control our loop that gets the sleep
MinsInBed = fitbit_sleep['sleep'][0]['timeInBed']

#Get the total sleep value
MinsAsleep = fitbit_sleep['sleep'][0]['minutesAsleep']

...and that the [timeInBed] field could be used to determine how many per minute records there are.  Hence you can loop, logging each of the per minute records:

#Loop through the lot
  for i in range(0,MinsInBed):
    SleepVal = fitbit_sleep['sleep'][0]['minuteData'][i]['value']
    TimeVal = fitbit_sleep['sleep'][0]['minuteData'][i]['dateTime']
    MyFile.write(DateForAPI + ',' + TimeVal + ',' + SleepVal + '\r\n')
    
If this is the first loop iteration, grab the time which is when Fitbit thought I went to bed:

    if (i == 0):
      FirstTimeInBed = TimeVal

Logging to File
In the loop above I log the per minute data to file.  I also log summary data to file like this:

#Write a log of summary data
  SummaryFile.write(DateForAPI + ',' + str(MinsInBed) + ',' + str(MinsAsleep) + ',' + FirstTimeInBed + '\r\n')

One of these logs is written per day.

Analysing the Summary Data
I'll leave analysis of the per minute data until later, (can't think what to do with it now).  The summary data looks like this:

2015-01-27,478,434,22:21:00
2015-01-28,447,420,22:50:00
2015-01-29,491,446,22:11:00
2015-01-30,414,359,23:29:00


The format is:
<Date>,<Minutes in bed>,<Minutes asleep>,<Bed Time>

So for the infographic I used Excel to:

1-Add up all the time asleep values to give the total time asleep (expressed in hours).

2-Worked out the average time asleep each night (again expressed in hours).

3-Used a pivot table to examine the data on a per day of week basis.


So the Saturday to Sunday sleep is my best at a mean of 7.6 hours.  Sunday to Monday sleep is the worst at 5.8 hours (this is because I do a swim training session which finishes late on a Sunday night so I go to bed later and am also a bit wired so it takes me longer to get to sleep). 

4-Worked out an "efficiency" figure by calculating what proportion of time in bed I'm actually asleep for.

5-Looked at the frequency distribution of Fitbit logged bed time:


The mode value of 22:12 went on the infographic.  Latest bed time is still before midnight.  Rock 'n' roll baby!

Full Code
Secret stuff changed of course.  Look at my previous posts to see how to authenticate etc.

#V1 - Minute by minute data
#V2 - Summary data added

import fitbit
from datetime import datetime, timedelta

#Constants
CLIENT_KEY = 'uptownfunkyouup'
CLIENT_SECRET = 'livinlifeinthecity'
USER_KEY = 'iliketomoveitmoveit'
#USER_KEY = 'iliketomoveitmove'
USER_SECRET = 'iliketohuhmoveit'

#The first date I used Fitbit
FirstFitbitDate = '2015-01-27'

#Determine how many days to process for.  First day I ever logged was 2015-01-27
def CountTheDays():
  #See how many days there's been between today and my first Fitbit date.
  now = datetime.now()                                         #Todays date
  FirstDate = datetime.strptime(FirstFitbitDate,"%Y-%m-%d")    #First Fitbit dat
e as a Python date object

  #Calculate difference between the two and return it
  return abs((now - FirstDate).days)

#Produce a date in yyyy-mm-dd format that is n days before today's date (where n is a passed parameter)
def ComputeADate(DaysDiff):
  #Get today's date
  now = datetime.now()

  #Compute the difference betwen now and the day difference paremeter passed
  DateResult = now - timedelta(days=DaysDiff)
  return DateResult.strftime("%Y-%m-%d")

#Get a client
authd_client = fitbit.Fitbit(CLIENT_KEY, CLIENT_SECRET, resource_owner_key=USER_KEY, resource_owner_secret=USER_SECRET)

#Find out how many days to compute for
DayCount = CountTheDays()

#Open a file to write the output - minute by minute and summary
FileToWrite = '/home/pi/sleep/' + 'minuteByMinute_' + datetime.now().strftime("%Y-%m-%d") + '.csv'
MyFile = open(FileToWrite,'w')
SummaryFileToWrite = '/home/pi/sleep/' + 'summary_' + datetime.now().strftime("%Y-%m-%d") + '.csv'
SummaryFile = open(SummaryFileToWrite,'w')

#Process each one of these days stepping back in the for loop and thus stepping up in time
for i in range(DayCount,-1,-1):
  #Get the date to process
  DateForAPI = ComputeADate(i)

  #Tell the user what is happening
  print 'Processing this date: ' + DateForAPI

  #Get sleep
  fitbit_sleep = authd_client._COLLECTION_RESOURCE('sleep',DateForAPI)

  #Get the total minutes in bed value.  This will control our loop that gets the sleep
  MinsInBed = fitbit_sleep['sleep'][0]['timeInBed']

  #Get the total sleep value
  MinsAsleep = fitbit_sleep['sleep'][0]['minutesAsleep']

  #Loop through the lot
  for i in range(0,MinsInBed):
    SleepVal = fitbit_sleep['sleep'][0]['minuteData'][i]['value']
    TimeVal = fitbit_sleep['sleep'][0]['minuteData'][i]['dateTime']
    MyFile.write(DateForAPI + ',' + TimeVal + ',' + SleepVal + '\r\n')
    #If this is the first itteration of the loop grab the time which says when I got to bed
    if (i == 0):
      FirstTimeInBed = TimeVal

  #Write a log of summary data
  SummaryFile.write(DateForAPI + ',' + str(MinsInBed) + ',' + str(MinsAsleep) + ',' + FirstTimeInBed + '\r\n')
#We're now at the end of theloops.  Close the file
MyFile.close()
SummaryFile.close()