Posts

Python script to query NTP status and return it as JSON

Setting up and automatically keeping an eagle eye on NTP from software can be a bit of a (difficult) black art. A common way to query NTP status is to run:

ntpq -p

This returns some NTP status information. It’s output can be a bit difficult to work with from your software, so I have included a small python 3 script which runs runs “ntpq -p” parses, and prints the output in Json format. So instead of shelling out to ntpq you can shell out to this script and deal with the resulting json string instead… This can be handy if you don’t have access to a Python NTP library.

#!/usr/bin/python
#   Copyright 2018 Kevin Godden
#
#   Licensed under the Apache License, Version 2.0 (the "License");
#   you may not use this file except in compliance with the License.
#   You may obtain a copy of the License at
#
#       http://www.apache.org/licenses/LICENSE-2.0
#
#   Unless required by applicable law or agreed to in writing, software
#   distributed under the License is distributed on an "AS IS" BASIS,
#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#   See the License for the specific language governing permissions and
#   limitations under the License.
import subprocess
import json
import re
import sys
# Shell out to 'ntpq -p'
proc = subprocess.Popen(['ntpq', '-p'], stdout=subprocess.PIPE)
# Get the output
stdout_value = proc.communicate()[0].decode("utf-8")
#remove the header lines
start = stdout_value.find("===\n")
if start == -1:
    # We may be running on windows (\r\n), try \r...
    start = stdout_value.find("===\r")
    if start == -1:
        # No, go, exit with error
        result = {'query_result': 'failed', 'data': {}}
        print(json.dumps(result))
        sys.exit(1)
# Get the data part of the string
#pay_dirt = stdout_value[start+4:]
pay_dirt = stdout_value[start:]
# search for NTP line starting with * (primary server)
exp = ("\*((?P\S+)\s+)"
       "((?P\S+)\s+)"
       "((?P\S+)\s+)"
       "((?P\S+)\s+)"
       "((?P\S+)\s+)"
       "((?P\S+)\s+)"
       "((?P\S+)\s+)"
       "((?P\S+)\s+)"
       "((?P\S+)\s+)"
       "((?P\S+)\s+)")
regex = re.compile(exp, re.MULTILINE)
r = regex.search(pay_dirt)
# Did we get anything?
if not r:
    # No, try again without the * at the beginning, get
    # the first entry instead
    exp = (" ((?P\S+)\s+)"
           "((?P\S+)\s+)"
           "((?P\S+)\s+)"
           "((?P\S+)\s+)"
           "((?P\S+)\s+)"
           "((?P\S+)\s+)"
           "((?P\S+)\s+)"
           "((?P\S+)\s+)"
           "((?P\S+)\s+)"
           "((?P\S+)\s+)")
    regex = re.compile(exp, re.MULTILINE)
    r = regex.search(pay_dirt)
data = {}
if r:
    data = r.groupdict()
# Output Result
result = {'query_result': 'ok' if r else 'failed', 'data': data}
print(json.dumps(result))

GitHub repo is here.

Streaming mjpeg video with a web2py and python server

Recently I investigated whether I could implement software streaming of mjpeg video on a camera that uses web2py and python as its HTTP interface. Web2py makes most common operations quite straightforward and streaming is no exception, although I had to go digging for some details, this post along with SergeyPo’s answer to this question were of immense help.

mjpeg or Motion JPEG streaming over HTTP is a popular way of streaming video from IP cameras. If your camera has already gone to the effort of jpeg’ing its frames then it is reasonably cheap to stream them out via software over HTTP as an mjpeg video stream (as opposed to encoding to h264 etc.).

Mjpeg can be viewed natively by most browsers (except Internet Explorer) and can also be viewed on video viewers like VLC etc. An mjpeg stream is quite simple, it really just contains the individual jpeg frames’ data separated by a defined frame boundary marker.

Web2py takes care of chunked streaming via the stream() function, we just have to write a file object that we pass to stream(), this will take care of loading the jpeg frames and providing the data back to stream(), which in turn will stream this down the line to the client.

So to create an mjpeg stream we:

1.) Add a web2py handler for the stream, for example /mjpeg

2.) Add the following to the response headers: “multipart/x-mixed-replace; boundary=the_answer_is_42”
“the_answer_is_42” is just a made-up boundary marker which we hope won’t appear in the jpeg frame data, it can be changed to something else.

3.) Call stream() passing an instance of our MJPEGStreamer file object (see #4!), and a fairly arbitrary chunk size.

4.) Define a file compatible object that will loop ‘forever’ and provide the data to stream(), it will provide the data for the individual jpeg frames as well as insert the frame boundaries (–the_answer_is_42) between frames, it also inserts the headers for the individual frames (Content-Type: image/jpeg).

The following provides an overview of a stream implementation. Here we imagine that the latest jpeg frame is to be found in a ram-disk file called image.jpg, we load this data so that we can stream it to the client.

So in our web2py controller we have something like:

#
# Proof of concept web2py mjpeg streamer
#
def mjpeg():
    # Set the initial response header
    response.headers['Content-Type'] = 'multipart/x-mixed-replace; boundary=the_answer_is_42'
    # stream the mjpeg data (via MJPEGStreamer) with a chunk size of 
    # 30000, the chuck size must be less than the size of the smallest 
    # jpeg frame or the stream will stall....
    return response.stream(MJPEGStreamer(), 30000)
class MJPEGStreamer(): 
    def __init__(self): 
        self.iterator = self.make_iterator() 
    def make_iterator(self): 
        while True:
            out = ''
            # Read a jpeg frame from image.jpg
            data =  open('/media/ram/image.jpg', 'rb').read()
            
            # Add the frame boundary to the output
            out += "--the_answer_is_42\r\n"
            
            # Add the jpg frame header
            out += "Content-Type: image/jpeg\r\n"
            # Add the frame content length
            out += "Content-length: "+str(len(data))+"\r\n\r\n"
            # Add the actual binary jpeg frame data
            out += data
            
            yield out  
            # Sleep for a bit..
            time.sleep(0.07)
    # stream() calls read() to get data.
    def read(self, n):
        # Get some more stream data
        n = self.iterator.next() 
        return n 
#
#

To view the mjpeg stream, open VLC viewer and choose Media / Open Network Stream and provide the stream URL (e.g. http://my_streaming_host/mjpeg) and hit play…

I found that it was important to set the chunk size to be (a good bit?) less than the data size of the smallest frame to be sent or else the stream will stall…

Send event to intercom.ie – example python code

So as mentioned in a previous post I have been doing some python software work to integrate a client’s could application with intercom.io. The intercom REST interface is nicely designed and quite straightforward to use.

Applications send events to their intercom account (with optional event data) when their users do interesting things, intercom can be configured with rules so that messages are automatically sent in response to a particular combination of events.

It is a nice idea that shows great potential, however the intercom rules engine is quite limited and probably needs quite a bit of work before it is really useful, for example, at the time of writing, it is not possible to include any of the event data as part of the decision process of a rule.

Anyway to post an event to intercom.io you do an HTTP POST to /events passing some data, including: the event name, the user’s email address a time-stamp. You can also send optional data that will be stored with the event (but which can’t bu used as part of a messaging rule).

The following code defines a function called raise_event() which can be used like this:

raise_event('project_created', email_address,
		{'project_id': id, 'project_name':name})

This raises an event named ‘project_created’ with some extra data: project id and project name.

import httplib
import json
APP_ID = 'Place your App Id here'
API_KEY = 'Place your API key here'
API_URL_STUB = 'api.intercom.io'

def raise_event(name, user_email, extra_data = None):
 
	return _api_post('events',
			 {'event_name': name,
			  'created_at':_timestamp(),
			  'user_id': user_email,
			  'email': user_email,
			  'metadata': extra_data,
					 }
	)
def _api_post(endpoint, data):
    userAndPass = b64encode(b'%s:%s' % (APP_ID, API_KEY)).decode("ascii")
    headers = {"content-type": 'application/json', 'Authorization' : 'Basic %s' %  userAndPass}
    h = httplib.HTTPSConnection(API_URL_STUB)
    url = '/%s' % endpoint
    h.request('POST', url, body=json.dumps(data), headers=headers)
    response = h.getresponse()
def _timestamp():
    now = datetime.now()
    return long(time.mktime(now.timetuple()))

Of course this is all very interesting but for a real python API wrapper you should probably checkout python-intercom here:

https://pypi.python.org/pypi/python-intercom

web2py – External connections to your Web App from the network on port 80

If you don’t want to use apache or the like to publish your web2py app then you can use web2py’s own little web server (rocket) by running web2py.py like this:

python web2py.py
 

This won’t accept external connections however, and it also listens on port 8000 rather than port 80.

 

So, to allow external connections from web clients on port 80, run web2py.py as follows:

python web2py.py -i  0.0.0.0 -p 80

I keep forgetting this, so it’s getting written down for once and for all!

 

Raspberry Pi – Restart / Shutdown your Pi from Python code

Here’s something that you probably won’t want to do, but how can you restart or shut-down your Pi from software in python code?

 

Well it’s both quite simple and quite tricky all at the same time, but in summary you can run some python code like this to shell out to the shutdown command:

 
def restart():
    command = "/usr/bin/sudo /sbin/shutdown -r now"
    import subprocess
    process = subprocess.Popen(command.split(), stdout=subprocess.PIPE)
    output = process.communicate()[0]
    print output
 

Executing this python code is like running the following in bash:

 
sudo shutdown -r now
 

Which will reboot the system. Note that the shutdown command is sudo’d and that the full paths to sudo and shutdown are specified.

 

Now for this code to work it must be run in the context of a user that has sufficient privileges (e.g. the pi user). If the python code is being executed by a user that does not have the necessary permissions then the it will not work – for example when running this code as part of my Web2py application the code initially had no effect as it was being run as the apache www-data user.

 

To fix this problem you need to add the user in question (in my case www-data) to the sudoers file (for the shutdown command). Edit the sudoers file by issuing the following command in bash:

 
sudo visudo
 

Then add the following lines to the end of the file remembering to ‘www-data’ to your target user:

 
 
www-data ALL=/sbin/shutdown
www-data ALL=NOPASSWD: /sbin/shutdown
 
 

And hopefully that should do it!

 

Web2py and python – Change to model definition may break web site

In Web2py if you want to change a model’s definition, you can first of all change the model’s software definition via its python code and then, (depending on your migration settings), the next time you launch your web app, web2py will attempt to alter your database structure to match your updated model.

 

If this works, then it works and your database will reflect your new model, however sometimes web2py can’t make the necessary changes and your app may not subsequently load any more at all. If this happens then I have found the best way to fix the app is to delete all of the files in the app’s database directory – the next time you launch your app web2py will recreate your DB from scratch and it should launch again.

 

Now this is fine assuming that your database didn’t have loads of important data in it that you didn’t want to loose. So, to get around this problem, before trying to change a model’s definition I export all of the data in the database to a file (using a tool like this) so that if the problem happens and I have to get web2py to recreate the database ‘in its own image’ I can later reimport the previously saved data…

 

Web2py / Apache – admin disabled because no admin password

If you are running Web2py on Apache and get this error message when you try to log into the admin interface (/admin):

 

admin disabled because no admin password

 

Then you may have forgotten to provide a password during set-up and will have to provide one now to proceed to the admin screens.

 

To do this you need to can do the following:

 

Stop Apache:
[crayon]
apachectl -k stop
[/crayon]
cd to the web2py directory and run web2py.py specifying a new password (greater than 4 characters long?) like this:
[crayon]
sudo python web2py.py -a the_new_passsword
[/crayon]
Then kill the web2py, e.g. via ctrl-c, and then copy parameters_8000.py to parameters_443.py, like this:
[crayon]
sudo cp parameters_8000.py parameters_443.py
[/crayon]
Restart the apache server:
[crayon]
sudo apachectl -k restart
[/crayon]
After doing all of this when you visit /admin you should be asked to provide your password and you should be able to login – well it worked for me anyway! ;-)

 

Python & Web2py for Smart Camera user interface?

I am currently in the planning phase of the software build for a new machine vision smart camera. The camera will have an ARM soft core and will run linux. Part of the planning involves thinking about a user interface to control and configure the camera – at the moment I am leaning towards an HTML5 based interface built using python, web2py (or flask) along with a twitter bootstrap based theme.

 

I did briefly toy with the idea of bringing node.js into the mix, but I just can’t bring myself around to the idea, I have a love hate relationship with javascript, that sometimes leans more to the ‘hate’ end of the spectrum!

 

Now here’s the fun bit – I don’t yet have my hands on the camera hardware so I think I will try to run up a proof of concept on my raspberry-pi!

 

Although the pi will be a good bit less powerful than the smart cam it should provide a useful reference point…

 

Put this little fella to work!

 

#Python – Splitting a List into Sub Lists

I found this handy post which shows how to neatly split a list in python into a list of sub-lists:

 

http://www.garyrobinson.net/2008/04/splitting-a-pyt.html

 

Cheers Gary!

 

#Python Convert datetime to Unix Epoch Timestamp

There doesn’t seem to be a way to get a Unix timestamp directly from a datetime object in Python, the best workaround example I could find is:

 
from datetime import datetime
import time
now = datetime.now()
u = str(long(time.mktime(now.timetuple())))

This gets the timestamp as a string, it seems a bit long winded, so if anybody knows a better way please let me know!