If you are having problems creating task-queue tasks after upgrading the Google App Engine SDK to 1.7.2 on a Mac, then I am reliably informed that running the following should help the situation:
sudo echo "127.0.0.1 b1.localhost" >> /etc/hosts
Thanks Lukasz!
Since a recent Google App Engine SDK release ( 1.6.3?) the development server (localhost) has started appearing to hang every now and again this causes it to run very slowly. During the ‘hang’ the console output says that it’s updating index.yaml. Sometimes this ‘hang’ can last a few minutes, after a few minutes server will then continue to run as normal – it’s a real pain!
Google says that they will fix this problem in the next SDK release (1.6.5) but in the meantime there is a workaround, just empty out index.yaml and then the development server will stop trying to continually update it.
To do this either open it in an editor, delete its contents and save it, issue a command like this (when in the same directory):
[code]
echo ” > index.yaml
[/code]
just remember not to commit your emptied index.yaml file back into your source code repository!
On Monday Google announced that their cloud platform, Google App Engine now fully supports Python 2.7 which is great news!
More details can be found here:
http://googleappengine.blogspot.in/2012/02/announcing-general-availability-of.html
and here:
http://code.google.com/appengine/docs/python/python27/newin27.html
When using a db.ReferenceProperty in google app engine, the function get_value_for_datastore() can be used to get the key of a referenced datastore model without causing GAE to automatically fetch the model – this can be very handy if you are trying to optimise the performance of your cloud code.
from google.appengine.ext import db
class A(db.Model):
b = db.ReferenceProperty(B)
class B(db.Model):
name = db.StringProperty()
In the above code, A contains a reference to B, given a model of type A, b’s key can be retrieved as follows:
a = A().get_by_id(1234354)
b_key = A.b.get_value_for_datastore(a)
This will get b’s key without fetching its model!
In google app engine’s cloud based Datastore db.ReferenceProperty can be used in a Model to reference another Model like this:
from google.appengine.ext import db
class A(db.Model):
b = db.ReferenceProperty(B)
class B(db.Model):
name = db.StringProperty()
In the above example A references B, if you have an object of type A you can directly access its referenced B like this:
b_name = a.b.name
When you access a.b, GAE will automatically fetch b from the datastore without you having to do anything special. This is handy, but sometimes you may not want the automatic datastore fetch to happen, instead you may just want know the referenced object’s key.
If you were dealing with 1000s of objects, the extra DB fetch for each object could add a huge unnecessary time overhead to your processing (for example if you were using memcache for caching, or if you already had a list of all the relevant b’s in memory).
So how do we retrieve the key of the referenced object without triggering an datastore fetch? To achieve we have to invoke a rather obtuse function called (wait for it): get_value_for_datastore()
For example, We can find out what b’s key value is like this:
b_key = A.b.get_value_for_datastore(a)
The above won’t cause GAE to fetch the referenced B from the datastore. Its all a bit long winded and confusing but it does seem to work!
I gave a talk some time ago to the Google Users Group in Dublin about how to go about coding up the the integration of paypal redirect payments into a Google App Application with python, if anybody’s intersted the talk slides are here:
In summary its possible and not that difficult!
A Google App Engine cloud application that we have been involved in developing suddenly started getting snowed under by a storm of GAE’s most excellent DeadlineExceededError exceptions.
The app never really suffered from this before (unless a rogue task actually took too long that is) but suddenly these errors started happening on a huge number of requests during the module load or _ah/warmup phases, they didn’t even get to run any of our code before they were killed with this most beautiful of error messages. It literally brought the app to its knees…
For example, we kept finding stuff like this in the logs:
class 'google.appengine.runtime.DeadlineExceededError'>:
Traceback (most recent call last):
File "/base/data/home/apps/appname/9993.355980156233423494/warmup.py", line 1, in module>
import appengine_config # Cache by import
File "/base/data/home/apps/appname/9993.355980156233423494/appengine_config.py", line 27, in module>
....
After a little research it seems that this has started happening to loads of GAE users since December 2011, have a look at this thread:
Loads of annoyed people out there having the same type of problem. The general recommendation was to move to a high replication datastore (rather than master/slave), this move is not always simple however and I believe will incur higher costs!
To cut a long story short, our app was dead in the water so we had no choice other than to migrate it to a high replication datastore and hope that this fixed the problem (and hope that the migration didn’t create too many new problems). Once we carried out the migration the DeadlineExceededError excpetions went away, so that was good but we were a bit peeved that our hand was forced in such a way.
So I don’t know what google changed but it seems that they want people on high replication or else! Once again we find ourselves seriously questioning whether google app engine is a sutibaly stable platform for deploying real-world apps – the jury is still out…
Anyway if you find yourselves with the same problem they you may be headed the HRD way!
Just catching up with things after Christmas, one thing that escaped my notice in December was that Google released version 1.6.1 of their Cloud App Engine SDK on 13th. of December. There’s not a huge amount of interest really in this release, but two items did catch my eye:
You can now select how much CPU power and memory is available for your fronted instances, you can select this from a small set of presets in the app dashboard – however google warns that selecting a higher preset will incur extra cost! We sometimes run into soft memory limit problems on our frontends so this may be of great use to us, but I do worry about the $$ !
Google have also released a new experimental document conversion API which they say contains OCR functionality – I look forward to testing this and will be very interested to see how well the OCR preforms as it can be a very tricky thing to get right! I suppose that this is an API to the same OCR functionality that they have in google docs….
Oh and they have added an API for programmatically reading the application logs, this may come in handy for automated testing!
More details about the release can be found here:
http://googleappengine.blogspot.com/2011/12/app-engine-161-released.html
Yesterday Google released v1.5.5 of their App Engine SDK. There are loads of nice changes in this release including the easing of some previously annoying limits, the one I am especially happy about is the doubling of the frontend request deadline from 30s to 60s, this will make our life a lot easier!
More details can be found here:
http://googleappengine.blogspot.com/2011/10/app-engine-155-sdk-release.html
On Wednesday Google released v1.5.3 of their App Engine cloud platform SDK, the release notes are here. There are some interesting new additions to the API, however the one that really caught our eye is the addition of a Compare and Swap (CAS) primitive to their Memcache API.
A CAS primitive can be used to build all sorts of higher level synchronisation primitives like mutexes etc. and so the existence of such a primitive may be very handy for synchronisation of GAE taskqueue tasks! The SDK is currently quite lacking in such synchronisation tools so it will be interesting what can be done with this new addition.
Address: 43 Churchgate, Wicklow, Co. Wicklow, Ireland.
Telephone: +353 87 2236429
email: info@ridgesolutions.ie