Posts

#GAE Development Server Hangs updating index.yaml

/in /by

Since a recent Google App Engine SDK release ( 1.6.3?) the development server (localhost) has started appearing to hang every now and again this causes it to run very slowly. During the ‘hang’ the console output says that it’s updating index.yaml. Sometimes this ‘hang’ can last a few minutes, after a few minutes server will then continue to run as normal – it’s a real pain!

 

Google says that they will fix this problem in the next SDK release (1.6.5) but in the meantime there is a workaround, just empty out index.yaml and then the development server will stop trying to continually update it.

 

To do this either open it in an editor, delete its contents and save it, issue a command like this (when in the same directory):

[code]
echo ” > index.yaml
[/code]

just remember not to commit your emptied index.yaml file back into your source code repository!

 

Google APP Engine now supports Python 2.7

/in /by

On Monday Google announced that their cloud platform, Google App Engine now fully supports Python 2.7 which is great news!

 

More details can be found here:

 

http://googleappengine.blogspot.in/2012/02/announcing-general-availability-of.html

 

and here:

 

http://code.google.com/appengine/docs/python/python27/newin27.html

 

get_value_for_datastore() example

/in /by

When using a db.ReferenceProperty in google app engine, the function get_value_for_datastore() can be used to get the key of a referenced datastore model without causing GAE to automatically fetch the model – this can be very handy if you are trying to optimise the performance of your cloud code.

 

from google.appengine.ext import db
class A(db.Model):
    b = db.ReferenceProperty(B)
class B(db.Model):
    name = db.StringProperty()

In the above code, A contains a reference to B, given a model of type A, b’s key can be retrieved as follows:

 

a = A().get_by_id(1234354)
b_key = A.b.get_value_for_datastore(a)

This will get b’s key without fetching its model!

 

Google App Engine – Access ReferenceProperty without fetching referenced object

/in , /by

In google app engine’s cloud based Datastore db.ReferenceProperty can be used in a Model to reference another Model like this:

 


from google.appengine.ext import db
class A(db.Model):
    b = db.ReferenceProperty(B)
class B(db.Model):
    name = db.StringProperty()

In the above example A references B, if you have an object of type A you can directly access its referenced B like this:


b_name = a.b.name

When you access a.b, GAE will automatically fetch b from the datastore without you having to do anything special. This is handy, but sometimes you may not want the automatic datastore fetch to happen, instead you may just want know the referenced object’s key.

 

If you were dealing with 1000s of objects, the extra DB fetch for each object could add a huge unnecessary time overhead to your processing (for example if you were using memcache for caching, or if you already had a list of all the relevant b’s in memory).

 

So how do we retrieve the key of the referenced object without triggering an datastore fetch? To achieve we have to invoke a rather obtuse function called (wait for it): get_value_for_datastore()

 

For example, We can find out what b’s key value is like this:

 

b_key = A.b.get_value_for_datastore(a)

The above won’t cause GAE to fetch the referenced B from the datastore. Its all a bit long winded and confusing but it does seem to work!

 

Google App Engine, DeadlineExceededError snow storm in _warmup and module load

/in /by

A Google App Engine cloud application that we have been involved in developing suddenly started getting snowed under by a storm of GAE’s most excellent DeadlineExceededError exceptions.

 

The app never really suffered from this before (unless a rogue task actually took too long that is) but suddenly these errors started happening on a huge number of requests during the module load or _ah/warmup phases, they didn’t even get to run any of our code before they were killed with this most beautiful of error messages. It literally brought the app to its knees…

 

For example, we kept finding stuff like this in the logs:

 


class 'google.appengine.runtime.DeadlineExceededError'>:
Traceback (most recent call last):
File "/base/data/home/apps/appname/9993.355980156233423494/warmup.py", line 1, in module>
import appengine_config # Cache by import
File "/base/data/home/apps/appname/9993.355980156233423494/appengine_config.py", line 27, in module>
....

 

After a little research it seems that this has started happening to loads of GAE users since December 2011, have a look at this thread:

 

http://groups.google.com/group/google-appengine/browse_thread/thread/369a9a76c394c99e/976722e37ad07d0c

 

Loads of annoyed people out there having the same type of problem. The general recommendation was to move to a high replication datastore (rather than master/slave), this move is not always simple however and I believe will incur higher costs!

 

To cut a long story short, our app was dead in the water so we had no choice other than to migrate it to a high replication datastore and hope that this fixed the problem (and hope that the migration didn’t create too many new problems). Once we carried out the migration the DeadlineExceededError excpetions went away, so that was good but we were a bit peeved that our hand was forced in such a way.

 

So I don’t know what google changed but it seems that they want people on high replication or else! Once again we find ourselves seriously questioning whether google app engine is a sutibaly stable platform for deploying real-world apps – the jury is still out…

 

Anyway if you find yourselves with the same problem they you may be headed the HRD way!

 

Compare and Swap added to Google App Engine Cloud

/in /by

On Wednesday Google released v1.5.3 of their App Engine cloud platform SDK, the release notes are here. There are some interesting new additions to the API, however the one that really caught our eye is the addition of a Compare and Swap (CAS) primitive to their Memcache API.

 

A CAS primitive can be used to build all sorts of higher level synchronisation primitives like mutexes etc. and so the existence of such a primitive may be very handy for synchronisation of GAE taskqueue tasks! The SDK is currently quite lacking in such synchronisation tools so it will be interesting what can be done with this new addition.

 

Eclipse & Google App Engine – Variable references empty selection: ${project_loc}

/in /by

If, when you try to debug or run a PyDev Google app engine project in eclipse after following these instructions, you may experience this error message:

 
Variable references empty selection: ${project_loc}
 

If you get this error you should be able to make it go away by first clicking on you project in the PyDev package explorer window before trying to debug or run your project. Eclipse & PyDev can be just a little quirky…

 

Google App Engine error 500 on Upload to Cloud

/in /by

Q: I keep getting ‘Error 500’ when I try to upload / deploy to my code to the google app engine via ‘appcfg.py update’ or similar, what’s going on?

 

A: Most often this error is caused by problems with the Google cloud infrastructure and as such there is nothing you can do to fix the problem except wait and keep retrying every now and again, the problem normally goes away in an hour or 2. Do however check that your various quotas havent run out.

 

Don’t be suprised if when you view the System Status table, you see that it is all green – in our experience this status grid rarely represents the status of the cloud as experienced by developers and users. You could report the problem to Google here as suggested in the error message however if you were to report every service interruption you wouldn’t be left with very much time to do much else… ;-)

 

Google App Engine – TransientError in Cloud

/in /by

We were engulfed by a blizzard of Google task queue ‘TransientErrors’ over the weekend, the ‘TransientError’ is one most ellusive of google app engine errors. This is what the official documentation has to say of it:

 

exception TransientError(Error)
There was a transient error while accessing the queue. Please try again later.

This very detailed description has left many scratching their heads wondering (a) Why they are getting the errors and (b) What should they do about them!

 

So far, the best description I could find of the error is here:
http://osdir.com/ml/GoogleAppEngine/2009-06/msg01337.html

 

A TransientError is an unexpected but transient failure. Typically it
is a deadlined or otherwise missed connection between two backends in
our system. We distinguish TransientError from InternalError to say
that transient errors failed but were expected to succeed (and
retrying will probably work), whereas InternalError is quite
unexpected and retries will have no effect.

 

This seems to suggest that you shouldn’t worry too much about them as they happen rarely and the automatic retry should work – no harm done…

 

In our experience however it seems that the error is far from transient in that it often occurs in batches with the retries experiencing the error too – it causes our app to grind to a halt, the whole thing tends to snowball. It is like something fairly serious goes wrong with the google cloud and things just stop working.

 

Does anybody know if there is any reliable way of handling these errors when they occur?

 

And more seriously, can an application be made reliable on the google cloud with nasty undocumented errors like this cropping up in batches every few months? I suppose software development for cloud environments is still really only in its infancy, developers and platform suppliers have a lot to learn, but unexplained stealth errors like the TransientError really doesn’t help the situation!

Store Datastore Models in Google App Engine Cloud

/1 Comment/in /by

Yesterday I came across this gem of a Blog post on Nick Johnson’s Blog about how to go about efficiently storing Google App Engine datastore objects in Memcache on the cloud.

 

I had previously read that the Datastore models were large and carried around a lot of baggage (software baggage that is, not the stuff on wheels!) – this is one reason why you are discouraged from passing them as arguments to tasks on task-queues.

 

This blog shows you how to extract and store only the essential info from the objects for storage in memcache using th methids model_to_protobuf () and model_from_protobuf()

 

Anyway, enough chat – here’s the post:

 

http://blog.notdot.net/2009/9/Efficient-model-memcaching