Posts

get_value_for_datastore() example

/in /by

When using a db.ReferenceProperty in google app engine, the function get_value_for_datastore() can be used to get the key of a referenced datastore model without causing GAE to automatically fetch the model – this can be very handy if you are trying to optimise the performance of your cloud code.

 

from google.appengine.ext import db
class A(db.Model):
    b = db.ReferenceProperty(B)
class B(db.Model):
    name = db.StringProperty()

In the above code, A contains a reference to B, given a model of type A, b’s key can be retrieved as follows:

 

a = A().get_by_id(1234354)
b_key = A.b.get_value_for_datastore(a)

This will get b’s key without fetching its model!

 

Google App Engine – Access ReferenceProperty without fetching referenced object

/in , /by

In google app engine’s cloud based Datastore db.ReferenceProperty can be used in a Model to reference another Model like this:

 


from google.appengine.ext import db
class A(db.Model):
    b = db.ReferenceProperty(B)
class B(db.Model):
    name = db.StringProperty()

In the above example A references B, if you have an object of type A you can directly access its referenced B like this:


b_name = a.b.name

When you access a.b, GAE will automatically fetch b from the datastore without you having to do anything special. This is handy, but sometimes you may not want the automatic datastore fetch to happen, instead you may just want know the referenced object’s key.

 

If you were dealing with 1000s of objects, the extra DB fetch for each object could add a huge unnecessary time overhead to your processing (for example if you were using memcache for caching, or if you already had a list of all the relevant b’s in memory).

 

So how do we retrieve the key of the referenced object without triggering an datastore fetch? To achieve we have to invoke a rather obtuse function called (wait for it): get_value_for_datastore()

 

For example, We can find out what b’s key value is like this:

 

b_key = A.b.get_value_for_datastore(a)

The above won’t cause GAE to fetch the referenced B from the datastore. Its all a bit long winded and confusing but it does seem to work!

 

Google App Engine, DeadlineExceededError snow storm in _warmup and module load

/in /by

A Google App Engine cloud application that we have been involved in developing suddenly started getting snowed under by a storm of GAE’s most excellent DeadlineExceededError exceptions.

 

The app never really suffered from this before (unless a rogue task actually took too long that is) but suddenly these errors started happening on a huge number of requests during the module load or _ah/warmup phases, they didn’t even get to run any of our code before they were killed with this most beautiful of error messages. It literally brought the app to its knees…

 

For example, we kept finding stuff like this in the logs:

 


class 'google.appengine.runtime.DeadlineExceededError'>:
Traceback (most recent call last):
File "/base/data/home/apps/appname/9993.355980156233423494/warmup.py", line 1, in module>
import appengine_config # Cache by import
File "/base/data/home/apps/appname/9993.355980156233423494/appengine_config.py", line 27, in module>
....

 

After a little research it seems that this has started happening to loads of GAE users since December 2011, have a look at this thread:

 

http://groups.google.com/group/google-appengine/browse_thread/thread/369a9a76c394c99e/976722e37ad07d0c

 

Loads of annoyed people out there having the same type of problem. The general recommendation was to move to a high replication datastore (rather than master/slave), this move is not always simple however and I believe will incur higher costs!

 

To cut a long story short, our app was dead in the water so we had no choice other than to migrate it to a high replication datastore and hope that this fixed the problem (and hope that the migration didn’t create too many new problems). Once we carried out the migration the DeadlineExceededError excpetions went away, so that was good but we were a bit peeved that our hand was forced in such a way.

 

So I don’t know what google changed but it seems that they want people on high replication or else! Once again we find ourselves seriously questioning whether google app engine is a sutibaly stable platform for deploying real-world apps – the jury is still out…

 

Anyway if you find yourselves with the same problem they you may be headed the HRD way!

 

Store Datastore Models in Google App Engine Cloud

/1 Comment/in /by

Yesterday I came across this gem of a Blog post on Nick Johnson’s Blog about how to go about efficiently storing Google App Engine datastore objects in Memcache on the cloud.

 

I had previously read that the Datastore models were large and carried around a lot of baggage (software baggage that is, not the stuff on wheels!) – this is one reason why you are discouraged from passing them as arguments to tasks on task-queues.

 

This blog shows you how to extract and store only the essential info from the objects for storage in memcache using th methids model_to_protobuf () and model_from_protobuf()

 

Anyway, enough chat – here’s the post:

 

http://blog.notdot.net/2009/9/Efficient-model-memcaching