Posts

Building snappy compression library on windows with Visual Studio 2012

/in /by

Snappy is a little fast software compression library by google. It eschews high compression rates for speed, it is very easy to use. Building it on windows for use with visual studio etc. is relatively straight-forward due to this github project by kmanley which provides it with a visual studio solution.

https://github.com/kmanley/snappy-msvc

To build the snappy static libraries, follow the instructions on the github page (README.rst).

If you get a build error like this:

.
Error	15	error C2061: syntax error : identifier 'ssize_t'
.

Then copy the following code and place it at the top of the snappy.h header file (say, after #define UTIL_SNAPPY_SNAPPY_H__):

//
#include 
typedef SSIZE_T ssize_t;
//

Once it has built, snappy.lib can be found in the Debug or Release folders, note that the debug build of the snappy lib is very slow so don’t do any tome testing with it! The release build screams along!

To use snappy in your C/C++ project you will need to #include “snappy.h” (which in-turn uses “snappy-stubs-public.h”) and also add snappy.lib to your project. Here is a little silly example:

//
#include 
#include "snappy.h"
void test_compress() {
	std::string in("This is not a very long string at all!!!  So the compressed output may be larger than its size, but don't worry it will work well for longer things");
	printf("Abut to compress string (size is %d)\n", in.size());
	// 
	// Compress the string...
	//
	std::string out;
	auto comp_size = snappy::Compress(in.c_str(), in.size(), &out);
	
	printf("Compressed size is %d\n", comp_size);
	//
	// Uncompress it..
	//
	std::string plain;
	snappy::Uncompress(out.data(), comp_size, &plain);
	printf("Un-compressed size is %d\n", plain.size());
	printf(plain.c_str());
}
//

If you get an error like this when you try to compile your client code:

.
Error	1	error LNK2038: mismatch detected for '_ITERATOR_DEBUG_LEVEL': value '2' doesn't match value '0' 
.

Then it means you are trying to use the debug version of snappy.lib in a release build or vice versa, use the correct version and the error should go away….

Many thanks to kmanley!

Google App Engine, DeadlineExceededError snow storm in _warmup and module load

/in /by

A Google App Engine cloud application that we have been involved in developing suddenly started getting snowed under by a storm of GAE’s most excellent DeadlineExceededError exceptions.

 

The app never really suffered from this before (unless a rogue task actually took too long that is) but suddenly these errors started happening on a huge number of requests during the module load or _ah/warmup phases, they didn’t even get to run any of our code before they were killed with this most beautiful of error messages. It literally brought the app to its knees…

 

For example, we kept finding stuff like this in the logs:

 


class 'google.appengine.runtime.DeadlineExceededError'>:
Traceback (most recent call last):
File "/base/data/home/apps/appname/9993.355980156233423494/warmup.py", line 1, in module>
import appengine_config # Cache by import
File "/base/data/home/apps/appname/9993.355980156233423494/appengine_config.py", line 27, in module>
....

 

After a little research it seems that this has started happening to loads of GAE users since December 2011, have a look at this thread:

 

http://groups.google.com/group/google-appengine/browse_thread/thread/369a9a76c394c99e/976722e37ad07d0c

 

Loads of annoyed people out there having the same type of problem. The general recommendation was to move to a high replication datastore (rather than master/slave), this move is not always simple however and I believe will incur higher costs!

 

To cut a long story short, our app was dead in the water so we had no choice other than to migrate it to a high replication datastore and hope that this fixed the problem (and hope that the migration didn’t create too many new problems). Once we carried out the migration the DeadlineExceededError excpetions went away, so that was good but we were a bit peeved that our hand was forced in such a way.

 

So I don’t know what google changed but it seems that they want people on high replication or else! Once again we find ourselves seriously questioning whether google app engine is a sutibaly stable platform for deploying real-world apps – the jury is still out…

 

Anyway if you find yourselves with the same problem they you may be headed the HRD way!

 

Google App Engine & Paypal Talk

/in /by

Hi, just a note to let you know that I will be giving a talk at the next Dublin Google Technology User Group (GTUG) meeting on how to go about hooking up Paypal payments to the Google App Engine (GAE). The meeting will be held next Tuesday the 28th at 6:30pm.

 

So if you feel up to being bored to within an inch of your life by some gory GAE & Paypal details then you now know where to go!

 

More details can be found on the Dublin GTUG page here:

 

http://groups.google.com/group/dublin-gtug

 

I have uploaded the slides to my talk on integrating paypal and the google app engine, they can be downloaded from here

 

 

Kevin.

Google profits up – boost to web sector

/in /by

In this gloomy climate it’s great to see some good news – and the Irish Times reports here that Google has seen a sharp rise in its profits for the first quarter of this year – more evidence they say that the web advertising and technology sectors are bouncing back from the recession. It seems that revenue rose 28% which is the the biggest rate of growth since the third quarter of 2008.

 
Good news indeed for Google’s many Irish employees, and good news also for the online industry if this trend continues – it certainly seems that ‘online’ is where we all need to be as there aint much moving and shaking in the traditional marketing arena!