E Miller _ Note : Appengine and handle

Unless otherwise noted, all work licensed under :

Appengine and handle_entity

Jun 17 2009, 12:52AM

Just spent a frustrating evening with Google AppEngine - and I want to document the problem I had in case it helps others.

I wanted to upload data to AppEngine from a CSV file - however the values in said file needed some adjustments (they don't directly correspond to the fields in the table I have, and some are ReferenceProperty fields). The AppEngine documentation on uploading data says nothing about overriding handle_entity to deal with these requirements.

I found a couple of pages that appeared to be what I needed. They document how to override HandleEntity() in your loader class. I spent ages futzing with this. It wasn't until I looked in the SDK source code that I realized the function to override is called handle_entity()! There is an alias_old_names() in there you can call - but who knew?

For the record, I'm using Release 1.2.2 of the AppEngine SDK

Here is what my loader class looks like now:


import datetime, os, sys
# 'sources' and 'models' down there are not on the python path
curr_path = os.path.abspath(os.path.dirname(os.path.realpath(__file__)))
sys.path = ['/'.join(curr_path.split('/')[:-1])] + sys.path

from google.appengine.ext import db
from google.appengine.tools import bulkloader
from sources import cct
from models import Label, cctRegion, cctVariable, Record

class RecordLoader(bulkloader.Loader):
    def __init__(self):
        bulkloader.Loader.__init__(self, 'Record', 
                                   [('variable', str), 
                                    ('var_type', str),
                                    ('region', str), 
                                    ('year', int),
                                    ('month', int),
                                    ('quantity', float)])
    def handle_entity(self, entity):
        kname = ''.join([entity.region, entity.variable, 
                         str(entity.year), str(entity.month)])
        if Record.get_by_key_name(kname):
            return
        cct.support_classes() # needed classes exist in db
        q = entity.quantity
        dt = datetime.datetime(entity.year, entity.month, 1)
        label = Label.all().filter('name = ', 'cct').get()
        variable = cctVariable.all().filter('short_name = ', entity.variable).get()
        region = cctRegion.all().filter('short_name = ', entity.region).get()
        # reset the entity here to avoid 
        # unneeded attributes, and set keyname
        entity = Record(key_name=kname)
        entity.label = label.key()
        entity.variable = variable.key()
        entity.region = region.key()
        entity.time = dt
        entity.quantity = q
        return entity

loaders = [RecordLoader]

Tags : appengine code

Ethan Miller

Appengine and handle_entity