Oh, around the end of April, we made some enhancements to our DB structure and our queries. This round of changes focused on the JOINS and UNIONS and COUNTS and some others, giving us a nice speed boast in our benchmarks. We thought not much of it and went on to other things. Logging in to Google Webmaster Central gave me quite a shock when I saw the chart for time spent downloading a page (in milliseconds). We’ve gone from an average of about 4,300 milliseconds to less than 400, an order of magnitude difference!

We have another trick up our sleeve, but need to do an intense code review and QA before pushing it out to production. This other trick is a tad more complicated, but watching the site run on the dev server is amazing. Hopefully, we can wrap that optimization up and get it live shortly. I’ll probably blog about it when we do.

Go Servers, Go

We have outgrown Google Apps, (formerly GAFYD (formerly GFYD)), or something along those lines. The problem of exporting your mail from Gmail is not a trivial one. From discussions by Opera Software’s lead QA for Opera Mail’s posting on Gmail’s Buggy IMAP Implementation to Matt Cutts’ posting on How to back up your Gmail on Linux in four easy steps to LifeHacker’s posting on Back up Gmail on Linux with Getmail to Wired’s recent wiki entry on Make a Local Backup Of Your Gmail Account, it seems that there is no one definitive source on how to pull your mail and retain your labels.

So here is what we have done to solve this problem:

1. Use getmail – this has been the best archiver we have run across. There are other applications – isync, OfflineIMAP, Fetchmail, etc. – that probably do a decent job, but getmail is still the best in my view. There are other hacks – use Mail.app to synch the Gmail IMAP directory, then convert emlx to maildir; same for Thunderbird and mbox; etc – but we wanted something a little more straightforward – Occam’s razor, right?
2. Install getmail – On my dev machine, I used macports (port install python25; port install getmail) to install the latest getmail which had dependencies on Python 2.5. After this was done, I set up the getmailrc config file and fired off an attempt using SimpleIMAPSSLRetriever… which failed due to a lack of SSL in the newly installed Python. I had to go back and install Readline (port install py25-readline), then install SSL for Python (port install py25-socket-ssl).
3. Patch Python – There is malloc bug in imaplib when fetching large documents using SSL. So open up imaplib.py from your Python lib dir (in my case /opt/local/lib/python2.5/) and replace:

   data = self.sslobj.read(size-read)

   with

   data = self.sslobj.read(min(size-read, 16384))

   to maintain a 15MB memory block if necessary.

4. Configure getmail – Now that most of the fun is taken care of, we need to set up a configuration file for getmail (~/.getmail/getmailrc) and create the proper local destination. First the getmailrc file:

   [retriever]
   type = SimpleIMAPSSLRetriever
   server = imap.gmail.com
   mailboxes = ("[Gmail]/Starred",)
   username = username@yourdomain.com
   password = xxx
   
   [destination]
   type = Maildir
   path = ~/Maildir/
   
   [options]
   verbose = 2
   message_log = ~/.getmail/gmail.log

   First of all, we are using IMAP to retrieve mail as POP has a limit of 99 documents per access and that would take forever.

   Second, we are using the Maildir format for the destination so we need to make sure the target directories have been created (~/Maildir/cur, ~/Maildir/new, ~/Maildir/tmp).

   Third, we need to specify a mailbox or mailboxes to download or the INBOX will be the default.

   Fourth, we need a trailing comma on the list of mailboxes to download due to a parsing error in getmail (actually the mailboxes option needs to be a tuple, but the trailing comma negates that).

   Fifth, we need to know the syntax of Gmail’s internal IMAP structure to pull down discrete folders. Non-label folders (Starred, Sent Mail, Drafts, etc.) are accessed with “[Gmail]/Starred” (as in the above config) and labels are accessed directly. For example, the label “Important Project” would have this in the config:

   mailboxes = ("Important Project",)

5. Download your Gmail – For every folder/label I had within Gmail, I downloaded to a separate folder so I could import into dovecot IMAP without hassle. This entailed changing the mailboxes option in getmailrc, running getmail, renaming Maildir to label/directory name, rinsing, repeating.

If dovecot turns out to be a hassle, I’ll blog about that next. Or about bricking my iPhone with the 1.2 firmware because I didn’t read the instructions (yes, I got into the iPhone developer program).

Update: Because maildir uses the modification time of every file to determine the sent date, all emails pulled by the above method will basically lose their sense of time. The below PHP script will restore the modification times:

/* VARS ***********************************************************/
$box = '';
$stem = SITE_DIR.'Maildir/'.$box.'/new/';
/******************************************************************/

$dir_contents = scandir($stem);
foreach($dir_contents as $item) {
	if(!ListFind('.,..,.DS_Store',$item)) {
		$file = $stem.$item;
		$content = file_get_contents($file);
		$date = extractText($content,"\nDate: ","\n");
		$utime = strtotime($date);
		$converted = date('YmdHi.s',$utime);
		shell_exec('touch -mt '.$converted.' "'.$file.'"');
	}
}

function extractText($content,$start,$end) {
	if(strrpos($content,$start)===false) { return false; }
	$startpoint = strpos($content,$start)+strlen($start);
	$endpoint = strpos($content,$end,$startpoint);
	$length = $endpoint - $startpoint;
	return trim(substr($content,$startpoint,$length));
}

function ListDeleteAt($inList, $inPosition, $inDelim = ',') {
	$aryList = _listFuncs_PrepListAsArray($inList, $inDelim);
	array_splice($aryList, $inPosition-1, 1);
	$outList = join($inDelim, $aryList);
	return $outList;
}

function _listFuncs_PrepListAsArray($inList, $inDelim) {
	$inList = trim($inList);
	$inList = preg_replace('/^' . preg_quote($inDelim, '/') . '+/', '', $inList);
	$inList = preg_replace('/' . preg_quote($inDelim, '/') . '+$/', '', $inList);
	$outArray = preg_split('/' . preg_quote($inDelim, '/') . '+/', $inList);
	if(sizeof($outArray) == 1 && $outArray[0] == '') {
		$outArray = array();
	}
	return $outArray;
}

We are looking to hire a CSS/HTML developer. If you want to work in a great, fast-paced startup environment and feel that you are the best in your field, please submit your resumé through our CSS/HTML Developer page.

PropertyMaps Is Hiring For CSS/HTML

Not to start throwing numbers around, but with over 1,000,000 page views last month, we felt it time to further decentralize our infrastructure. Unlike the big buzzword of the day, viral growth, our server infrastructure growth feels more bacterial. As one system is scaled vertically to the breaking point, it is scaled horizontally until it must divide again.

We feel very comfortable with this scaling methodology and are completely confident dividing a cell during business hours without disrupting the main website. That is exactly what we did today.

Once the system is completely decentralized, we can start throwing servers at it and see an almost immediate benefit throughout. Google sure knew what they were doing with MapReduce.

Good ol’ E. Coli

We whipped up a quick neighborhood finder with polygon overlays to display Zillow’s recently released ESRI neighborhood shapefiles. I applaud the effort to release this information under a Creative Commons license and the attempt to crowdsource future development.

I’ve been looking at neighborhood boundary information for ages and there really is no standardized classification of what defines a “neighborhood”. The most common method of defining a neighborhood is to group tract data by subpopulation and aggregate data in an attempt to create homogenized areas. The NCDB (Neighborhood Change Database) by GeoLytics does this. I’m not an expert in this area yet, so I’d be curious to find out what other methods are employed to define a neighborhood.

Check out the Neighborhood Boundaries.

Neighborhood Boundaries

I recently dove deep in Google Analytics. I found a wealth of information but something really caught my eye. Not just that, it improved our website. If you log into your analytics account and view your website stats, under the content section is something titled “Top Exit Pages”. I’m sure everyone understands the usefulness of this tool but not even I understood the full potential at first.

Obviously when viewing your top exit pages, you see the pages people leave from the most along with percentages and information. If you put two and two together you realize, this tells you what pages people dislike or what is wrong with your SEO strategy. A big light bulb went off in my head when I saw this. Our top exit pages were delisted properties. Then I looked up our SEO stats. Users were searching for specific MLS numbers and found our properties. Unfortunately some of those properties were no longer listed on our site either because they were sold or canceled. This made our potential customers become discouraged and leave. We had hundreds of exits from the delisted page a month. Did these people ever come back? I didn’t know, and that was a problem. All I knew was that the exit rate was over 85% and the bounce rate was over 90%. They come in via search engine, see our delisted page, then leave.

So we had two options. Either we can try and get our content more actively updated on Google, or we could change the delisted page to keep users from leaving. So we attempted both and now the number of users seeing the delisted page is zero. Without the analytics I would not have noticed this problem. Our problem was mostly due to Google’s cache of data no longer available. So in a way this both helped to make our site a little better and benefited our SEO. So take a look at your analytics. Sometimes those numbers can tell you a lot more than you think.

We have added a reverse geocoder along with our regular geocoder. This reverse geocoder scans our database of millions of addresses to pull back the proper location for the lat/lng coordinates you enter. As with the regular geocoder, the reverse geocoder requires an API key if you want to bypass the built-in four second delay. If you want an API key, please contact us.

With (or without) an API key, you can access the reverse geocoder directly with the following GET string: http://www.propertymaps.com/geocode.reverse?latitude=[LATITUDE] &longitude=[LONGITUDE]&api=[API Key]

Here is an example: Click Here.

You can also read our blog entry on the regular geocoder.

As we continue to add MLS data to the map, we are sometimes faced with the dreaded RETS format for retrieving said MLS data. The problem with RETS is that there are so many version standards and so many different interpretations of those standards. A script that works on one MLS system does not necessarily work on another system, even if both are using the same RETS specification.

With that said, our team faced a challenge in tackling an antiquated RETS/1.0 system that not only had a flakey implementation, but was often times returning corrupt XML due to the DB server hanging during the SQL query. We were finally able to get the data we wanted, but it wasn’t until two of us trial and errored out a script in five hours.

For anyone that wants to deal with a flakey RETS/1.0 implementation, you can have at our script. We had to bypass the usual nonce, cnonce, opaque, response rigamarole because the RETS server was not replying with the correct HTTP headers. Instead we initiated the RETS login, grabbed a plain cookie and used that to keep the session alive. The script is based on the fine work done over at http://www.thompsonbd.com/RETS/rets_example.php.

Here is the RETS PHP Script.