PropertyMaps Corporate Blog

We’ve just added Miami-Dade and the surrounding areas for a total of over 60,000 new properties and 500,000 property images. We now have the Southeast Area of Florida covered. We are very excited to be working with one of the premier brokerages in Miami-Dade to cover this area: Terrabella Realty.

Miami-Dade County is the most populous county in Florida with a population of 2,402,208, according to the US Census Bureau. We’re extremely pleased to add such a huge region to our map as our national rollout continues. Shortly, we will begin publishing statistics about our property and image counts.

Update: You can read more about our partnership with Gustavo Blachman and Terrabella Realty in our Miami-Dade Properties Press Release.

PropertyMaps Adds Miami-Dade And Surrounding Areas

According to this press release, I guess we moved. I was too busy to notice until my 30″ Cinema Display arrived. Then I felt like there was more room. With a resolution of 2560 x 1600, I have more than 4 million pixels to work with.. In addition to my laptop that has a 1440 x 900 display and powers the entire rig, I have somewhere around 5.3 million pixels total. The good part is I have totally mobility, the bad part is I miss my 4 million extra pixels when I’m not in the office.

Oh, the new office. I think it is around 5,000 square feet and can hold maybe 30 people.

PropertyMaps Moves Into Larger Offices

We have outgrown Google Apps, (formerly GAFYD (formerly GFYD)), or something along those lines. The problem of exporting your mail from Gmail is not a trivial one. From discussions by Opera Software’s lead QA for Opera Mail’s posting on Gmail’s Buggy IMAP Implementation to Matt Cutts’ posting on How to back up your Gmail on Linux in four easy steps to LifeHacker’s posting on Back up Gmail on Linux with Getmail to Wired’s recent wiki entry on Make a Local Backup Of Your Gmail Account, it seems that there is no one definitive source on how to pull your mail and retain your labels.

So here is what we have done to solve this problem:

1. Use getmail - this has been the best archiver we have run across. There are other applications - isync, OfflineIMAP, Fetchmail, etc. - that probably do a decent job, but getmail is still the best in my view. There are other hacks - use Mail.app to synch the Gmail IMAP directory, then convert emlx to maildir; same for Thunderbird and mbox; etc - but we wanted something a little more straightforward - Occam’s razor, right?
2. Install getmail - On my dev machine, I used macports (port install python25; port install getmail) to install the latest getmail which had dependencies on Python 2.5. After this was done, I set up the getmailrc config file and fired off an attempt using SimpleIMAPSSLRetriever… which failed due to a lack of SSL in the newly installed Python. I had to go back and install Readline (port install py25-readline), then install SSL for Python (port install py25-socket-ssl).
3. Patch Python - There is malloc bug in imaplib when fetching large documents using SSL. So open up imaplib.py from your Python lib dir (in my case /opt/local/lib/python2.5/) and replace:

   data = self.sslobj.read(size-read)

   with

   data = self.sslobj.read(min(size-read, 16384))

   to maintain a 15MB memory block if necessary.

4. Configure getmail - Now that most of the fun is taken care of, we need to set up a configuration file for getmail (~/.getmail/getmailrc) and create the proper local destination. First the getmailrc file:

   [retriever]
   type = SimpleIMAPSSLRetriever
   server = imap.gmail.com
   mailboxes = ("[Gmail]/Starred",)
   username = username@yourdomain.com
   password = xxx
   
   [destination]
   type = Maildir
   path = ~/Maildir/
   
   [options]
   verbose = 2
   message_log = ~/.getmail/gmail.log

   First of all, we are using IMAP to retrieve mail as POP has a limit of 99 documents per access and that would take forever.

   Second, we are using the Maildir format for the destination so we need to make sure the target directories have been created (~/Maildir/cur, ~/Maildir/new, ~/Maildir/tmp).

   Third, we need to specify a mailbox or mailboxes to download or the INBOX will be the default.

   Fourth, we need a trailing comma on the list of mailboxes to download due to a parsing error in getmail (actually the mailboxes option needs to be a tuple, but the trailing comma negates that).

   Fifth, we need to know the syntax of Gmail’s internal IMAP structure to pull down discrete folders. Non-label folders (Starred, Sent Mail, Drafts, etc.) are accessed with “[Gmail]/Starred” (as in the above config) and labels are accessed directly. For example, the label “Important Project” would have this in the config:

   mailboxes = ("Important Project",)

5. Download your Gmail - For every folder/label I had within Gmail, I downloaded to a separate folder so I could import into dovecot IMAP without hassle. This entailed changing the mailboxes option in getmailrc, running getmail, renaming Maildir to label/directory name, rinsing, repeating.

If dovecot turns out to be a hassle, I’ll blog about that next. Or about bricking my iPhone with the 1.2 firmware because I didn’t read the instructions (yes, I got into the iPhone developer program).

Update: Because maildir uses the modification time of every file to determine the sent date, all emails pulled by the above method will basically lose their sense of time. The below PHP script will restore the modification times:

/* VARS ***********************************************************/
$box = '';
$stem = SITE_DIR.'Maildir/'.$box.'/new/';
/******************************************************************/

$dir_contents = scandir($stem);
foreach($dir_contents as $item) {
	if(!ListFind('.,..,.DS_Store',$item)) {
		$file = $stem.$item;
		$content = file_get_contents($file);
		$date = extractText($content,"\nDate: ","\n");
		$utime = strtotime($date);
		$converted = date('YmdHi.s',$utime);
		shell_exec('touch -mt '.$converted.' "'.$file.'"');
	}
}

function extractText($content,$start,$end) {
	if(strrpos($content,$start)===false) { return false; }
	$startpoint = strpos($content,$start)+strlen($start);
	$endpoint = strpos($content,$end,$startpoint);
	$length = $endpoint - $startpoint;
	return trim(substr($content,$startpoint,$length));
}

function ListDeleteAt($inList, $inPosition, $inDelim = ',') {
	$aryList = _listFuncs_PrepListAsArray($inList, $inDelim);
	array_splice($aryList, $inPosition-1, 1);
	$outList = join($inDelim, $aryList);
	return $outList;
}

function _listFuncs_PrepListAsArray($inList, $inDelim) {
	$inList = trim($inList);
	$inList = preg_replace('/^' . preg_quote($inDelim, '/') . '+/', '', $inList);
	$inList = preg_replace('/' . preg_quote($inDelim, '/') . '+$/', '', $inList);
	$outArray = preg_split('/' . preg_quote($inDelim, '/') . '+/', $inList);
	if(sizeof($outArray) == 1 && $outArray[0] == '') {
		$outArray = array();
	}
	return $outArray;
}

Our land grab in Florida continues with the addition of Flagler County. This is a very nice addition to the PropertyMaps Florida property coverage as Flagler has many beautiful cities: Beverly Beach, Bunnell, Flagler Beach, Marineland, and Palm Coast. Flagler County is also the fastest growing county in the US with 66.7% population growth since the 2000 US Census. We look forward to working with our brokers in the area to make the expansion into Flagler County a success.

PropertyMaps Further Expands In Florida

We are looking to hire a CSS/HTML developer. If you want to work in a great, fast-paced startup environment and feel that you are the best in your field, please submit your resumé through our CSS/HTML Developer page.

PropertyMaps Is Hiring For CSS/HTML

Over 64,000 new properties were just added to Arizona, giving us a total of over 100,000 listings for this state. We are very happy to be working with Tom Tokoph of Urban Realty & Development to make this rollout a success. We have added new properties in Scottsdale, Phoenix, Tempe, Chandler, Tucson, Flagstaff, Tombstone (what a cool name) and many more cities and towns. Over the next several months we are going to really start burning through the backlog of MLS data we have, so expect our listing counts to explode.

Little known fact about Arizona:

“Gunsmith Ottmar Klempfer, an immigrant from Trenton, New Jersey, moved to Tombstone in 1881 and soon patented a bullet that would make a 90-degree turn, enabling gunfighters to shoot around corners.

During a demonstration, Klempfer successfully fired two .44 caliber rounds around the corner of a brick building. But his third round may have been over-engineered. It turned and headed back whence it had come. Klempfer was disappointed, but not for long.”

— courtesy of http://www.wickenburg-az.com

Update: You can read more about this in the Phoenix Broker Press Release.

PropertyMaps Expands Through Phoenix, Arizona

If it’s time once again to upgrade your memory, check out the prices at MemorySuppliers.com. I recently needed a somewhat older chip for a demo machine, and after some surfing around I found they had it in stock at the best price. Service was smooth and the shipping was expeditious. I’d recommend giving them a try on your next upgrade.

Not to start throwing numbers around, but with over 1,000,000 page views last month, we felt it time to further decentralize our infrastructure. Unlike the big buzzword of the day, viral growth, our server infrastructure growth feels more bacterial. As one system is scaled vertically to the breaking point, it is scaled horizontally until it must divide again.

We feel very comfortable with this scaling methodology and are completely confident dividing a cell during business hours without disrupting the main website. That is exactly what we did today.

Once the system is completely decentralized, we can start throwing servers at it and see an almost immediate benefit throughout. Google sure knew what they were doing with MapReduce.

Good ol’ E. Coli

With the recent announcement of Yahoo jumping on OpenID, the number of accounts has almost tripled to 400 million users. Yahoo’s OpenID platform will be ready to go on 1/30 and we will support it from day 1. Google is also dabbling in the OpenID world and has recently became a provider with Blogger in Draft.

PropertyMaps now supports both the OpenID specification 1.1 and 2.0 as a consumer (not provider), meaning you can use almost any OpenID to simplify your PropertyMaps registration and sign in. This also means no more CAPTCHA and no more email validation to create an account. Go ahead an begin your OpenID registration.

If you don’t have an OpenID yet, or want to learn more about it, head on over to the OpenID website.

OpenID Sign In Now Available

We whipped up a quick neighborhood finder with polygon overlays to display Zillow’s recently released ESRI neighborhood shapefiles. I applaud the effort to release this information under a Creative Commons license and the attempt to crowdsource future development.

I’ve been looking at neighborhood boundary information for ages and there really is no standardized classification of what defines a “neighborhood”. The most common method of defining a neighborhood is to group tract data by subpopulation and aggregate data in an attempt to create homogenized areas. The NCDB (Neighborhood Change Database) by GeoLytics does this. I’m not an expert in this area yet, so I’d be curious to find out what other methods are employed to define a neighborhood.

Check out the Neighborhood Boundaries.

Neighborhood Boundaries