Ars Technica has overview of Obama team’s superior back office platform:
One of the biggest problems in the last campaign was that you had all these people who are out in the field who are volunteering who start building their own versions of these rogue tools to do the same thing over and over again,” said Clint Ecker, senior engineer for Obama for America (and an Ars Technica alum). Every field office assembled its own patchwork of tools using spreadsheets or a hacked Web application to track operations. They communicated over Google groups or simple e-mail lists. “It made it hard to keep everyone on the same page,” he added[…]
“We definitely bumped up against some limitations with RDS but they were largely self-inflicted,” Ecker said. “We were able to work around those and stretch how far we were able to take RDS. If the campaign had been longer, we would have definitely had to migrate to big EC2 boxes with MySQL on them instead.” Not having to switch off RDS meant that the Obama campaign saved “a shitload of money” on hiring additional database administrators, Ecker added.
Ecker says the team also tested Amazon’s DynamoDB “NoSQL” database when it was introduced. While it didn’t replace the SQL-based RDS service as Narwhal’s data store, it was pressed into service for some of the other parts of the campaign’s infrastructure. In particular, it was used in conjunction with the campaign’s social networking “get-out-the-vote” efforts.
The integration element of Narwhal was built largely using programs that run off Amazon’s Simple Queue Service (SQS). It pulled in streams of data from NGP VAN’s and Blue State Digital’s applications, polling data providers, and many more, and handed them off to worker applications—which in turn stuffed the data into SQS queues for processing and conversion from the vendors’ APIs. Another element of Narwhal that used SQS was its e-mail infrastructure for applications, using worker applications to process e-mails, storing them in S3 to pass them in bulk from one stage of handling to another.