Offermann.us

(Or, How I wrote almost 800 words about changing two lines of code…)

My wife and I keep a blog about our recent move to Buenos Aires. Michele is a non-stop, daily blogging machine, while I write a post, oh about once a month…which may not be a very impressive body of work, but it certainly beats my track record around here lately!

In addition to my relative lack of posts, what’s also been missing on our blog has been my Twitter updates in the sidebar. Why? Because while Twitter provides a nice, simple widget that works well for including Twitter updates on the blog, it only works for a single user.

But, what if you have a blog with two writers, and they both want to include their tweets in the sidebar? Here’s how I finally solved that problem.

What Goes Wrong When You Add Two Twitter Widgets, Anyway?

I’m no javascript expert, but Twitter’s widget code is fairly simple:

First, Twitter generates the following lines of HTML, which you paste into your blog’s template. With WordPress, it’s easiest to put it in a Text Widget in the Sidebar.

The HTML for the second widget looks like this:

Looks familiar, right? In fact, the only difference is the different Twitter user name in the second script. It’s now MichRee.json.

Notice that the <ul> in this widget has the same exact id as the first one: “twitter_update_list“. That’s where the problem lies, because watch what happens:

1. In the first Widget Text Box, toffermann.json calls twitterCallback2, which looks for the first unordered list element with an id of “twitter_update_list” in the page’s HTML, where it inserts the list of toffermann’s tweets.

2. Then, in the second Widget Text Box, MichRee.json calls the same twitterCallback2 function, which again searches for the first unordered list with an id of “twitter_update_list.” And, rather than find that list to be empty, this time it finds that toffermann’s tweets have already been inserted…so it goes ahead and overwrites them with MichRee’s tweets.

Final result: One widget displays the wrong updates, while the other widget is blank.

Oops.

My Not-So-Easy Two Step Solution

1. Change the twitterCallback2 Function

   I modified the twitterCallback2 function so that it looks for an element ID that includes the Twitter user name, not the hard-coded “twitter_update_list” ID.

   After running the blogger.js code through javascript tidy and changing just the last line of the function, it now looks like this:

   I saved the modified file as wp-includes/js/twitter-simple.js.

2. Change the HTML in the Widget Text Boxes

   Include the Twitter user name in the <ul> element ID, and then point to our newly saved twitter.js file.

   That means, in Widget Text Box #1, you put:

   And, in Widget Text Box #2:

And, that’s that! You now have updates for two Twitterers on the same page.

That Seems Complicated. Aren’t There Any Other Solutions?

Why, Yes. There are a ridiculous number of Twitter-Wordpress solutions out there. Here is a list of the main contenders I considered that can handle multiple Twitter users…and the nitpicky reasons why I ultimately rejected them:

1. Twitter for WordPress by Ricardo González.

   A strong contender. But, it’s not super configurable, so I couldn’t get the output to look exactly like the output of the Twitter widget. It can’t make the timestamp linkable. And, it doesn’t always show timestamps for tweets in “human readable format”. I prefer it to say a tweet was posted “about 4 days ago”, rather than posted on “03/22/09”, but this plugin switches to the dd/mm/yy format if the tweet is more than 24 hours old.

2. twitterjs by Remy Sharp.

   Looked promising, but since it’s not a WordPress plugin, it requires a bit more configuration than I was hoping for. Plus, it also doesn’t do “human readable format” timestamps for tweets older than 24 hours. (Though it looks like earlier versions did do it, so I wonder why it changed.)

3. Twitter Widget Pro by Aaron Campbell.

   I came very close to using this. It’s almost perfect. But, here’s where I get really picky…In addition to the text of the tweet, it also shows its “source”. (As in “this post was sent by TweetDeck.”) I know that twitter.com shows the source as well, but I think it’s superfluous information, and I don’t care to show it.

Uh oh.

Marissa Mayer kicks off her speech by talking about artists and fashion designers creating custom background images to use on iGoogle home pages. This is what’s exciting at Google right now?

Minutes go by, and still more fluffy iGoogle talk. Now, I’m really worried…she’s going to crash and burn. I mean, she’s speaking at a conference for developers…and she’s talking about wallpaper?

Thankfully, once she really got going, her speech turned out to be quite entertaining and was filled with funny stories about the early days of Google.

Some tidbits:

• She once asked Sergey Brin to explain the spare design of the original Google home page:

  “We didn’t have a webmaster…and I don’t do HTML.”
  

• Early user testers would just stare at the Google home page…and not type anything into the search box.

  Why?

  They were waiting for the rest of the page to load.

  To solve this problem, Google had to add a footer, so people knew the home page was done loading.

  “Our copyright notice isn’t legally required.  It’s punctuation.”
  

• Every search query on Google hits between 700 to 1000 servers before serving the response. Think about that the next time your search results return in 0.09 seconds.

• “Design is becoming more of a science than an art.”

  They do tons of A/B Testing at Google, testing every tweak to the site on a small percentage of their users and then watching to see how it impacts their “Happiness Metrics”. Which means that they don’t have to guess which design or feature is better. They have numbers, hard evidence that one is better than another.

  Two examples:

  • Less white space around their logo made their users more satisfied.
  • Displaying more search results per query caused happiness to plummet, because adding even the tiniest fraction of a second to the page response time resulted in decreasing user satisfaction.

I’m at the Google IO Conference in San Francisco. It’s late at night…I’m still decompressing and trying to make sense of my notes from the day’s sessions.

I’ve put myself on a self-imposed “App Engine Only” track, so I haven’t really followed the news and announcements regarding Android, or OpenSocial, or Google Gears, or any other Google initiative.

Though they did show a way cool Android mobile phone demo in the Keynote. The phone had a built-in compass, so you could run the Google Maps application and look at the Street View, and then as you moved the phone, the Street View would sync to the new compass direction. In other words, as you moved the phone from North to South, the Street View photos would also pan from the North view to the South view. Spontaneous cheering from the crowd.

App Engine Talks

In any event, here are my takeaways from today:

• Use Django. Don’t use the one that’s bundled with App Engine (v. 0.96). No, check out the latest development version of Django from subversion (currently 0.97). What’s interesting is that even though Google App Engine ships with the webapp framework and with Django 0.96, Guido van Rossum chose to devote his talk to how to install and use Django 0.97 on Google App Engine. I was curious what that meant for webapp, so I asked if there was ever a reason to use webapp instead of Django. Guido’s answer was that webapp was simple and enabled you to get started very quickly. But, Django was more powerful.

• If you use the Datastore correctly, scaling comes for free. But, it takes a lot of work to use the datastore correctly. Further complicating the issue is that nobody knows exactly what “correctly” means yet.

• Pay special attention to Datastore writes. “Reads are cheap! Writes are expensive!” Every write is a serialized transaction that hits the disk. Rule of thumb says you can’t do more than ~100 seeks/sec, so that’s the upper limit for write speed.

• For high-contention write situations (like a counter), don’t write to a single, global counter class. Instead, use “sharded writes” where you write to several entities, and then sum the totals from all the shards to get the total count.

• Entity Groups are for transactions. I did not understand the point of parent-child hierarchies for entities before today, but now I see that their only purpose is to group entities that need to be changed together as a part of a transaction.

• The Datastore will not help you maintain data integrity. It’s entirely up to you, especially when you are updating or deleting entities with ReferenceProperties. Seeing the code examples today that showed how to handle deletes properly makes me appreciate how nice it is to have a relational DB handle this automatically.

• Don’t use count(). Ever. This was stressed in several presentations. Not only can it not count past 1000, but it also requires a scan of every entity, thereby using way too much processing power.

• You can’t use JOINs in queries, but you can utilize the foreign-key-like ReferenceProperty to associate entities and therefore, do JOIN-like queries on them. In Rafe Kaplan’s talk about “Working with Google App Engine Models”, he walked through one way to model one-to-many and many-to-many relationships. I found this particularly interesting, in that it demonstrates how to have a relatively normalized data model, directly contradicting the emerging idea that the best way to take advantage of the Datastore is aggressive de-normalization.

• The index.yaml file defines the composite keys needed for complex queries. All queries rely on the Indexes, which are separate BigTable tables. If a property value isn’t indexed, it can’t be found by a query. Now, that’s not a problem for querying by kind or single property values, since indexes are created for these automatically, but it is relevant for complex queries, since composite keys are not created automatically. (The dev server automatically updates the index.yaml file as complex queries are run in the dev environment, so as long as you test every query on dev, you’ll be fine.)

Several themes came up over and over in the Q&As that followed the talks:

• Maintaining Data Integrity, Especially over Time. How do I migrate data models? How do I delete columns, or rename columns? How do I rename classes? How can I make changes and not break data integrity? Nobody’s figured out good answers to these questions yet.

• Bulk Data Operations. How can I import and export large data sets? How can I bulk delete? The App Engine team stressed that they were well aware of this limitation.

• Full-Text Search. How can I do full-text search over my data? Shockingly, developers uniformly expect Google to be really good at full-text search…imagine that! Sadly, it’s not there yet, though Google engineers mentioned that there’s “currently a hack in place that kind of works”, and that they would provide this feature sometime in the future.

In his presentation on how the Datastore works under the hood, Ryan Barrett offhandedly mentioned the three big priorities for the Google App Engine Team right now:

1. Data Import/Export
2. Additional languages
3. Billing

All in all, the Google App Engine talks were fascinating. Using Datastore is so new, and it’s such a departure from using a relational DB, that it’s exciting to see everyone try to figure out how to best utilize it. Though, one thing I was struck by was how tentative some of the suggestions were from the Google engineers. I had expected that since they have several years of experience building applications on top of BigTable, that they would have created more definitive best practices for how to handle common data models and for maintaining data integrity. Perhaps it’s so new that even Google is still figuring out how best to work with it.

Portland’s second ever BarCamp (a free-form “unconference”) was held this weekend, and since there were no soccer or Little League games for me on Sunday, I was able to catch one day’s sessions.

Not knowing what to expect when I arrived, I quickly found an enthusiastic crowd, plentiful food (bagels stacked high and all the Bubble Tea you could drink), and lots of interesting sessions, with topics ranging from WordPress to CrossFit to Bikes to Patents to Predicting the Stock Market.

You are Your Twitter Name

As you register, you’re handed a badge so you can write your name and other key information. But, you don’t identify yourself by your email address, or your blog, or your domain name. Instead, at BarCamp Portland, your identity is your Twitter identity.

As I looked around at other people’s badges, I noticed that nearly everyone had a Twitter name.

I was surprised at how ubiquitous Twitter was. “What’s wrong with the alternatives?”, I wondered. Here’s my stab at some answers:

• Email. Too much spam. People suffer from email overload. Plus, there’s no way to learn more about someone if you have only their email address.
• Blog. Not everyone has a blog. It’s much easier to start twittering than it is to set up and maintain a blog.

Whatever the reason, Portland BarCampers have clearly chosen Twitter their default online profile.

WordPress as CMS

I sat in on several WordPress-oriented sessions, and “Using WordPress as a CMS” was my favorite. Ben Hengst from Powell’s Books talked about they were using WordPress on powells.com.

The Powell’s Web site is made up of many different systems (some written in perl, some in PHP), and WordPress currently only powers the PowellsBooks.Blog section of the site on the front-end. But, as the developers and Marketing folk began to use WordPress more and more, they found that it offered a simple interface for Marketing folk to input reusable content, and that it’s database structure could handle the different types of content they required. Ben said that using a combination of tags and categories gave them all the flexibility they needed.

So, now, they are inputting content with the WordPress Admin interface on the back-end, but then pulling that content from the WordPress database with the perl and PHP code that runs the rest of the site. In other words, much of the content that is entered in WordPress is never displayed by WordPress.

Random Observations

Sometimes it’s the little things that you see or hear about that make the event worthwhile.

• I heard the story of writing an innocent blog post about how you are unable to burp on demand…and watching a whole community of non-burpers spring up around it.
• Reid Beels was giving an overview of how to write a WordPress plugin…and then offhandedly demoed a really cool javascript photo display (hover over one photo and see how the rest of the photos evenly resize).
• I spotted someone wearing not only a Utilikilt, but Vibram Five Fingers shoes as well. A somewhat radical look…but he totally pulled it off.

It’s all Aaron Hockley’s fault.

All last week, he’s been on Twitter relentlessly beating the drum for Portland bloggers to add OpenID support to their blogs.

So, I did.

And, he was right. Setting it up was a breeze with the WordPress OpenID plugin. Now, thanks to Aaron, you can save yourself the trouble of typing in your name and email address when you leave a comment, so long as you have an OpenID account.

(If you don’t have an OpenID account, not to worry. Leaving a comment will work exactly as it did previously.)

Amazon and Google are true pioneers, making cloud computing accessible to everyone. I’m fascinated with their radically different approaches, and I find I just can’t read enough about either of them!

Recently, I’ve collected so many links about Google App Engine and Amazon Web Services, that I’ve decided to share them on a linkblog that I’m calling Amazoogle News.

My goal is to post daily links that cover news, commentary, success stories, developer HOWTOs…basically, anything I find interesting that relates to either Amazon Web Services or Google App Engine.

For anyone similarly obsessed, check it out.

Thirty minutes of playing with the Guestbook application in the Google App Engine SDK was all it took for me to get something up and running:

http://tofferapp.appspot.com

Getting started really couldn’t be much simpler.

Today’s theme is “Lock-In”. If you build a Web app with Google App Engine, can you ever pick up and leave?

This fear can certainly be overblown. As DeWitt Clinton (Google developer) points out “beyond the datastore interface, there are few parts of the stack that are all Google-specific.” He also clears up a common misconception: utilizing Google Accounts for authenticating your users is strictly optional. You won’t get locked in there unless you want to be.

However, the concern is legitimate when it comes to the database. Once you build on top of BigTable through the Datastore API, you’re not going to be able to move easily, because, right now, there’s nowhere else to go.

But, that’s right now. In the near future, I think you’ll have options, and lock-in may not be as much of a concern. At least, that’s a common thread through several interesting posts I found today:

• Cloud: commodity or proprietary?: Doug Cutting (Head of the Hadoop Project) stresses open-source as the best alternative. He thinks that we can’t count on “cloud providers” to build on top of a non-standard APIs, and that only open-source implementations can counter their inevitable proprietary APIs.
• App Engine: Commodity vs. Proprietary: Ian Bicking is a bit more optimistic. To him, Google’s APIs look “reasonably abstract and reasonably minimal” and “seem to lack the stench of proprietary APIs”.
• Will Google fulfill the promise of WinFS?: Ivan Krstić predicts that we’ll see “people begin to implement their own Google datastore API-compatible backends on top of everything from Apache CouchDB to (awkwardly-fitting) relational databases like MySQL and Postgres — which, make no mistake, will happen very soon.” Ivan also dissects the in-memory backend of the datastore that Google includes in the SDK, and concludes that this technology is not suited solely for the Web, but that it’s “march to the desktop is all but inevitable.”

“Ultimately, we are trying to provide an simpler alternative to the traditional LAMP stack.” —Kevin Gibbs, App Engine Tech Lead

Google’s announcement of their new App Engine was a bombshell. A big, big story that probably inspired more analysis and commentary than any story since Facebook opened up their Applications to all developers.

So, how did most bloggers frame this story? They described it as a Battle of the Titans. Goliath vs. Goliath. Google vs. Amazon. Look through the coverage on TechMeme and just try to find an article that doesn’t compare and contrast Google App Engine to Amazon’s Web Services (EC2, S3, etc.).

But, comparing Google App Engine to Amazon’s offerings misses the point. Google didn’t set out to build a killer of Amazon Web Services.

Google set out to build LAMP 2.0.

Google App Engine and Amazon Web Services are Nothing Alike

Amazon pioneered offering Web services in the Cloud, and they have an extensive array of individual Web services on their menu. One service for servers on demand (EC2). One for storage (S3). Another is a database-like service for querying structured data (SimpleDB). And, those are just the most prominent ones; they’ve got plenty more.

By offering so many tools, Amazon provides a great deal of flexibility and power to complete a wide variety of computing tasks. Photo storage, backup solutions, next-generation search engines, video publishing, Facebook Apps, data analysis and big-time number crunching…they’ve all been built with Amazon’s Web services. Now, to use all that power, you need a great deal of sysadmin smarts. But, if you’ve got the system administration know how, then you’re not limited to building only Web applications. You really can build just about anything.

In fact, when Amazon’s on-demand server farm was first introduced, it was best suited for offline tasks, and it wasn’t all that great for building complete Web applications. You couldn’t get a static IP. Amazon didn’t offer a database service. And, if you ran MySQL on an EC2 instance, you had to be very careful about your backup strategy, because if your EC2 instance went down unexpectedly, then your database files disappeared as well. To their credit, Amazon has been constantly iterating and improving their Web services, so with the recent introduction of Elastic IPs and SimpleDB, building a Web app with their services has gotten dramatically easier.

In contrast, Google is not offering developers a la carte access to individual pieces of their infrastructure. We don’t get the ability to run MapReduce tasks on their infrastructure. They’re not offering API access to BigTable.

Instead, Google’s App Engine is designed for one thing, and one thing only: building and deploying Web apps. It may lack flexibility, but it’s a comprehensive system for building Web apps the way Google prescribes. (Given it’s “There’s only one way to do it” motto, Python is a fitting choice as the first and only language supported by App Engine.) How comprehensive is it? Here’s a quick list of things App Engine handles for you that you would have to do for yourself with Amazon’s Web Services: load balancing, deployment, versioning, and backups.

What’s So Great About LAMP?

Like the Google App Engine, the LAMP stack (Linux, Apache, MySQL, PHP) is also a complete system for building and deploying Web apps. It’s widely used because it offers a great number of advantages for developers:

• Easy to code. Novices can build something and get it up and running very quickly with PHP and MySQL.
• Easy to deploy. Since PHP is a standard apache module, it’s easy to deploy a PHP app. Once you’ve got MySQL running, simply upload your .php files.
• Develop locally. It’s easy to set up LAMP on your laptop, build your app locally, then deploy on the Web.
• Cheap and ubiquitous hosting. Even the cheapest Web hosts options allow you to run PHP and MySQL.

Sounds great, right? It is…at least until Web app starts attracting significant traffic, or your Web app gets slashdotted, or suffers from the Digg Effect. Then, unfortunately, your cheap, little, ubiquitous shared Web server just falls down and dies. Just when you need it most, your Web app disappears. That’s the biggest disadvantage to using the LAMP stack.

That’s not to say that the LAMP stack can’t be scaled. Facebook and Wikipedia are just two examples that prove that it can be. But, to do so requires considerable expertise and some serious cash. You’re going to need to beef up your Web Operations team to deploy an array of load balancers, memcached servers, and Web servers. You’re going to have to manage replicated and sharded MySQL databases. You’ll need to start worrying about geopraphical redundancy. And, once you have all these systems in place, then you’ll need someone to tweak and tune your Web machine to best handle the constantly changing traffic patterns for your Web app.

LAMP 2.0: The LAMP That Scales Cheaply and Effortlessly

But, what if scaling could be handled automatically? What if you could still build Web apps quickly, easily, and cheaply, and have it also scale automatically? What if deployment was still easy? What if you could still develop your Web app on your laptop?

Even better, what if you get all that, and then running your Web app was free?

No doubt these are ambitious goals. But, what if Google can actually follow through on all these promises?

Well, then I’d say Google will have just created LAMP 2.0.

UPDATE: Found a couple more interesting discussions about how App Engine is a significantly different offering than Amazon’s Web Services.

Nate Westheimer thinks that App Engine has more in common with Facebook’s Platform. I disagree with that idea, but in our discussion in the comments of his post, he pushed back against my “LAMP 2.0” idea, rightly pointing out that Google’s services were proprietary and closed-source, and so not very LAMP-like at all.

With my LAMP analogy, I was really trying to stress the completeness of Google’s solution, and it’s emphasis on making building and deploying Web Apps easy for developers, but perhaps the “open vs. closed” debate distracts from my point.

For that reason, I think that Rich Skrenta has come up with an even better analogy: Google App Engine is “Web Hypercard”.