Entries Tagged 'Web' ↓

Some follow-ups, corrections, and expansions.  Being correct takes effort.

First, someone very knowledgeable on the internals noted that I was sloppy with the terms ‘datastore‘, ‘GBase‘, ‘GoogleBase‘, ‘GQL‘, and ‘BigTable‘.  Mea culpa.  Datastore is the most generic term and the specific one for Google App Engine is referred to as the “App Engine datastore“.  The App Engine datastore is accessed through GQL, a language reminiscent of SQL.  The App Engine datastore is built on BigTable and exposes some of BigTable’s capabilities (see Wikipedia, a formal paper,  or video documentation).    GoogleBase is an independent Google product that is also built on BigTable.  GBase is a guitar search engine and the naturally elided form of “GoogleBase” after saying it a hundred times.   Whew!  Terms.

Speaking of terms, the contract term dealing with indemnification in the Terms of Service:

13.1. You agree to hold harmless and indemnify Google, and its subsidiaries, affiliates, officers, agents, employees, advertisers, licensors, suppliers or partners, (collectively “Google and Partners”) from and against any third party claim arising from or in any way related to (a) your breach of the Terms, (b) your use of the Service, (c) your violation of applicable laws, rules or regulations in connection with the Service, or (d) your Content or your Application, including any liability or expense arising from all claims, losses, damages (actual and consequential), suits, judgments, litigation costs and attorneys’ fees, of every kind and nature. In such a case, Google will provide you with written notice of such claim, suit or action.

The annoying clause is “(b) your use of the Service”.   Given how claims are written in patents, it is entirely likely that use of the API would be an actionable breach if the Google App Engine violated patents.  Google could require indemnification by users of the API.  For most people and smaller companies, the reputation of Google and pledges to “not be evil” should be sufficient.

Speaking of evil (note the clever transistion), the lazy index evaluation that makes the database look like “read committed” is discussed here.

Finally, high availability is hard, and “9′s” go faster than you remember.

90% (1-nine) is a downtime of 36.5 days per year.
99% (2-nines) is a downtime of 3.65 days per year.
99.9% (3-nines) is a downtime of 8.65 hours per year.
99.99% (4 nines) is a downtime of 52 minutes per year.
99.999% (5 nines) is a downtime of 5.2 minutes per year, or six seconds per week.

A claim of about 2-nines reliability is reasonable.  Google App Engine was launched around four months ago in mid-April, so about a day of downtime is 2-nines.  It was down on June 17 for some unreported number of hours, and was down again on June 19 and June 25.  Add in little outages where various features broke, blocking PayPal, and other nits.  There is a list that occasionally reports downtime, but no exact statistics are available.  If there were no future outages, and you wanted to demonstrate four nines reliability, it would need to take years to overcome the existing outages.

Keep on working on it.  I hope that Google App Engine will be more fun in the future.

80% of effects come from 20% of causes
— Pareto Principle by Vilfredo Frederico Damaso Pareto, 1906

90% of Everything is Crud
– Sturgeon’s Law by Theodore Sturgeon, 1956

100% of Everything is Crud
– Linear extrapolation of above, 2006.

There seems to have been a natural tendency for us to look at past as some magical time when quality mattered. Really, only the pure quality of finished work survived. We forget the uncountable steps towards quality.

In the world of desktop environments, the journey towards quality continues. KDE 4.1 makes a step, following trends, and aiming to be The One Last and True Windowing System.

Quality Steps and Missteps

I installed Kubuntu with KDE 4 on my laptop, and am using it to write this post. That it is written at all shows a minunum level of quality. That the post starts with these quotes shows a maximum level.

I find immediate problems when installing. The very first action, the “Read Me” during the LiveCD boot, comes up in a clipped and illegible font. During partitioning the disks, the progress bar hangs at 0% for ten minutes. The task bar lacks resizing and basic functionality. The quick launcher, Katapult, disappeared. A new program launcher experiment falls flat with some buttons activating by click and other by hovering the mouse.  Konquer still crashes.   I discover new bugs a few times per hour.  These are the the nits and bugs of a new system.

The subtle problems are the problems repeated from previous years: the LILO boot system that unhelpfully refers to Vista as “longhorn” and Kubuntu as “generic Ubuntu core”; the cobbleware of screen layout that has fonts too big for buttons, text too wide for dialogs, and odd alignments; the usual flakiness with power and wireless management. These problems persist for ages from expectation, difficulty, or blindness.

So quality is a step downward while the easy bugs are fixed. Some nice features, like FileLight are a definite step up that I expect every other distribution to copy soon. KDE did buck the trend in releasing a quality downgrade.

Trends in Window Managers

KDE 4 follows the collective wisdom of other software competitors including GNOME, Sugar, Microsoft’s Windows line, and Apple’s OS X line. It tries to be different just like everyone else.  It adds new functionality in pieces and parts.  The hodgepodge of bundling, or smush, that make up windowing systems includes GUI and interface candy, applications, and APIs.

Smush is not pejorative. Open source swaps in and out competing components and the windowing system selects components under its seal of approval and delinates APIs outside its control. Quality for each component is involved at several layers. When the power management on my laptop fails to hibernate before powering off, I could file a bug with KDE, Ubuntu (link to shuttleworth), Debian, the Linux ACPI mailing list, hardware discovery, or just fix it myself. When searching for a workaround, it could exist anywhere.

KDE is following the trend towards aggressively  cross platform deployment. By using Qt 4 as it’s underlying graphics engine, KDE hopes to deploy on desktops (Linux, MS Windows, and OS/X), cellular phones, and some embedded devices. It is currently hampered by Qt 4 having no LGPL or BSD license, requiring a special licensing cycle to deploy any commercial application. The previous Qt 4 copyright holder, TrollTech needed this revenue to continue operations. The new holder, Nokia, may free the code in order to encourage wider adoption and easier developer migration to its cellular phones.

KDE also follows the practice of wrapping more and more functionality into API layer plug-ins. Rather than commit to a scripting language, Kross wraps or interfaces to Python, Ruby and JavaScript. Rather than commit to a multimedia engine, Phonon creates one more layer of indirection for a common interface to GStreamer, QuickTime, DirectShow, and others. The new release switches many of the wrappers so now it is Phonon (multimedia), Solid (device integration), Plasma (a new desktop), Kross (scripting), DXS (application data updates), Decibel (human communications), and D-BUS replacing DCOP (application messaging). It is unclear if “one more layer of indirection” will be the correct solution in the long run.

In the long run, of course, KDE hopes to birth “The One Last and True Windowing System.”

The One Last and True Windowing System

Results 1 – 10 of about 1,040
– Google search for “One API to Rule Them All”

Windowing systems race towards the goals of full functionality, sweet abstractions, and wide deployment with the winner creating a work that will last decades. Each release brings new experiments and implements growing standards. Implementations trump ideas. Differing implementations get abstracted and merged. Engineers have always raced towards sweet solutions.

Software does get finished. We still use the C programming language after more than a third of a century. The syntax, style, and assumptions are passed to new languages such as C++, Python and Ruby. Other contenders failed. Hardware architectures now handle pointer indirections, sequential arrays, and null terminated strings. C goes through occasional updates and forks, but it endures. Also enduring are the most of the Internet stack including TCP/IP, DNS/Bind, and packet switching; HTTP protocols, SQL, and many more. These become the common parts upon which we solve new problems. Many hope to finish writing the desktop software.

FreeDesktop.org gently pushes towards standardization by facilitating discussion. It speeds the process of making compatible, then integrating, then merging competitng development that has become similar. It’s influence on KDE is unmistakable.

So, will KDE give birth to the “One Last and True Windowing System” that lasts for decades?

Conclusions

KDE took a significant risk. It’s adoption of Qt 4, dropping of DCOP, and making so many changes significantly hurt its quality. On the other hand, it might provide a new level of functionality to catapult into the top three windowing systems. Time will tell. For people just wanting a desktop to work, stay away for six months. Continue reading →

Problem: People balk at having online access to seldom used financial, insurance, and service accounts.

Financial and insurance institutions want consumers to access accounts online.  It lowers costs and provides consistent service.

People note the potential for high cost losses, the one-sided user agreements, and complete lack of recourse for most fraud.  Some may choose online access for primary banking; the convenience outweighing the risk.  Many more will avoid online access to a 401(k) plan.

Solution: Provide information only accounts by default.

Information about an account is less valuable than the money in it.  Being a little careless with your account, such as accessing it from a computer, becomes similiar to throwing out printed statements without shreading them.

These accounts would let people see their balances and fees, make most routine changes, but not make ones that severly compromise security.  For example, transfering money from savings to checking is OK; changing mutual funds is OK; adding a newborn to your insurance is OK; changing your address is not OK; disbursing money to outside accounts is right out.  The risk of catastrophic loss decreases.

There are some implementations approaching this solution, aimed at preventing catastrophic losses.  Bank of America provides alert notifications by email when selected activities occur, such as adding a new Payee.   E*Trade uses separate passwords for viewing information and trading securities.

This idea provides lower costs to institutions, more convenience to consumers, and less loss to fraud all around.  What’s not to like?

Have you ever had a downgrade when you thought it was an upgrade?

Now, NetGear make on this an option. You may have seen that my previous review was less than glowing. Now it comes with brand-new backup features in an upgrade!

1. BitTorrent Software installed to download and serve torrents to/from your backup device.
2. A proprietary photo gallery to browse pictures on your backup device
3. A new add-on infrastructure widgets to manage your new BitTorrent and Photo Gallery.
4. Support for new photo standards used by some unnamed new cameras and cutting edge hardware like the XBox 360.

And New Bug Fixes:

1. Flash auto-copy is enabled by default, which transparently affects the way files are copied from some devices in order to better support the new photo gallery!
2. Some bugs with the fan may have been fixed. Or may not have.
3. Lots of small stuff.

And New Bugs:

1. Yes, 10 outstanding issues that affect user performance in this update! Plus lots of “easter eggs” bugs that haven’t been mentioned yet.
2. Now broken with with Opera, Safari, Vista, Web/DAV, wireless, Mac AFP, and installing new drives when the machine is off.
3. The upgrade is “pray-ware”. It cannot be reversed or uninstalled, so no backups for the backup server. Oddly, the upgrade does not state as to if it destroys the data on your drives. It shouldn’t, but don’t assume.

How are these people employed?

Problem:  The One Laptop Per Child (OLPC) XO Laptop coud get viruses

The XO laptop will be a perfect breeding ground for viruses.  The system is monolithic, with an expectation of identical hardware and operating systems being deployed across entire countries.   IT support is expected to be non-existent.   The machine networks heavily and promiscuously.

The primary defense against the spread of viruses is the built-in BitFrost security system.  This assemblage uses virtual machines to restrict certain combinations of rights and attempts to prevent some of the higher harm viruses from occurring.  One can expect many installations to disable any security measures found burdensome, as occurred with the rights based security in J2EE.

Solution:  Make a bootable, read only, USB key to validate that all installed software is open source.

On the other hand, the OLPC has a few truly special features that allow effective anti-virus software to exist.   All, or almost all, software on an OLPC is expected to be open source.   It becomes reasonable to validate that all possible changes to configurations, applications installed, and binaries from a known list of ‘good’ applications.  While “enumerating badness” is a doomed strategy, enumerating allowed systems is possible.

Consider a repository that keeps track of all known contributed applications and language packs in source form.  An automated script builds these into sugarized files and keeps track of the file system effects from installing these onto virtual machines.   The results are placed onto a self-booting USB key that boots up, examines the file system of the XO, and declares it to be clean or infected.

This does require some tricky programming, mostly in reducing the combinatorially exploding possible configurations into a linearly growing set of rules for allowable configurations.  The problem is engineering, not requiring perfect implementation.

This feels like a good PhD Thesis for someone.

Problem: Ubuntu troubleshooting is scattered about useless message boards

Solution: Rate messages for useful information

Ubuntu, the popular Linux distribution, discusses problems in message boards. Someone poses a problem and various people pose possible solutions. The total volume means that great posts detailing the actual way to set environment variables or handle XYZ Mark 32.42 video cards are lost. Users spend way too much time digging through chaff.

One simple solution is to rate the value of posts, ala Slashdot and others. As a user is browsing through the discussion forum, she can tag an answer as especially clear and well written. Alternately, she could tag a message as a question or otherwise content free. Later users might set a minimum threshold to search for well written explanations.

Problem: Good or Cheap Internet Hosting Companies

Solution: Good and Cheap Internet Hosting Companies

The cost of casual web hosting has dropped through the floor and one can expect it to remain there. If you need to host a dozen domains, downloading about 500 gigabytes of bandwidth each month, storing some hundreds of gigabytes of graphics, handling email accounts, and programming with a full Unix shell including tools like Subversion, Python, and MySQL, then that will set you back about $8 per month. DreamHost’s feature list (sign up here) would be unheard of five or ten years ago. With minor variations, so does everyone else. The catch? These servers are not 100% reliable; more like 98% reliable. I noticed DreamHost had about two or three days of outages or severely degraded performance last year.

Hold it, you think, we handle nearly 100% reliability with unreliable hard drives, unreliable communications channels, and even unreliable employees. Why can’t we handle nearly 100% reliability with unreliable web hosting? What can we do?

Redundant Arrays of Independent Service Providers

Tactic 0 — Sharing the Static Load

Web hosting companies slow down or run out of bandwidth quota far more often than actually crashing. Also, graphic files and other static data account for the bulk of the data served. One common tactic is to serve the the static files from several different domains. One can use an Apache plug-in to rewrite outgoing html, a periodic (cron) program to occasionally rewrite your images directory location, a client-side JavaScript routine to display from whichever server provided the picture fastest, mirroring downloads onto a new server, or several other methods. The effect is that loading a web page may load data from more than one domain.

Only the smaller main page needs to be served from the primary ISP when it is slow. All other data comes from faster, inexpensive ISPs. Similarly, a graphics intensive site, such as Girl Genius, could use this method to serve many terabytes of data for about a hundred dollars per month.

Tactic 1 — Dumb DNS Failover or Load Splitting

Another tactic that commonly works to failover to another site. Using various tricks with DNS or BGP, traffic to a downed site can route to a backup or traffic from different geographies can route to different servers. The site is mirrored, but the the URL remains the same.

Tactic 2 — Failover with Transactions Intact. Distributed ACID Transactions.

The hard part of failing over is making sure the important state is preserved when a site goes down. There are ways to do this, with varying degrees of bandwidth efficiency and latency. Each service would need to notify other servers in some scheme in order keep databases intact. Lots of trade-offs present themselves. It is doable, but requires a fair amount of custom work.

Summary

Getting the high availability out of these redundant ISPs is just work. More complex tactics require more engineering. The business model, as always, is to mask all of the complexity of these tactics so that the customer’s problem just “goes away”.

One could provide a service that set up all the failover options, signed up for the ISPs, wrote the scripts to maintain transaction state for a flat fee. Alternately, one could simply sell themselves as a “high availabilty, high transaction volume” service provider while actually running the service off a number of inexpensive ISPs.

There is money sitting there people. Go pick it up!

Just for fun, I ran an experiment last month advertising this blog using Project Wondeful. I spent about $8 to place small graphics alongside popular web comics such as Buck Godet. There was one honest banner (Ideas for Free), and on less honest banner (Sex for Free). No surprise; sex sells.

• Budget: $10.00
• Actually Spent: $8.45
• Ad Impressions: 1,520,000 (Yikes!)
  
• Clicks to the Blog: 617, for a 0.04% click through rate.
• Readership Before Ads: About 2 or 3 per day
• Readership During “Free Ideas” Ad: About 15 per day
• Readership During “Free Sex” Ad: About 30 per day
• Readership After Advertising: About 2 or 3 per day
• Odd Controls: I only updated the blog once during the advertising period.

Well, this was fun! I got my $8.45 worth of entertainment.

Problem: It’s easy to be distracted by the Internet.

Solution: A Back to Work buzzer.

We have a 2006 Honda Accord with a correctly annoying seat belt buzzer. Yes, the little light goes on when you the car is on and the seat belts are not buckled. If you drive off without putting on the seat belts, the car bides its time for about thirty seconds. After all, you might be sipping your coffee, adjusting your phone headset, fiddling with CD Player, and using a pinky and a thin slice of your brain to back into traffic. After a minute or so, the car has had enough and blinks the seat-belt icon and beeps at you. Then it calms down for a minute. Then it starts again. About every minute, the car beeps at you for a few seconds. it is surprisingly effective at reminding us to put on the seat belts.

Now there is a new problem with my computer. It’s connected to all the Internets, which is a complicated series of tubes. Down these tubes come blogs and videos and flash games and all manner of distractions. I don’t want the computer to stop me from following a random link from a mailing list. Still, my will power is weak and the number of videos on YouTube are many. What I want is the annoying Back to Work Buzzer.

If FireFox notices that I’ve been goofing off for too long, a Back to Work Buzzer should buzz at me. Then it should chill for a minute while I finish. Then it should buzz again. It’s my willpower. I just want the reminder to break me out of the infinite Digg crawl.

Implementation is left as an exercise to the reader.

Tome of Naga Summoning