Siri at Apple 25 years ago

Spirit of Siri at Apple 25 years ago

Wed, Jun 13, 12

As a budding standup comedienne, Siri opened Apple’s WWDC 2012 Monday morning and concluded her act with the prophetic:

It’s really hard for me to get emotional, because as you can tell, my emotions haven’t been coded yet.

Clearly, Siri is a work in progress and she knows it. What others may not know, though, is that while Siri is a recent star in the iOS family, her genesis in the Apple constellation goes far back.

The Assistant and Assist

Nearly three decades ago, fluid access to linked data displayed in a friendly manner to mere mortals was an emerging area of research at Apple.

Samir Arora, a software engineer from India, was involved in R&D on application navigation and what was then called hypermedia. He wrote an important white paper entitled “Information Navigation: The Future of Computing.” In fact, working for Apple CEO John Sculley at the time, Arora had a hand in the making of the 1987 “Knowledge Navigator” video — Apple’s re-imagining of human-computer interaction into the future:

Unmistakably, the notion of Siri was firmly planted at Apple 25 years ago. But “Knowledge Navigator” was only a concept prototype, Apple’s last one to date. Functional code shipped to users along the same lines had to evolve gradually over the next few years.

After the “Knowledge Navigator,” Arora worked on important projects at Apple and ran the applications tools group that created HyperCard and 4th Dimension (one of the earliest GUI-based desktop relational databases). The group invented a new proprietary programming language called SOLO (Structure of Linked Objects) to create APIs for data access and navigation mostly for mobile devices.

In 1992, Arora and the SOLO group spun off from Apple as Rae Technology, headquartered on the Apple campus. A year later, Rae Assist, one of the first Personal Information Managers (PIMs), was introduced. Based on 4th Dimension DB, Assist combined contact management, scheduling and note taking in an integrated package (automatically linking contact and company information or categorizing scheduled items, etc) for PowerBook users on the go. Although three versions of Assist were released in the following two years, Rae didn’t make any money in the PIM business. But as Rae also worked with large enterprise customers like Chevron and Wells Fargo in database-centric projects, the company realized the SOLO frameworks could also be used to design large-scale commercial websites:

SOLO is based on a concept that any pieces of data must accommodate the requirement of navigation and contextual inheritance in a database environment. In layman terms, it means that every piece of text, graphics and page is embedded with an implicit navigation framework based on the groupings or order in which the items are organized. In other words, a picture, which is a data object, placed in this programming environment will automatically know the concept of ‘next’ and ‘previous’ without having to write an explicit line of code. This simplifies the coding process. Since the information and business logic organization models were already completed for the client-software, converting this to a web application was simply a recompilation of the codes for a different delivery platform. The project was completed within four weeks and we were stunned as to how simple it was. This was an important validation point illustrating the portability of our technology for cross-platform development.

It wasn’t long before we realized that SOLO, a technology based on information organization models, could be adapted and modified for an application to build web sites. A prototype was developed immediately and soon after a business plan was developed to raise venture funding. NetObjects was founded.

Rae quickly applied for patents for website design software and transferred its technology IP to NetObjects. With seed money and the core team from Rae, NetObjects had a splashy entry into what later came to be known as Content Management Systems (CMS). Unfortunately, the rest was rough going for the fledging company. Not long after IBM invested about $100M for 80% of NetObjects, the company went public on NASDAQ in 1999. Heavily dependent on IBM, NetObjects never made a profit and it was delisted from NASDAQ. IBM sold it in 2001.

Outside Apple, SOLO traveled a meandering path into insignificance. Rae Technology became Venture Capital and NetObjects eventually atrophied.

Flying through WWW

Only three years after the SOLO group left Apple for Rae, Ramanathan V. Guha, a researcher in Apple’s Advanced Technology Group, started work on the interactive display of structured, linkable data, from file system hierarchy to sitemaps on the emerging WWW. Guha had earlier worked on CycL knowledge representation language and created a database schema mapping tool called Babelfish, before moving to Apple to work for Alan Kay in 1994.

His new work at Apple, Project X (HotSauce, as it was later called), was based on 3D representation of data that a user could “fly through” and Meta-Content Format (MFC), a “language for representing a wide range of information about content” that defined relationships among individual pieces of data. At an Apple event at the time, I remember an evangelist telling me that HotSauce will do for datasets what HTML did for text on the web.

Apple submitted MCF to IETF as a standard for describing content and HotSauce (with browser plugins for Mac OS and Windows) found some early adopters. However, shortly after Steve Jobs’ return in 1997, it was a casualty of the grand house cleaning at Apple. Guha left Apple for Netscape, where he helped create an XML version of MCF, which later begot RDF (W3C’s Resource Description Framework) and the initial version of RSS standards.

It’s the metadata, stupid!

Even in its most dysfunctional years in the mid-199os, Apple had an abiding appreciation of the significance of metadata and the relationships among its constituent parts.

SOLO attempted to make sense of a user’s schedule by linking contacts and dates. HotSauce allowed users to navigate faceted metadata efficiently and with some measure of fun to find required information without having to become a data architect. The Assistant in the “Knowledge Navigator” had enough contextual data about its master to interpret temporal, geo-spatial, personal and other contextual bits of info to draw inferential conclusions to understand, recommend, guide, filter, alert, find or execute any number of actions automatically.

There is an app for that

A decade later, Apple was now in need of technology to counter Google’s dominance in search-driven ad revenue on its iOS platform. A frontal assault on Google Search would have been silly and suicidal, notwithstanding the fact that Apple had no relevant scalable search technology. But there was an app for that. And it was called Siri.

Siri was a natural language abstraction layer accessed through voice recognition technology from Nuance to extract information from primarily four service partners: OpenTable, Google Maps, MovieTickets and TaxiMagic. Siri was on the iPhone first but it was headed to BlackBerry and Android. Apple bought Siri on April 28, 2010 and that original app was discontinued on October 15, 2011. Now Siri is a deeply embedded part of iOS.

Of course, the Siri code and the team came to Apple from an entirely different trunk of the semantic forest, from SRI International’s DARPA-funded Artificial Intelligence Center projects: Personalized Assistant that Learns (PAL) and Cognitive Assistant that Learns and Organizes (CALO), with research also conducted at various universities.

What made Siri interesting to Apple wasn’t the speech recognition or the simple bypassing of browser-based search, but the semantic relationships in structured and linkable data accessed through natural language. It was SOLO redux at scale and HotSauce cloaked in speech. It wasn’t meant to compete with Google in search results but to provide something Google couldn’t: making contextual sense.

Unlike Google, Siri knows, for example, what “wife” or “son’s birthday” means and can thus provide, not a long list of departures for further clicks, but precise answers. Siri delivers on the wildest dreams of SOLO and HotSauce of an earlier generation. In two years, even as limited to just a few service partners, Siri progressed far more than the developers of SOLO or HotSauce could have imagined. It now speaks the vast majority of the world’s most prominent languages, with connections to local data providers around the globe.

Having intimate conversations with Samuel Jackson and John Malkovich, Siri has become a TV star. Most iOS users already think Siri has a personality, if not an attitude all together. Hard to say what will happen when she actually gets her “emotions coded.”