This week"s StackOverflow Podcast is up: episode 20. 
“Another programmer came to us. ‘I thought you should know that people are really unhappy,’ he said bluntly, ‘and it’s starting to make it so that people just complain all day, instead of doing their work, and that’s not good.’”
“Do you have the E71?” I asked. 
The pictures are, um, well, cellphone quality. I uploaded a few sample pictures. It"s for snapshots and memories, not photography.
Here at Fog Creek Software we get a lot of requests for a tour of the office, which we usually have to decline: we have this unusual obsession with giving programmers quiet working conditions.
We"re moving, though, to IT Conversations, a huge network of terrific audio shows about technology. Just looking at all the great shows they have there makes me feel a bit like a kid in jeans and a T-shirt with a dirty slogan who just walked into Chez Panisse.
Eventually we settled on the Details adjusTables Series 7. We didn"t like the desk surface that those came with (with rounded corners and a chubby profile, it"s just too blah) so we ordered a custom desk surface from Steelcase with something called a knife edge profile. That makes the desk look paper-thin:
The next podcast is up. Today we talked about why we"re doing a podcast in the first place, took some questions/suggestions from listeners, and got into a fight over whether programmers should learn C. Guess which side I took.
Registration is now open for Business of Software 2008 (the first ever Joel on Software conference). Neil has lined up great speakers:
![[1] to [1]](http://www.joelonsoftware.com/items/2008/03/17One-One.png)
![[1] to [many]](http://www.joelonsoftware.com/items/2008/03/17One-Many.png)
![[1,2,3] to [MANY]](http://www.joelonsoftware.com/items/2008/03/17Seq-Many.png)
![[MANY] to [MANY]](http://www.joelonsoftware.com/items/2008/03/17Many-Many.png)
![[MANY (web browsers)] - [MANY (web sites)]](http://www.joelonsoftware.com/items/2008/03/17Many-ManyWeb.png)
![[MANY (web browsers)] - A SINGLE STANDARD - [MANY (web sites)]](http://www.joelonsoftware.com/items/2008/03/17Many-ManyStds.png)
The summer internships at Fog Creek are already full, but Seth Godin, my marketing rebbe, has marketing internships. Paid! “The idea is to find a diverse group of motivated young people who want to join together to create a few really neat projects. The tools used will range from online video to blogs to copywriting to design. Topics might include politics or Squidoo or book promotion or inventing a new kind of web interaction...no scut work, no cold calling, stapling, collating or errands.”
We came up with three fairly interesting new positions that might be a perfect fit for you or someone you know. It"s relatively rare for Fog Creek to hire full time engineers; many of the people we"ve hired in the past have been former summer interns, the company is quite small (20 people), and we don"t hire lightly, so this is a rare opportunity to get in the door and take advantage of the fact that Fog Creek was designed from the ground up to be the kind of company where the best software developers want to work (about Fog Creek).Last week, Microsoft published the binary file formats for Office. These formats appear to be almost completely insane. The Excel 97-2003 file format is a 349 page PDF file. But wait, that’s not all there is to it! This document includes the following interesting comment:
Each Excel workbook is stored in a compound file.
You see, Excel 97-2003 files are OLE compound documents, which are, essentially, file systems inside a single file. These are sufficiently complicated that you have to read another 9 page spec to figure that out. And these “specs” look more like C data structures than what we traditionally think of as a spec. It"s a whole hierarchical file system.
If you started reading these documents with the hope of spending a weekend writing some spiffy code that imports Word documents into your blog system, or creates Excel-formatted spreadsheets with your personal finance data, the complexity and length of the spec probably cured you of that desire pretty darn quickly. A normal programmer would conclude that Office’s binary file formats:
- are deliberately obfuscated
- are the product of a demented Borg mind
- were created by insanely bad programmers
- and are impossible to read or create correctly.
You’d be wrong on all four counts. With a little bit of digging, I’ll show you how those file formats got so unbelievably complicated, why it doesn’t reflect bad programming on Microsoft’s part, and what you can do to work around it.
The first thing to understand is that the binary file formats were designed with very different design goals than, say, HTML.
They were designed to be fast on very old computers. For the early versions of Excel for Windows, 1 MB of RAM was a reasonable amount of memory, and an 80386 at 20 MHz had to be able to run Excel comfortably. There are a lot of optimizations in the file formats that are intended to make opening and saving files much faster:
- These are binary formats, so loading a record is usually a matter of just copying (blitting) a range of bytes from disk to memory, where you end up with a C data structure you can use. There’s no lexing or parsing involved in loading a file. Lexing and parsing are orders of magnitude slower than blitting.
- The file format is contorted, where necessary, to make common operations fast. For example, Excel 95 and 97 have something called “Simple Save” which they use sometimes as a faster variation on the OLE compound document format, which just wasn’t fast enough for mainstream use. Word had something called Fast Save. To save a long document quickly, 14 out of 15 times, only the changes are appended to the end of the file, instead of rewriting the whole document from scratch. On the hard drives of the day, this meant saving a long document took one second instead of thirty. (It also meant that deleted data in a document was still in the file. This turned out to be not what people wanted.)
They were designed to use libraries. If you wanted to write a from-scratch binary importer, you’d have to support things like the Windows Metafile Format (for drawing things) and OLE Compound Storage. If you’re running on Windows, there’s library support for these that makes it trivial... using these features was a shortcut for the Microsoft team. But if you’re writing everything on your own from scratch, you have to do all that work yourself.
Office has extensive support for compound documents, for example, you can embed a spreadsheet in a Word document. A perfect Word file format parser would also have to be able to do something intelligent with the embedded spreadsheet.
They were not designed with interoperability in mind. The assumption, and a fairly reasonable one at the time, was that the Word file format only had to be read and written by Word. That means that whenever a programmer on the Word team had to make a decision about how to change the file format, the only thing they cared about was (a) what was fast and (b) what took the fewest lines of code in the Word code base. The idea of things like SGML and HTML—interchangeable, standardized file formats—didn’t really take hold until the Internet made it practical to interchange documents in the first place; this was a decade later than the Office binary formats were first invented. There was always an assumption that you could use importers and exporters to exchange documents. In fact Word does have a format designed for easy interchange, called RTF, which has been there almost since the beginning. It’s still 100% supported.
They have to reflect all the complexity of the applications. Every checkbox, every formatting option, and every feature in Microsoft Office has to be represented in file formats somewhere. That checkbox in Word’s paragraph menu called “Keep With Next” that causes a paragraph to be moved to the next page if necessary so that it’s on the same page as the paragraph after it? That has to be in the file format. And that means if you want to implement a perfect Word clone than can correctly read Word documents, you have to implement that feature. If you’re creating a competitive word processor that has to load Word documents, it may only take you a minute to write the code to load that bit from the file format, but it might take you weeks to change your page layout algorithm to accommodate it. If you don’t, customers will open their Word files in your clone and all the pages will be messed up.
They have to reflect the history of the applications. A lot of the complexities in these file formats reflect features that are old, complicated, unloved, and rarely used. They’re still in the file format for backwards compatibility, and because it doesn’t cost anything for Microsoft to leave the code around. But if you really want to do a thorough and complete job of parsing and writing these file formats, you have to redo all that work that some intern did at Microsoft 15 years ago. The bottom line is that there are thousands of developer years of work that went into the current versions of Word and Excel, and if you really want to clone those applications completely, you’re going to have to do thousands of years of work. A file format is just a concise summary of all the features an application supports.
Just for kicks, let’s look at one tiny example in depth. An Excel worksheet is a bunch of BIFF records of different types. I want to look at the very first BIFF record in the spec. It’s a record called 1904.
The Excel file format specification is remarkably obscure about this. It just says that the 1904 record indicates “if the 1904 date system is used.” Ah. A classic piece of useless specification. If you were a developer working with the Excel file format, and you found this in the file format specification, you might be justified in concluding that Microsoft is hiding something. This piece of information does not give you enough information. You also need some outside knowledge, which I’ll fill you in on now. There are two kinds of Excel worksheets: those where the epoch for dates is 1/1/1900 (with a leap-year bug deliberately created for 1-2-3 compatibility that is too boring to describe here), and those where the epoch for dates is 1/1/1904. Excel supports both because the first version of Excel, for the Mac, just used that operating system’s epoch because that was easy, but Excel for Windows had to be able to import 1-2-3 files, which used 1/1/1900 for the epoch. It’s enough to bring you to tears. At no point in history did a programmer ever not do the right thing, but there you have it.
Both 1900 and 1904 file types are commonly found in the wild, usually depending on whether the file originated on Windows or Mac. Converting from one to another silently can cause data integrity errors, so Excel won’t change the file type for you. To parse Excel files you have to handle both. That’s not just a matter of loading this bit from the file. It means you have to rewrite all of your date display and parsing code to handle both epochs. That would take several days to implement, I think.
Indeed, as you work on your Excel clone, you"ll discover all kinds of subtle details about date handling. When does Excel convert numbers to dates? How does the formatting work? Why is 1/31 interpreted as January 31 of this year, while 1/50 is interpreted as January 1st, 1950? All of these subtle bits of behavior cannot be fully documented without writing a document that has the same amount of information as the Excel source code.
And this is only the first of hundreds of BIFF records you have to handle, and one of the simplest. Most of them are complicated enough to reduce a grown programmer to tears.
The only possible conclusion is this. It"s very helpful of Microsoft to release the file formats for Microsoft and Office, but it"s not really going to make it any easier to import or save to the Office file formats. These are insanely complex and rich applications, and you can’t just implement the most popular 20% and expect 80% of the people to be happy. The binary file specification is, at most, going to save you a few minutes reverse engineering a remarkably complex system.
OK, I promised some workarounds. The good news is that for almost all common applications, trying to read or write the Office binary file formats is the wrong decision. There are two major alternatives you should seriously consider: letting Office do the work, or using file formats that are easier to write.
Let Office do the heavy work for you. Word and Excel have extremely complete object models, available via COM Automation, which allow you to programmatically do anything. In many situations, you are better off reusing the code inside Office rather than trying to reimplement it. Here are a few examples.
- You have a web-based application that’s needs to output existing Word files in PDF format. Here’s how I would implement that: a few lines of Word VBA code loads a file and saves it as a PDF using the built in PDF exporter in Word 2007. You can call this code directly, even from ASP or ASP.NET code running under IIS. It’ll work. The first time you launch Word it’ll take a few seconds. The second time, Word will be kept in memory by the COM subsystem for a few minutes in case you need it again. It’s fast enough for a reasonable web-based application.
- Same as above, but your web hosting environment is Linux. Buy one Windows 2003 server, install a fully licensed copy of Word on it, and build a little web service that does the work. Half a day of work with C# and ASP.NET.
- Same as above, but you need to scale. Throw a load balancer in front of any number of boxes that you built in step 2. No code required.
This kind of approach would work for all kinds of common Office types of applications you might perform on your server. For example:
- Opening an Excel workbook, storing some data in input cells, recalculating, and pulling some results out of output cells
- Using Excel to generate charts in GIF format
- Pulling just about any kind of information out of any kind of Excel worksheet without spending a minute thinking about file formats
- Converting Excel file formats to CSV tabular data (another approach is to use Excel ODBC drivers to suck data out using SQL queries).
- Editing Word documents
- Filling out Word forms
- Converting files between any of the many file formats supported by Office (there are importers for dozens of word processor and spreadsheet formats)
In all of these cases, there are ways to tell the Office objects that they’re not running interactively, so they shouldn’t bother updating the screen and they shouldn’t prompt for user input. By the way, if you go this route, there are a few gotchas, and it"s not officially supported by Microsoft, so read their knowledge base article before you get started.
Use a simpler format for writing files. If you merely have to produce Office documents programmatically, there’s almost always a better format than the Office binary formats that you can use which Word and Excel will open happily, without missing a beat.
- If you simply have to produce tabular data for use in Excel, consider CSV.
- If you really need worksheet calculation features that CSV doesn’t support, the WK1 format (Lotus 1-2-3) is a heck of a lot simpler than Excel, and Excel will open it fine.
- If you really, really have to generate native Excel files, find an extremely old version of Excel… Excel 3.0 is a good choice, before all the compound document stuff, and save a minimum file containing only the exact features you want to use. Use this file to see the exact minimum BIFF records that you have to output and just focus on that part of the spec.
- For Word documents, consider writing HTML. Word will open those fine, too.
- If you really want to generate fancy formatted Word documents, your best bet is to create an RTF document. Everything that Word can do can be expressed in RTF, but it’s a text format, not binary, so you can change things in the RTF document and it’ll still work. You can create a nicely formatted document with placeholders in Word, save as RTF, and then using simple text substitution, replace the placeholders on the fly. Now you have an RTF document that every version of Word will open happily.
Anyway, unless you’re literally trying to create a competitor to Office that can read and write all Office files perfectly, in which case, you’ve got thousands of years of work cut out for you, chances are that reading or writing the Office binary formats is the most labor intensive way to solve whatever problem it is that you’re trying to solve.
You get a link back by email, with a beautifully organized itinerary, showing all your travel data plus maps, weather reports, and all the confirmation numbers for your flights and address for your hotels and so on.
Remember Fog Creek Copilot? The app that our 2005 interns built that lets you remote control someone"s computer over the Internet to help them with technical problems?
This year when Neil approached me about co-sponsoring the conference, I thought, why not? It"s exactly the kind of conference I would organize if I were organizing a conference about the software business, which, thankfully, I"m not, but Neil is, and he"s doing a bang up job.
This particular data center is in a secure building in downtown Manhattan, in a large facility operated by Peer 1. It has backup generators, several days of diesel fuel, and racks and racks of batteries to keep the whole thing running for a few minutes while the generators can be started. It has massive amounts of air conditioning, multiple high speed connections to the Internet, and the kind of "right stuff" down-to-earth engineers who always do things the boring, plodding, methodical way instead of the flashy cool trendy way, so everything is pretty reliable.
Michael got dressed and took the subway into Manhattan. The server seemed to be up. The Peer1 network connection was fine. The problem was something with the network switch. Michael temporarily took the switch out of the loop, connecting our router directly to Peer 1"s router, and lo and behold, we were back on the Internet. 
Keeping internet services online suffers from the problem of black swans. Nassim Taleb, who invented the term, defines it thus: "A black swan is an outlier, an event that lies beyond the realm of normal expectations." Almost all internet outages are unexpected unexpecteds: extremely low-probability outlying surprises. They"re the kind of things that happen so rarely it doesn"t even make sense to use normal statistical methods like "mean time between failure." What"s the "mean time between catastrophic floods in New Orleans?"