Databases Before Computers — the forgotten story.

Systems before computers had better database technology

The answer to why systems didn’t fail before computers may surprise you. It isn’t because the computing elements were human beings, it’s because back then they were better database engineers than we are today.

Lance Gutteridge
codeburst

Paper systems had innovative access and processing

Before computerization, enterprise systems were done on time, on budget, and could be changed without major problems. There wasn’t any concept of system failure. People just assumed that if a new system was to be implemented it would be done without any particular problems. People took for granted that if a system needed to be modified that too would be done without problems.

Today’s enterprise systems are slow moving disasters. Any kind of new system or change to an existing system is a huge risk. Security is routinely breached by criminal elements and mistakes are so common as to not be newsworthy unless they are huge.

Despite this the IT industry is hugely profitable. It is the organizations using these systems and the public at large that suffer the consequences.

There is no real incentive to change anything. That is, other than a sense of shame over our pitiful performance and the feeling that history will be scathing in its assessment of how we have behaved.

This isn’t acceptable. We have to do something about it. That something is revisiting the technical decisions we have made over the last sixty years and identifying what has gone wrong and what different technologies might accomplish. For a comment on this see my article:

Why do we, with our incredible technologies, find so difficult what people using previous paper-based technologies found so straightforward?

We have an arrogant conceit that our technology is so powerful that previous generations have nothing to teach us that could possibly be of value. When I have tried to get enterprise programmers and analysts to opine on this, the normal explanation, usually accompanied by a shrug of indifference, is that systems using people are just easier because people have common sense and can adapt to changing circumstances.

The real reason that systems were not subject to these problems in the past is that they had better database theories. To some it will seem strange to be talking about database theories before the invention of modern computers. However, just because information is stored on paper doesn’t mean that it doesn’t form a database.

Here is a common definition of a database:

A collection of information that is organized so that it can be easily accessed, managed, and updated.

The paper forms of the pre-computer era were databases. They certainly were a collection of information. They were filed in cabinets and other collections (see the image at the top of this article which shows the creativity of the ways in which forms were organized). These organizations of forms were designed so they could be easily accessed, managed, and updated, given the technology of the day.

  • So if the paper forms of previous times formed a database what kind of database was it?
  • Can we learn anything from this database?
  • Does it explain why systems were so much easier to implement back then?

Databases used by organizations before computers

One of the most common misconceptions is that back then data sets were small and they didn’t handle the amount of data that is common today.

Consider the following images of the Metropolitan Life Insurance company in 1904.

In these rooms they handled 10,000,000 insurance policies worth around $400 million ($9 billion in today’s dollars).

It was all done using paper filing cards and forms.

And it was done accurately, without significant errors.

Large amounts of data were routinely handled by paper-based systems in the first half of the twentieth century.

It is not because the computing elements were human beings as is commonly thought. When you ask IT people why those systems just seemed easier to set up you get pretty much the same answer — that it was because they were instructing people, and human beings have attributes like common sense and can respond to unusual situations.

When we look closer it becomes clear that that is not the case.

If you look at the above picture you will realize that these people are following exact instructions. Most of you reading this article are experienced managers and will realize that when you are coordinating the activities of hundreds of people there is no way you can tolerate individual actions. These people were programmed. Their instructions were of the order of “transfer the number from this form to column A and add it to column B.”

These people were the computing elements of systems, and the accountants were the programmers.

The database was the set of forms and ledger books that they worked with.

We have to give up our 21st century conceit that databases are something solely connected to computers. Conceptually there is no difference between data stored on paper using ink markings and data stored using an electronically-readable medium. The forms used by organizations contained enough information to generate all their financial reports.

All of the reports that organizations require to operate in the 21st century were invented and produced in earlier eras. The only difference is the speed in which they were produced and distributed, and the amount of human labor required.

Organizing large groups of people to accomplish data processing tasks requires developing procedures that are algorithms.

Algorithm: a process or set of rules to be followed in calculations or other problem-solving operations

Developing algorithms is programming. These paper systems were programmed. They didn’t call the algorithms programs they called them “procedures.” They didn’t execute a procedure they “performed’ it.

Commodore Grace Hopper, USN

Commodore Grace Hopper built that terminology into COBOL hoping it would help make COBOL a language that would be usable by business people. Functions/sub routines in COBOL are called PROCEDUREs, and you PERFORM a procedure. She also structured arithmetic instructions with the same kind of syntax that the paper-era procedures had.

SUBTRACT COST-OF-SALES FROM GROSS-PROFIT GIVING NET-PROFIT

That effort failed because most of the work in implementing computer systems involves the computer technology itself rather than the business issues. (Interestingly, however, some of the ideas in COBOL, such as the representation of currency values, are superior to those used in most computer languages today.)

Any attempt to try to incorporate the ideas of the pre-computer era into modern technology stopped with COBOL. The young programmers who were developing the new languages and databases had no experience with the pre-computer systems and put no value in them. Hence after the 1950s they were ignored.

The technology of the 1960s and early 1970s wasn’t up to the challenge of implementing any kind of reasonable simulation of paper forms. It wasn’t able to handle relational databases either. It wasn’t until 1974 that IBM started work on relational databases and despite all the hype around them it wasn’t until 1979 that Relational Software (now Oracle) was able to start selling a commercial product. The problem was that disk storage wasn’t commonly available until the 1970s and even then it was very small by today’s standards, and slow. This made direct access of data, which all database structures require, very slow. The major reason that relational databases were so slow to appear on the market is that while an in-memory implementation of relational tables and the relational calculus is relatively straightforward, an implementation where most of the data is stored on a block storage medium like a disk is fiendishly difficult.

It is unfortunate that all that effort was put into relational databases, which were never really suited to enterprise systems. If it had been put into trying to reproduce the effective systems of the paper era, the subsequent tragedy of massive and continual system failures would have been ameliorated if not avoided entirely. For my comments on the general unsuitability of relational databases please see my article

So how would we describe the database of paper forms used before computers in terms of 21st century database theory?

The paper forms comprised a “graph database” that evolved to serve the needs of organizations. A graph database is described by a set of nodes with named properties that are connected by edges. Paper forms are linked by ‘codes’, things like employee numbers, project numbers, and G/L codes.

All companies have forms and unique IDs for the entities represented by these forms. These form a natural graph of nodes (forms) and edges (the codes that point to other forms).

This is unlike a relational database where there are no direct connections, except maybe for primary keys which are a kludge added onto relational databases when it became clear that the relational model was too slow for some applications. A graph database stores the relationships explicitly.

A graph database can access data far faster by following edges rather than using a complicated algorithm like a table join. For a more formal description of the kind of graph databases this requires please see my article as follows:

Hidden in the processing of paper forms was the concept of “context.” Context is a fundamental concept in business processing, so fundamental that it is usually overlooked in business systems analysis. Everything in business is processed within a particular context. For example, that could be the context of a project, an employee, or a time sheet. When processing data inside a context everything is restricted to that context. When in the context of a project only things that relate to that project are worth considering. If you request a list of time charges then only charges to the project are relevant. When built into systems this context reduces the complexity of development by an order of magnitude.

For a more detailed description of the graph database of forms used by organizations before the advent of computers please see the following article:

Typical business operations using a context-sensitive graph database can be done with a few mouse clicks by a non-programmer. This is as opposed to the current standard practice of having this kind of business logic programmed by an expensive database specialist and taking a minimum thousand-fold more in time to implement.

This is one of the main reasons why it was so much easier to develop systems before computers than it is today. It isn’t a matter of human beings being easier to instruct than computers, it is a matter of organizing the data properly and understanding issues like context. The IT industry never really tried to understand what principles these older system ran under. There was a sense of superiority, that the new technologies were so much better that there was nothing to learn from these older systems based on simpler technologies.

Basically the IT industry threw out the baby with the bath water. Rather than taking lessons from the past and emulating the easy-to-use mechanisms they developed, the IT industry has adopted a hyper-technical and fragile approach that has wasted huge amounts of money and time of organizations all over the world.

Hopefully this article has given you an overview of why the database technology of the past was much more sophisticated and powerful than the IT industry has ever given it credit for. In fact, given that our current database technologies seem to be leading our systems into all-too-frequent disasters it is fair to say that they were superior.

For software that implements these ideas go to the Formever designer’s website.

No responses yet

What are your thoughts?