24. Wandering Samurai
Treading water
For years I worked as an odd job man in tech. That’s the only way I can describe the life of a consultant, especially one with my profile. I hoped to be able to use my knowledge and experience (and minor fame) to help companies work on cutting edge technologies, pushing the frontiers of knowledge and practice. It turns out consulting is never about pushing the envelope of change. It’s about helping people take baby steps and teaching the basics of something they can’t create the capacity to innovate for themselves.
I got to see work from the outside for the first time. Usually, we’re so embedded in work that we can’t see it for what it is. For most companies it’s easier to make changes by hiring outside help than to rely on the people they’ve employed. The sad truth is that once people have been hired, they are typically condemned to be faceless cubicle workers, no longer viewed as trusted advisors, only as indentured slaves. This is the true face of work in our Western Democracy and everywhere else. Workers become downgraded to cattle or cogs in a machine of production. No one wants or expects anything from us. The mystery is gone. The person has been stamped for basic approval, and categorised for one minor purpose only. Only external advisors can be looked up to.
It was for this precise reason that I resisted being hired by any company. I knew that, as soon as I became an employee of some company, my status would be transformed into that of a minion and my capability to make any kind of difference to the world would be gone. Individuality within a larger organisation would take years to recreate, starting from the bottom again. I’d done it several times before when changing from physics to computer science, but I’d since been spoiled for too long by living off the reputation I worked for in academia to give it up so late in the game.
Around 2010, the term Internet of Things was coined in order to rebrand the long standing idea of Pervasive or Ubiquitous Computing that had been foreseen in the 1990s. The name “Internet of Things” was borrowed from the thesis of some student from Berkeley or Stanford. Silicon Valley analysts jumped on it, always happy to make the local students into the heroes of tech. Starved of novelty for too long, they hyped it. The commercial success of the Internet and the Web could now breathe new life into the old idea. This got the attention of network giant Cisco Systems, and that in turn led to the projects I worked on for the first years after quitting CFEngine.
Around 2015, I flew to San Francisco and drove to Milpitas, San José, in Silicon Valley to join a project looking at controlling the electronic system of control known as the CANBUS in smart cars. They wanted to apply a control model like CFEngine to manage the data collection, but they didn’t understand the scope of the problem. I told them the it was inconceivable that they would be able to collect the data from every vehicle to the cloud in real time At first they were bullish, and gave me their “We are Cisco!” look, which I liken to the Star Trek looks “We are Kingons!”. There is nothing too great for us!
A few weeks later, they admitted that perhaps I was right, but they wanted to do something anyway. We didn’t get very far with that, but ended up defining a draft standard for management of the Internet of Things devices with the IRTF. It was based on my scaling ideas in CFEngine and used a little Promise Theory. Mainly it was a common sense separation of scales.
The ideas from SDN (Software Defined Networking) were still fresh in my mind. There was a role for networks and data pipelines that was missing from the technology that no one was really thinking about. In Silicon Valley, people are too focused on money to see technology opportunities, but they assume they are the cream of the crop, so a new idea can’t be a good one. It sounds strange perhaps that the home of the computer revolution should be blinded by its own success. Yet it remains true. People are not as innovative by nature as they prefer to think.
The Cisco project was nice enough, but one advantage of being in Silicon Valley was that I could spend time with my many friends there. I would especially visit Paul Borrill for our regular breakfast sessions or Sunday afternoon Sushi and Mochi Ice cream sessions. I would sometimes drive out to Pleasanton to see my friend Steve Taugott and his lovely family, envying his Maker mindset and supportive family. I made new friends too, sometimes using Tinder to meet fresh faces, sometimes making acquaintances through work. I needed the escape.
My interest in embedded systems grew. I played around with a Raspberry Pi computer, but I was never able to find the components or the time to spend on turning that interest into something useful. Eventually the Cisco project dwindled as the various managers got shuffled like a deck of cards on a roulette wheel.
Smart spaces
What the Cisco project started for me was the interest in Smart Cities. They already has some people looking to develop a sales package to “digitalize” city operations. It wasn’t too advanced, but when new challenges arise the big companies band together to make money by selling dud equipment and software to big government. The corruption is stupendous, but it’s the norm in public life.
Cities and governments look to the large companies, who are set up to deal with the bureaucracy of government. They approach IBM, Cisco, Microsoft, Accenture, Cap Gemini, and so on, no matter whether they have an answer or not. These “trusted partners” are the basic laundromat of funding that flows with public money. They can always put together some project, steeped in mediocrity, to accomplish nothing for a lot of money. Just as the EU can throw a little money at innovators, then throw a massive auditing company ten times as much to check whether they used it “properly”. The truth is that many companies and government departments still use Excel spreadsheets on personal computers and store records on CDROM in archive cabinets. The ubiquity of the cloud and digital systems has been greatly exaggerated.
I loved the idea of Smart Cities, and how one could apply technology to improve infrastructure and community living. I had always wanted to work on a project of large social importance, so I worked for a while on how to understand what role embedded computing might plausibly have in making a city “smart” (whatever that meant). I looked for contacts and met a couple of enthusiasts in Norway who were doing smart city as a hobby. Alas, it didn’t get me any work, so I eventually gave it up. This is not how companies and projects are won. There was a new generation of developers by then. Who in that space would care about some unknown technologist? The documents I wrote were squandered on my disk for several years before I found use for them.
Computers, Physics, and Time
I got to know Paul Borrill after he reached out to me to have dinner in Palo Alto some time around 2011. Paul’ is a gregarious and perky Brit, with a varied career in tech and a smooth gentlemanly manner. I was still active with CFEngine at the time. He had seen some of my work and had become interested in my blend of physics and computer science. He had ambitions to develop technology based on an understanding of quantum physics. Through a friend of his, he was later hired by Apple for a time. He tried to bring me along too.
I was invited to Apple’s campus in Cupertino a couple of times to discuss issues with different persons or groups. Apple discussed terms of employment with me, after meeting with them and discussing some of the cloud challenges. I would have liked to join them for a while, but unfortunately all the offers started with my moving to Cupertino in the Bay Area permanently. They offered to pay for the entire move, but with my parents getting older and with health failing, as well as some personal exposure that kept me in place, I didn’t want to move that far away, so I turned them down and continued to work with more flexible companies. It was a pity, but perhaps also a blessing. Apple is a strangely cultish organisation, like Google. I wasn’t ever able to identify with it, no matter how many times I visited. I’m very glad I never kowtowed to these companies. I visited my old friend John Wilkes at Google once to see if there were any opportunities, but 30 minutes in the lunch cantine convinced me that it was a high tech kindergarten. I couldn’t wait to get away.
At a conference in San Jose, I’d met a very beautiful and elegant Taiwanese-American girl at a conference a few years earlier. She had been sitting next to me at lunch, bad mouthing CFEngine as “so last year” compared to the competing products, not realizing who she was sitting next to. Someone at our table nervously introduced me and she laughed. Later she apologised to me, but I laughed back. Of course, we briefly became lovers whenever I visited Cupertino where she worked. Like many of my liaisons, she found a nugget of something missing in her life. Our dalliance filled a need for a while and then vanished as quickly as it had come.
Projects for the big companies are short. They don’t want to put money into anything that doesn’t immediately make more money. Before long, I was looking for work again. My friend Mike Dvorkin, whom I had met at Cisco, suggested me as a possible fit for a company called Datera. I’d not heard of them, but they were the latest thing in Silicon Valley. They were apparently looking for a CTO. Datera invited me to visit. I thought I was going to meet and learn about the company, but it was an ambush. Rather than a first meeting, it turned into a job interview by implication, and quite a hostile one, as not everyone liked the idea of my being there.
One of the technical leads was very kind to me and enthusiastic about my work, but there was another company advisor who immediately announced that he'd been amongst the founders at VMware who looked down on me with such contempt that I was practically speechless within the first moments of our meeting. I was completely taken aback and didn't answer any of the hostile questions very well. This ambush changed to tone of the trip. By the end of the trip, the tone of the CEO had changed completely too, and he even asked me if I would go 50/50 on the travel costs with the company, because he felt it was a waste of time. I told him what he could do with his suggestion and have never spoken to them since.
Eventually the work around Silicon Valley dried up and I wasn’t able to visit for a while. The valley had become like a second home to me. I knew it as well as I knew the UK by this time, but its harsh realities hide with tooth and claw behind the veneer of pleasantry.
New York, New York
In 2015, a marketing company unexpectedly reached out to invite me to New York, all expenses paid. I was to speak at a conference they apparently held each year. I’d never heard of it before, but one of the co-founders had heard of me and Promise Theory. The event was called Transition 2015. It turned out to be an apt name. For me, it was a slightly mysterious but fascinating event, held just below Times Square, in midtown Manhattan. It changed my impression of who I could be and how far I should reach.
When you grow up from humble origins, you are taught not to think you can be anything. In Scandinavia, it’s called The Law of Jante. At this unexpected conference, I met a lot of people who were born into their roles–people who were used to schmoozing with the money in New York and putting on a face for the crowd. They knew what they could be, because they were part of a network of privilege. Even after Silicon Valley, it was an eye opener.
I had no idea what I was going to, but when I arrived, I quickly saw the calibre of the speakers at the guest reception. Bizarrely to me, I was amongst celebrity speakers! I was used to that in tiny narrow tech focused conferences, but never in this kind of world spanning event. My audacity at pushing the ideas around Promise Theory into a public domain seemed to have paid off in the sense that the marketing company organizing the event had mistaken me for one of them. I needed to put my best foot forward, to adapt quickly. Everyone was playing the same game with their faces, so I needed to learn how to project that fake confidence too. There were many talks back to back, and it was going to be a challenge to not be the worst of them.
The first person I met at the conference was Geoffrey West, of the Santa Fe Institute. I was surprised to learn that he was a Brit. In fact, we stemmed from the same milieu in theoretical particle physics. He had been at Sussex University when my co-Newcastler Mark Hindmarsh had been there, working on cosmic strings. We were but one degree of separation apart. At some point, he’d changed tack and gone into what now goes by the reprehensible name of Complexity Science today. That’s what the Santa Fe Institute was started to develop, around the time when I studied it a bit myself.
Santa Fe is a private institution that offers summer courses in complexity ideas, and places itself in the same league as Oxbridge and the elite universities, at least in its own mind. It has some good people and some mediocre people who all lift themselves up by their common brand. Over the years I’ve met far too many people who believed they knew more about physics than me because they had taken the Santa Fe summer course to become an instant expert in the true meaning of complexity.
In spite of some snobby airs and graces, the talk about scaling of cities by Geoffrey West impressed me the most. Scaling was a topic that I’ve been passionate about for some time. I could see that what he was talking about was something Promise Theory ought to be able to say something about. The methods he’d used might surely give me some insight into how to make quantitative predictions using Promise Theory. Later he wrote an excellent book called Scale, in which he described his journey. It’s a great book.
I liked several of the talks, and wanted to get to know several of the people in the audience, but suddenly it was my turn: my talk was upon me! The talk was scheduled for the late afternoon, so I had a disadvantage and decided that I would start with some audience interaction to wake everyone up. I asked everyone to stand up and take some breaths before launching into my talk. Now I had everyone’s attention. A simple trick settled my nerves and made a good impression.
After the conference, I went home for Christmas and spent the week between Christmas and New Year looking up, downloading, and working through Geoffrey West’s papers I found on the subject of biological and civic scaling. Having just completed my own second paper on Spacetime With Semantics II, dealing with some civic issues, such as tenancy and economics, I was sure that a Promise Theory model would be a great approach.
As always, when I get excited by someone’s work. I reached out to the authors of the papers enthusiastically. Geoffrey West didn’t even reply, despite our conversation in New York, but Luis Bettencourt did and was quite gracious. I wrote a paper of my own, to sketch out a Promise Theory, embedding the theory about cities. I learned a bit about scaling. In particular, I felt that I finally understood the relationship between a graph and a continuum spacetime. Spacetime is an average description! Just like Feynman diagrams, graphs told about causality. Spacetime tells us about average weights and measures.
It was fascinating work, but I didn’t have more time to spend on it. I needed to earn some money. So I moved on. But, the effort was not wasted. I could use what I’d learned for computer systems, and the scaling arguments have sat in the back of my mind growing like a seed–and probably led to my later work on “trust” as a guide potential in social science.
Treatise
Even as I’ve grown to loathe many aspects of the computer industry, I recognize that it’s still my bread bowl, and always will be. I needed to keep my head in the game. The projects I’d been hired to work on were all a result of the audacious books I’d written–much to the disgust of some. I was thus very much aware of the need to keep advancing the state of theory about infrastructure and its scaling.
As I’d worked on projects, particularly in cloud computing, I had begun to write down the practical experiences and lessons I’d had using and applying Promise Theory. Examples are golden, and can only come from actual experience.
First I wrote these insights as a set of notes. Then, insisting on keeping them consistent and clear–as they became too long, I decided to turn them into a book. Years ago (2003), I’d written a book called Analytical Network and System Administration as a teaching book at the university. As the years had passed, the publisher had made the book more and more expensive until no one could afford it. This is typical of academic titles. However, eventually they price themselves out of the market, and, when the sales are zero in a year, a book may be declared “out of print”. Authors can generally reclaim the rights to a book that’s out of print. I wrote to John Wiley and Sons to recover the rights to the book, updated it, and turned it into a part of the larger story.
Analytical Network and System Administration became volume 1 of A Treatise on Systems. Volume 2 would be the Promise Theory that I was working on. The volumes took some years to complete, but it felt like an important check on all the work to show that it was all consistent and fitted into a single scheme. I make them available for free online, so that they don’t repeat the mistake of the original.
I needed to continue the work on Semantic Spacetime, as this was leading to the Artificial Intelligence that was beginning to be revived as a topic, thanks to the success of Google’s Deep Learning methods. I’d already worked out the theory, so I wanted to test it on data. I wrote a simple Graph Database (as the existing ones were unsuitable at the time) and developed some code to analyse text.
From nightmarish experiences with IBM, I’d figured out that getting data was the hard part of trying to do empirical work. The only data I could imagine acquiring in easy reach was text. There are books and webpages as far as the eye can see, available to everyone. I wrote two papers showing how one could define concepts and derived them as scaled patterns, extracted from the streams of sensory experience. The papers Testing the Quantitative Spacetime Hypothesis using Artificial Narrative Comprehension parts 1 and 2, were partly successful in showing how concepts could emerge naturally from straightforward spacetime processes, as long as they had interior degrees of freedom. Thus they gave credence to the agent model of Promise Theory. But they were not exactly what I’d hoped. Some years, later in 2023, I came to realize that the results had a different interpretation: for understanding the meaning of trust. But now I’m jumping ahead.
Hungary for something else
In 2016, I was invited to keynote at the Craft Conference in Hungary. I prepared a talk “How do you know your systems are working well?” to try to bring up the newly hatched idea of “observability” from a scientific viewpoint. I’d begun to write down my experiences about systems and had it in mind to try to explain some basic measurement theory, which monitoring software was badly in need of understanding. For once I managed to give a fairly simple and concrete talk that people understood. I later gave the same talk again in Oslo and in London at different conferences.
The Budapest conference was too big to enjoy, but I got to hang out a little with Kelsey Hightower and Adrian Cockcroft, and bumped into Andrew “@littleidea” Clay Shafer. We suggested meeting up later to eat something, but in the end I didn’t go back. The trip reminded me of many things I didn’t like about tech and its people and I didn’t go back to the conference after the first day.
Knowing its proximity to Slovenia, I’d arranged to meet a lady friend I’d made through my work in Software Defined Networking and Promise Theory. She generously took a train to Budapest from Slovenia to meet me at my hotel and we were able to spend some time together in the city.
We wandered around the city and took in the sights before we both had to leave.
A year later, I went to visit her at her home city of Ljubljana, and we visited lakes and caves and amazing places. We took the hydrofoil to Venice for the day. It was a short but intense trip. My life still had many strings attached to it, not all good ones, but the generosity of my liaisons brought great memories. Those memories, they were good ones.
