Horizon IT system

This Fujitsu/ICL system for post offices clearly had some major failing in it.

But does anyone know what it was? General journalism isn’t likely to tell us and the computer press might not go into the details of the accounting.

Post offices were being registered as collecting more cash than they had. That’s why post masters were said to owe the system money, or to have been skimming cash and all that.

OK, but what was the error of the cash counting system? What bit was it that went wrong? Was it doubling the amount that should be there for TV licences or something? What, exactly, was the accounting error?

Anyone know?

34 thoughts on “Horizon IT system”

  1. Yes, I’m looking for something a little more specific. Not what was the cover up, who has been reprieved, the personal stories. What, exactly, was the fuck up in the system itself?

  2. My bet is on something as simple as an accidental double-click on a button not being handled correctly.

  3. Surreptitious Evil

    Tim,

    From an article linked in the CW article John G posted:

    The alleged problem investigated by the CWU involves the process where subpostmasters transfer money from a core Post Office branch to a remote branch created to serve rural areas, known as an outreach, which is basically a branch on a laptop. These processes are known as remittances.

    According to the CWU, branch payments can in certain circumstances be duplicated – in effect creating a record of money that does not exist, recorded by the system as being in the outreach branch. It would be the subpostmaster’s responsibility to make up this sum unless the duplicated payments are removed.

    The CWU said that, in a case documented by investigators, a core branch transferred cash to an outreach branch and Horizon reduced the accounts of the core by the amount transferred. The outreach transferred in the amount sent and Horizon accepted the correct amount.

    But Horizon did this four times, causing a loss of three times the actual amount sent. The discrepancy was for money that did not exist.

    “We have evidence that Horizon has duplicated the accepting in of a remittance (on at least three occasions). This is despite the [payment’s] barcode having only been scanned in once,” said the CWU email.

  4. Surreptitious Evil

    I would _guess_ that it was some sort of timing error incorrectly handled – payment submitted, confirmed as written to ledger = okay. Occasionally, and as we are talking about remote branches which are more likely to have [email protected] comms, you might get: payment submitted, no confirmation received; payment submitted etc … confirmation received =/= okay – incorrectly handled, you would get multiple deductions but only the one addition at the other end.

  5. BlokeInTejasInNormandy

    SE

    Any proper transaction processing system solved this decades ago, so I doubt that’s the explanation..

  6. “Any proper transaction processing system solved this decades ago, so I doubt that’s the explanation..”

    We’re talking about the Post Office here, a quasi-State body, so just because everyone else solved it years ago doesn’t mean they have………

  7. Yeah, yeah, yeah. State organisation. Government involved. Dot dot dot.

    But the need for ACID properties in bank-like systems is extremely well known. Leaving that out is like saying yeah, well, we’ll just write our own software to do addition. Or inventing a complete replacement to Unix to run the thing.

    Yes, it’s **possible**. But it’s unlikely. It’s probably much more to do with something stupid, like using the public internet to transfer between post office and central, and forgetting about some rare corner case when stuff can get there twice..

  8. Any proper transaction processing system solved this decades ago

    Indeed, but it seems very easy to write your own crap transaction processing system, especially when a billion pounds is dangling in front of you.

  9. In my understanding, what it was was:

    Correct functioning:
    PO sends ‘credit £x’
    HQ receives ‘credit £x’
    HQ credits account
    HQ sends ‘acknowledge credit £x’
    PO receives ‘acknowledge credit £x’
    PO removes item from queue

    Failed functioning:
    PO sends ‘credit £x’
    HQ receives ‘credit £x’
    HQ credits account
    HQ sends ‘acknowledge credit £x’
    PO /doesn’t/ receive acknowledge
    PO retries
    PO sends ‘credit £x’
    HQ receives ‘credit £x’
    HQ credits account
    HQ sends ‘acknowledge credit £x’
    PO receives ‘acknowledge credit £x’
    PO removes item from queue

    PO now has one ‘credit £x’ recorded, but HQ has two ‘credit £x’ recorded.

    It’s a classic network transaction confirmation problem. In fact, a Networking 001 problem. It’s not even undergraduate level concepts. How do you know where a failed message has failed? Has the message to HQ failed, or has the acknowledge failed? The solution is to either use a sequence chain, or *not* transfer ‘change’ messages, but transfer ‘updated balance’ messages:
    PO sends ‘account balance is £x’
    HQ receives ‘account balance is £x’
    HQ updates account
    HQ sends ‘acknowledge account balance is £x’
    PO /doesn’t/ receive acknowledge
    PO retries
    PO sends ‘account balance is £x’
    HQ receives ‘account balance is £x’
    HQ updates account
    HQ sends ‘acknowledge account balance is £x’
    PO receives ‘acknowledge balance is £x’
    PO removes item from queue

    This results in the PO recording a balance update to £x and HQ recording a balance update to £x.

    Of course, this has it’s own problems of multiple access/single resource (what happens if somebody else does a ‘balance is X’ between your retries) but is solid if you have exclusive access during the whole transaction. To do that you’d wrap it in ‘open for exclusive access’/’close for exclusive access’.

  10. Not getting that at all.

    Because for every shortfall in error in the outreach branch, there would have been a corresponding surplus error at the core branch. Surplus errors should have been (generally) identified around the same time as the shortfall errors, stonkingly obvious to anyone (and I mean anyone, even public sector) involved in the monitoring/auditing.

    Were surplus core branches quietly pocketing their surpluses? Were the investigators even more stupid than the usual “totally fucking dense” that passes for employable around those parts? Something else?

  11. We saw this all the time in projects.

    My late missus and I use to work on a project that was blighted by the fact that the designers did not know the basics of transaction processing and instead of buying a product to perform these tasks invented their own wheel. She (who was much brainier than me ) would listen in despair in meetings where she was supposed to implement a “solution” to a problem which had ceased to exist everywhere else north of the Limpopo 20 years before. My bit was less system-critical and I could afford to jettison the design completely, write something that worked and no one noticed.

    Proper use of an off-the-shelf transaction monitor would have prevented the Post Office problem, but software is a capital “cost” and it is bizarrely cheaper in accounting terms to waste money on half-trained Indian developers building hexagonal wheels with no spokes.

    A good example of how this sort of this “used” to occur. Thousands of years ago, I worked on a project where users were receiving doubled up versions of their data on their screens, but slightly ( by a few bytes) skewed so that what they saw was gobbledegook. It turned out that the user would send a request – the comms was X25, which in those days was still new – there was a delay in response ( for whatever reason, crap telecomms, slow mainframe ) and they would press “send” again, so two requests for data would be sent to the mainframe and they’d be both processed and returned but with a few microseconds difference causing the doubling of screen data. Someone had forgotten to program an “ACK” in the datastream handler. It was a problem that ran for weeks and was solved in a few seconds once someone had decoded the X25.

  12. My guess is that ultimately it’s because Fujitsu thought it could save a few quid by hiring Pajeet to do the coding instead of paying people who know what they’re doing.

    See also: Wipro, affectionately known as Shitpro by unfortunate customers.

  13. “Yes, it’s **possible**. But it’s unlikely. It’s probably much more to do with something stupid, like using the public internet to transfer between post office and central, and forgetting about some rare corner case when stuff can get there twice..”

    Read this tale of Post Office management incompetence and maybe you’ll reassess what the PO was capable (or not) of managing:

    https://www.postofficetrial.com/2020/08/trouble-up-north-of-england-pt1.html

    https://www.postofficetrial.com/2020/09/trouble-up-north-of-england-pt2.html

  14. Basic fault lay in PO top management. New computer system says that over 600 sub-postmasters/mistresses are embezzling in one year whereas previously it was one in six or seven years. Sub-postmasters say – we haven’t stolen a brass farthing, your computer is faulty. Computer guy insists “No, it’s fine” despite knowing that’s a lie. PO top management takes the one word of one liar over 600-odd honest men/women and fails to order a thorough investigation before calling in plod and sending innocent people to jail.
    Guy who perjured himself should be in jail, PO top management should be in the stocks.

  15. Steve,

    “My guess is that ultimately it’s because Fujitsu thought it could save a few quid by hiring Pajeet to do the coding instead of paying people who know what they’re doing.

    See also: Wipro, affectionately known as Shitpro by unfortunate customers.”

    It’s really just about shitty management from top to bottom.

    I would never hire one of the big agencies because their whole ethos is making money. I’ve worked for a couple when I got desperate. I don’t mind people making money, but I do think people should care about what they do when they’re doing it. They’re like timeshare salesmen or people running clip joints. They lie to customers, they do the shonkiest job that can get them past testing, with the most junior people they can get, because that maximises profit.

    The only defence that a client has is massive levels of testing. But if you’re the sort of idiot who hires these companies, you probably don’t do that. You think they’re professionals because they all turn up in suits and have shiny Powerpoint presentations. And ultimately, you’re in government, so who cares? Even if you fuck up and get fired, you get another executive job in some quango. How many of the Connecting for Health people are sucking dicks at truckstops because that’s the best job they can get?

  16. Steve +1000

    I was on a project where the architect actually tried to defraud his own company by not paying for software licences. The department that sold the software only found out because I was ordering hundreds of temporary test lics to do capacity testing when in fact they were not even aware that we had a copy (x4 actually) running on our servers. He was screwed anyway, because he was trying to run with software that was two versions out of date and would have blown up on the first day, which we discovered when we ran the load tests.

  17. The Pedant-General

    “The only defence that a client has is massive levels of testing.”

    But unless the testing was done on the actual conditions – shit laptop, shit connection, fat fingers – your test doesn’t replicate the real world condition.

    ““Yes, it’s **possible**. But it’s unlikely.”

    No – I reckon it’s a racing certainty. The Post Office KNEW that they had fucked up and they still insisted on bankrupting and jailing a large number of innocent people. There should be proper actual jailing and bankrupting of all involved – not the poor bloody taxpayer.

  18. “There should be proper actual jailing and bankrupting of all involved”: hear, hear.

    Though I’d settle for public floggings and the occasional hanging.

  19. Agree with BoM4, the IT agencies are timeshare salesmen. Same problem with Boeing outsourcing the MAX to India.

    Playing Devil’s Advocate though: you’re head of the Post Office, you need a new IT system, what are you supposed to do? Try to do it all in-house? Find an IT agency which isn’t incompetent? (How can you tell?) Find a second company to check the first one’s homework?

    Related anecdote: A couple of years ago we had a vacancy for a developer. We declined to interview the candidate who had worked on Capita’s army recruitment omnishambles.

  20. Pedant,

    “No – I reckon it’s a racing certainty. The Post Office KNEW that they had fucked up and they still insisted on bankrupting and jailing a large number of innocent people. There should be proper actual jailing and bankrupting of all involved – not the poor bloody taxpayer.”

    Not going to happen. You think the sort of wankers running the state are going to do anything but some public enquiries, a few slaps on the wrist? It’s taken over 15 years for this to reach this point. Add another 10 years at which point the execs will be too old and “not in the public interest”.

  21. Cant say as to the cause. But a popular –and one of only a handful left–PO went out of business near me–overnight– and I later found out it was to do with being accused of wrongdoing as part of this fiasco.

  22. Andrew M,

    Ideally, you buy something already built and in use by others. If that’s not available, you find a company already producing something closer to what you need and either change your process or get them to make small changes.

    If what you need isn’t those things, you should run it in house. There are no advantages of outsourcing custom code. Software like that isn’t a scale operation. It’s not like making family cars where there’s efficiency at scale. Each software team at IBM or Accenture is pretty much a silo. If you’re working at Virgin Media and call the IBM team, they’re a team of 6 people who are dedicated to Virgin Media. So what are the advantages of paying IBM rather than just setting up your own team of 6 people? They’re going to be on the similar wages either way.if they’re cheaper it’s because they’re inferior. Oh, and IBM are taking their cut.

    And the downside is that they have different motivations. I’ve seen a programmer get bollocked by her manager for telling a client an easier way of doing something. Why? Less billable hours. They also do things like pass off amateurs as specialists. Maybe they done have an ios developer, they’ll give someone a book and assign him.

    None of this means you can’t offshore work, but you offshore the low grade shit. You retain development people who can do the really critical parts, and they can also review the work of the offshore team. If you have some particular technical needs for a part, you find a small specialist agency to just do that one part. Even if you pay them more than big agency rates, they’ll be worth it because they’ll do a professional job.

    And if you’re CEO or some other senior exec, you should have met a good guy along the way who can run a software team. You hire/poach him, paying above market rate and get him to set it up.

  23. The government used to have CCTA, the Central Computer and Telecommunications agency. They were the in-house government computer consultancy for all government departments. They also provided the ‘Intelligent Customer’ function when government were buying big projects.

    They were closed down for political reasons in the 1990s. Since then, government projects have been crap…

  24. Remember 25 years ago doing user testing. on a very well known system and noticing that for the function to enter multiple transactions it was updating the header information but leaving the transaction data the same, how this has slipped past everyone to reach me a simple junior doing user testing that I was at the time was because I did the test using real data not all the fake rounded up similar amounts supplied for the use test so it was obvious that it was different.

  25. Fifty-odd years ago I was a trainee computer programmer and, during one Long Vac I was given the task of converting our in-house programme for granule drying into Algol 60 to run on the new computer at Wilton. It didn’t work (technical reason: I inputted 0 as a real number, not an integer but the compiler couldn’t cope) so we fudged it. Two months later my mentor (seriously bright – Girton Maths when Trinity didn’t take women) wrote a letter to me at my college to tell me that she had got “them” to correct the programme for the Algol Compiler on the new computer.
    If you really want to get it right, you can do so. Computers only do what you tell them to do (but they can be extremely irritating by interpreting what you tell them as the words of the prohgrammer not of your own).
    Neither ICT/Fujitsu nor the Post Office, nor Tony Blair cared enough about getting it right.

  26. dodgy geezer,

    “The government used to have CCTA, the Central Computer and Telecommunications agency. They were the in-house government computer consultancy for all government departments. They also provided the ‘Intelligent Customer’ function when government were buying big projects.

    They were closed down for political reasons in the 1990s. Since then, government projects have been crap…”

    CCTA didn’t run the projects and had no power over procurement or development. Their main activities were creating standards and methodologies.

    And government’s been crap for a very long time. British Leyland, Tanganyika groundnuts, Concorde. The US governments have the same issues.

  27. The hardest and most important part of any software project is writing the project specification.
    In almost every project, the spec is insufficiently well written, which has all sorts of detrimental effects on the outcome.
    How can the developers write the software if they don’t know exactly what it is they are required to produce? How can the client know that they are getting what they want, to a reasonable schedule, at the price that was contracted? How can it be tested to ensure it all conforms to the spec, if the spec is fuzzy and confused?

    Here’s an example of software that absolutely has to run perfectly, and how they do it;

    https://www.fastcompany.com/28121/they-write-right-stuff

  28. As BIB says, it often comes down to a rubbish spec up front. Of course the big four, a.k.a. the usual suspects, love this (obviously are going to keep quiet about any deficiencies they notice) – it means they can (deliberately ?) supply something that meets the spec but which they know isn’t what’s wanted/needed and make up their profits on the extra work when the customer realises. Of course, by that time the customer has two choices – pay your over-inflated rates for the extra work, or scrap the whole lot and pay someone else to do just the same all over again.
    But you have to remember that back when Horizon was built, “the internet” as a ubiquitous and mostly reliable network was just something the universities had and many had never heard of. For most people it was a choice between dial up (the old “listen while the modem screeches towards a connection” thing that’s thankfully a long gone memory for most of us), or the more expensive but also more (but not totally) reliable ISDN with fast connections, or the eye wateringly expensive leased lines. But regardless of that, there’s no excuse for the sort of errors the system was prone to – as others have already said, it was already a long solved problem if you cared enough to actually learn about your trade.
    But ultimately, the problem was the combination of attitudes across multiple organisations and manglement levels that refused to admit the possibility of “computer errors” in the face of overwhelming evidence. According to what I’ve seen & read (nothing more than is generally available to the public I’ll hasten to add), it’s not like there weren’t some people pointing out the problems – but being ignored.
    And compounded by the design of the system where the sub-postmasters were forced to accept what the computer told them (even if it was obviously wrong) in order to continue trading – and thus allowing the PO to accuse them of deliberate fraud, as in “well you clicked to accept those figures, you were obviously lying”. As an analogy as to why that’s worng, it would be like HMRC demanding that you do your tax return on 6th April (first day after the end of the tax year, long before some information is available, and when you know that the figures are incorrect), and then charging you with tax fraud for complying with their demands.

Leave a Reply

Your email address will not be published. Required fields are marked *