The actors
ACS: the Australian Customs Service.Murray Harrison: (a public servant) CIO for Australian Customs, 2002-.
Chris Ellison: (Liberal politician) the Federal Government Customs Minister, 2001-2007.
Eric Roozendaal (Labour politician) Ports Minister for the state of NSW, 2006-2007.
Joe Ludwig: (Labour politician) Senator. Opposition Shadow Minister for Ports 2004-2007
Tradegate Australia Ltd: Established in 1989 as an industry representative body to develop and implement e-commerce services for Australia's international trade and transport industry.
EDS: Ross Perot's international IT consultancy and outsourcing supplier.
ICS: the Integrated Cargo System (ICS), part of the CMR project.
CMR: the Cargo Management Reengineering programme initiated by the ACS.
COMPILE: The ACS system for managing import declarations, duty/tariff transactions, quarantine and bonding.
EXIT: The ACS system for managing export declarations and clearances.
Verisign Inc.: An internet security digital signature services company.
IBM: International Business Machines.
COBOL: One of the first, wide spread, and most successful programming languages.
C: A programming language
CICS: Customer Information Control System, mainframe transaction server.
IMS: The Information Management System (database with transaction processing capability).
P&T: Post and Telecommunications Agency (aka Plain old telephone service (POTS)).
(figure 1. Key actors and technologies prior to ICS go-live)
The ACS Expands Outsourcing Initiatives
Going back to the 70s, 80s and 90s the Australian Customs Systems Division was at the front of transactional systems development. They had decades long experience developing resilient, scalable, available tech systems over a national network employing electronic data interchange. The first systems were dial up, then dedicated leased lines from the P&T. Systems architectures ran on mainframe, using COBOL and CICS and IMS, some C development and other software."The switch to Java and internet in the late 90s put a huge strain on our capability, we just weren't staffed up in those areas. Stupid hiring freezes across the civil service. There were restrictions on hiring new programmers into the civil service, the Department of Finance had no problem giving permission to pay consultancies 100K or 150K a year for bodies, but point blank refused to sanction recruitment of graduate programmers at 30K (even if we could get them). The net result of the ban on being able to hire in people was to choke off our civil service tech capability in Java and Internet. [Customs]"In 1997 EDS, the global IT services company, secured a number of lucrative multi-annual service and development outsourcing contracts with the Australian Customs Service (ACS). EDS (Electronic Data Systems, founded by Ross Perot) specialises in business process and IT outsourcing services. Prior to the outsourcing contract with EDS the ACS managed, developed, deployed and delivered its core back-end systems almost wholly in-house. After the outsourcing deal, responsibilities for the ACS's core IT systems transferred over to EDS with gradually less direct involvement by ACS internal staff.
The old system worked well and was integral to the smooth operation of the national import and export system. It delivered high availability, responsiveness, and functionality for interactive or batch operation. Around this time Tradegate executives commented that the shift of system maintenance and development responsibilities (in additional to operational services) from ACS personnel to EDS resulted in a virtual shutdown in cooperation between the technicians on the 'inside' (of ACS-EDS) and those on the 'outside' (Tradegate).
The new supplier had no interest in involving Tradegate, you might say they went out of their way to ignore them. [Tradegate member]
We're all the victims of the tendering process. It was driven by the Department of Finance, the Tax Office. It was all about finance, cost savings, and Tradgate got slated. [Customs]One consequence of the ACS-EDS deal was that the ACS lost more and more of its own specialist Customs IT expertise as employees transferred into EDS and then gradually moved on into other areas in EDS globally. Staff transfers from ACS to EDS also altered the apparent cost structure for getting regular work done, for example the ACS paid contractor rates for the services of EDS consultants, many of whom had previously been public servants for the ACS.
Customs IT people had opportunities to join EDS, a lot of them did, a civil servant's pay isn't great, but some stayed, for different reasons. [Customs IT]
It creates tensions, side by side, people you used to work with but now they're getting paid more, a lot shipped out, our knowledge has a value elsewhere in the organisation [EDS]. [ex-Customs IT now EDS consultant]CMR and ICS
In 1998 the ACS and EDS announced a massive $A25 million programme of IT systems renewal called CMR (Cargo Management Reengineering). The CMR was going to be the Australian Customs Service's next generation IT system; a single system for managing, recording and controlling the movement of all goods movements into and out of the country. The Minister had made commitments in Parliament and the Customs IT innovation plans had been talked about nationally and internationally for several years. The proposed Integrated Cargo System (ICS) was to be the heart the CMR project. ICS would replace the ACS's venerable EDI based COMPILE system.
They [EDS] send in the 'A-Team' to get the CMR and ICS deals. They won. The 'A-Team' gets the deal the 'D-Team' gets to deliver it... people moved, Customs was hollowed out. [Customs IT staff]
It was a huge strategic shift, going from a 'community portal' sitting on top of Customs' EDI interfaces on mainframes to Customs outsourcing development and delivery to EDS. [Industry observer]After over five years of development, delays and an increasingly fraught relationship between the ACS and EDS, ICS finally went 'go-live' on October 12 2005. Its users were compelled to switch over to the new system from an electronic trade management system that had served them for over 15 years.
The ICS go-live, October 2005
At the time of the ICS roll-out on the 12th of October 2005, Port Botany in Sydney was handling $A100 Million in trade per day. Up to the point of ICS 'go-live' Tradegate had provided essential front-end services on top of the ACS back-end. On ICS go-live day the old systems were simply turned off!
...they brought us in for training three months before go-live. [Trader]
Only a couple of the biggest importers were involved in user acceptance, I'm not even sure they were involved at the early stages for user requirements. [Trader]
ICS has the requirements of other government departments: Border Protection, Statistics, Taxation. But no business logic worth talking about. [Trader]The operation of the new system was marred by widespread problems. Security access flaws allowed users to access and view other Traders’ import documentation, revealing confidential information like price, supplier details, quantities and delivery dates. The system was also perceived to be unreliable, badly designed, slow and buggy.
Errors come back ages after you make a clearance request. Why didn't the computer system pick up on it first? [Trader]
The error handling has problems, delays, I'm still waiting for responses to messages [in the system]. I might have 20 messages queued somewhere, meanwhile our containers are sitting on the tarmac at Kingsford-Smith (Sydney Airport). [Trader]
A couple of weeks after ICS go-live the key seaports, Port Botany and Melbourne Port, had ground to a halt as containers piled up. Trade effectively stopped due to ICS processing delays or other problems with the new systems. Australia's trade clearance system ground to a halt from October 12 and for the next six weeks containers and cargo accumulated at ports and bonded warehouses. The situation reached crisis point when Port Botany and Port of Melbourne ran out of room to store uncleared containers. The clearance backlog came about partially because ICS throughput was just 30% of normal handling rates, this led to excessive cargo dwell times at airports and sea ports.
I need Customs clearance to move them out of the port. [Trader]
That's the power of Customs, we are just pleased if we get their attention. [Trader]The spectre of vessels (sea and air traffic) unable to disembark cargo was now a fact, Australia’s connections with the international trading system had seized up.
The system has fallen apart. Traders are getting seven days turnaround instead of seven minutes. In COMPILE declarations had 98% success on first time entry. In ICS, at roll-out we started with 60%, that's 40% fail! It improved slowly, 60%, 70%, 80%. [Tradegate employee]
Let me put it like this, ICS was already a couple of years 'late'. Even when the pilot roll-out started in March 2005 it was obviously generating problems. Parallel running was a complete failure because no one was going to switch over until they shut down COMPILE. There was a push to switch people in July and then finally they shut down COMPILE in October and the problems grew and grew with the shipping volumes. For traders the run up to stocking Christmas shelves starts in August and goes exponential. That's when we found ourselves in a full-blown crisis; Airports 'chock-a-block', container terminals full, container ships parked off-shore unable to unload. Worse and worse, so no, we did not have a happy Christmas! [Trader]
Postscript
A year later Senator Joe Ludwig (Lab), speaking to the Customs Brokers and Forwarders Council of Australia, reflected on the aftermath of the CMR programme and the ICS roll-out in particular. By this time Customs had a new CEO, Michael Carmody."The Cargo Management Re-engineering project, which was part of the trade modernisation agenda will be remembered as one of the all-time greatest implementation failures of a government project in Australia’s history. Its budget blew out to ten times the original estimate, it saw constant delays and, when finally switched on, it nearly brought down Australia’s trade industry." [Senator Joe Ludwig (Lab)]
Edited narrative interview with Australia Customs IT programme manager.
"So the first wave of our outsourcing experience was a massive learning process. We chose a strategic partner, and then switched, and then switched again. Partially driven by Government policy, partially driven by necessity. Each partner was basically giving us a Rolls Royce approach. But as it turned out they were also using us as a kind of training centre for their new hires. We ended up being the first post for their graduates and internships. They'd be trained for a year or more on our systems, processes, tech skills, management skills then ship out to consultancy gigs elsewhere. [Customs]"
"Since then we've taken a different approach. We've hived off little pieces. The way we wrote our tendering would allow an incumbent to continue (rather than switching at each renew), but we also started to divide out the work. Now we split it across roles rather than systems. They (suppliers/vendors) supply so many people as system architects, business analysts, programmers. But we keep control of the systems, they're our systems, we own them. [Customs]"
"We have gradually ended up with two strategic partners and loads and loads of smaller external vendors. That has worked really well. While X are our dominant partner we have loads of small specialist partners, local, Australian. It plays out well politically but it has really paid out with a regional ecosystem of expertise. [Customs]"
"We now build, deliver, operate (and own) everything. The internal team is just 30 people, pure technical excellence, constantly training up on new tech, deep knowledge of our architecture. But it scales because we're a programme management operation. It is our own Customs system framework, we own, develop and manage it. My team work side-by-side with the vendor people. There might be ten or twenty times as many vendor people around but we're in charge. It took us 5 years to reach this point, building internal capability and having a capability for the future. [Customs]"
