The MGS Blog

Friday, January 14, 2022

ICS Go-Live 2005 (case)

Disclaimer: The case as presented here is a partial account and has been edited for teaching purposes. This material has been simplified for brevity and narrative coherence. The case has been designed as a teaching aide to encourage independent investigation, learning, discussion and debate rather than as criticism, illustration of management practice, endorsement of any particular approach or product. Interviews and notes were gathered from private conversations, discussions, and data from documentation in the public domain covering the period prior to 2005 and up to current.

Introduction
The Australian Customs Service (ACS) has been at the cutting edge of electronic systems development for over 30 years, controlling trade into and out of Australia’s borders. The ACS's earliest systems were delivered using a variety of hardware, networks and applications: mainframes, direct connection (EDI), community ISP facilitated EDI, and internet based portals. The ACS's applications and service delivery mechanisms were precursors to current international initiatives involving the WCO (World Customs Organisation), to develop standards based Single Window Access (SWA) systems for Customs services. Single Window Access offers the possibility of 'interoperable integrated international Customs systems' and thus the potential for global control, risk management and regulation of international trade.

Background
The ACS has long been recognised as a centre of excellence for Customs systems and innovation. Historically technology providers have serviced government's needs for computing hardware, networks and system software. The need to match application capability with international trade systems (domain knowledge) against a backdrop of constant regulatory evolution meant that Customs IT innovation has been driven by Customs agencies rather than by the IT industry.

In 2001, the Australian Government updated the Customs and other related Acts in an effort to allow Customs IT to change more freely. The 'International Trade Modernisation' package updated the legislative framework for Customs to allow for a more flexible approach. It did not prescribe individual systems but rather introduced a kind of indirect approach by requiring "that Customs gazette its IT requirements." In effect earlier Customs Acts, the law under which Customs operates, had specified the IT systems to be used through law, therefore changes to IT systems had to be reflected in changes in the Act. Once the modernisation package was adopted, changes to the IT could then be notified by gazette in Parliament rather than being enacted through legislation. The IT still had the authority of law, but the IT did not need to be specified by the law.

Customs sees its role in the national and international trading system not as a 'tax' on industry but as a necessary control; regulating and protecting society while at the same time enabling trade and economic growth. Consequently the World's earliest electronic, real-time, high transaction volume, standards based, trade management networks, were Customs import/export systems. In many ways the demands of Customs agencies over succeeding decades has driven the IT industry agenda, first by specifying and establishing EDI standards, and second by providing the necessary business case for suppliers and service providers. The business case for Customs IT innovation has been based on controlling while facilitating traders' imports and exports at the lowest possible cost.

ACS e-Commerce Systems
From 1989 to 1999 the Australian Customs Service (ACS) designed, delivered, and operated 'back-end' electronic systems to process import and export declarations, duty payments and other essential trade services. ACS e-Commerce systems were back-end mainframes running purpose built software like the COMPILE system for imports and the EXIT system for exports. Licensed traders could access them directly or via a third party service operator, Tradegate. Tradegate was a national trader community organisation that cooperated with the ACS to provide end-user interfaces to the ACS back-end; its mission was to develop and deliver the e-commerce 'face' of the ACS's import and export clearance systems. Tradegate was established in 1989 as a not-for-profit organisation comprising the importers and exporters and the wider trading community as members. Tradegate had also been responsible for provisioning a secure national electronic network prior to the widespread availability of the Internet by contracting and managing specialist telecommunications network providers, first with Paxus then AT&T EasyLink. Traders around Australia could use Tradegate to access COMPILE and EXIT from anywhere at any time.
COMPILE was very forgiving, I enter code '32' in the box and it always goes through.[Trader]
Tradegate was funded by member fees, a per transaction charge for portal use and fixed costs for network access and the ACS held a statutory directorship on the board. Tradegate's executive, its members and its board considered themselves to be partners with the ACS, providing a stable, secure and effective service and a key enabler for Australia's international economic success.
In the old days Tradegate and Customs had indirect avenues. The Tradegate guy was probably playing golf with this guy, he could walk upstairs and have a chat. They got things done. [Trader comment]

The ACS Expands Outsourcing Initiatives
In 1997 EDS, the global IT services company, secured a number of lucrative multi-annual service and development outsourcing contracts with the Australian Customs Service (ACS). EDS (Electronic Data Systems, founded by Ross Perot) specialises in business process and IT outsourcing services. Prior to the outsourcing contract with EDS the ACS managed, developed, deployed and delivered its core back-end systems almost wholly in-house. After the outsourcing deal, responsibilities for the ACS's core IT systems transferred over to EDS with gradually less direct involvement by ACS internal staff. Around this time Tradegate executives commented that the shift of system maintenance and development responsibilities (in additional to operational services) from ACS personnel to EDS resulted in a virtual shutdown in cooperation between the technicians on the 'inside' (of ACS-EDS) and those on the 'outside' (Tradegate).

The new supplier had no interest in involving Tradegate, you might say they went out of their way to ignore them. [Tradegate member]

Referring to this period in a submission to the Australian Government in 2001 Tradegate claimed that
"Contact with the key technology and service development staff [has] virtually ceased. The ability to progress externally beneficial service improvements through regular contact and 'brain storming' sessions came to an end with all contact placed on a formal basis presumably due to the dictates of the ACS-EDS contractual arrangements."
It was apparent that the ACS-EDS deal altered to the detriment the relationship between Tradegate and ACS, that the previously close contacts were disrupted and not re-established when EDS entered the equation.
We're all the victims of the tendering process. It was driven by the Department of Finance, the Tax Office. It was all about finance, cost savings, and Tradgate got slated. [Customs]
An unexpected consequence of the ACS-EDS deal was that the ACS lost more and more of its own specialist Customs IT expertise as employees transferred into EDS and then gradually moved on into other areas in EDS globally. Staff transfers from ACS to EDS also altered the apparent cost structure for getting regular work done, for example the ACS paid contractor rates for the services of EDS consultants, many of whom had previously been public servants for the ACS.
Customs IT people had opportunities to join EDS, a lot of them did, a civil servant's pay isn't great, but some stayed, for different reasons. [Customs IT]
It creates tensions, side by side, people you used to work with but now they're getting paid more, a lot shipped out, our knowledge has a value elsewhere in the organisation [EDS]. [ex-Customs IT now EDS consultant]
CMR and ICS
In 1998 the ACS and EDS announced a massive $A25 million programme of IT systems renewal called CMR (Cargo Management Reengineering). The CMR was planned to be Australia's next generation IT system, responsible for managing, recording and controlling the movement of all goods movements into and out of the country. The Minister had made commitments in Parliament and the Customs IT innovation plans had been talked about nationally and internationally for several years. The proposed Integrated Cargo System (ICS) was to be the heart the CMR project. ICS would replace the ACS's venerable EDI based COMPILE system.
They [EDS] send in the 'A-Team' to get the CMR and ICS deals. They won. The 'A-Team' gets the deal the 'D-Team' gets to deliver it... people moved, Customs was hollowed out. [Customs IT staff]
It was a huge strategic shift, going from a 'community portal' sitting on top of Customs' EDI interfaces on mainframes to Customs outsourcing development and delivery to EDS. [Industry observer]
After over five years of development, delays and an increasingly fraught relationship between the ACS and EDS, ICS finally went 'go-live' on October 12 2005. Its users were compelled to switch over to the new system from an electronic trade management system that had served them for over 15 years.
ACS_CMR_ICS
(figure 1. Key actors and technologies prior to ICS go-live)
The ICS go-live, October 2005
At the time of the ICS roll-out Port Botany in Sydney was handling $A100 Million in trade per day. Murray Harrison (a public servant) was the CIO for Australian Customs, Chris Ellison (Lib) was the Federal Government Customs Minister and Eric Roozendaal (Lab) was Ports Minister for the state of NSW. Tradegate Australia Ltd was established in 1989 to develop and implement e-commerce services for Australia's international trade and transport industry. EDS is an international IT consultancy and outsourcing supplier. ICS is the Integrated Cargo System (ICS), a key element of the Cargo Management Reengineering (CMR) programme initiated by the ACS. The ACS is the Australian Customs Service.
Up to the point of ICS go-live Tradegate had provided essential front-end services on top of the ACS back-end. The existing system worked and was integral to the smooth operation of the national import and export system. It delivered high availability, responsiveness, and functionality for interactive or batch operation. On the ICS go-live day the old systems were simply turned off!
The 'refreshed' systems roll-out, it was called ICS. They presented it like an update of COMPILE, like it was a version upgrade but it should have been called something else. [Trader]
They were supposed to have launched in March, last year or the year before![Trader]
...they brought us in for training three months before go-live. [Trader]
Only a couple of the biggest importers were involved in user acceptance, I'm not even sure they were involved at the early stages for user requirements. [Trader]
ICS has the requirements of other government departments: Border Protection, Statistics, Taxation. But no business logic worth talking about. [Trader]
The operation of the new system was marred by widespread problems. Security access flaws allowed users to access and view other Traders’ import documentation, revealing confidential information like price, supplier details, quantities and delivery dates. The system was also perceived to be unreliable, badly designed, slow and buggy.
Errors come back ages after you make a clearance request. Why didn't the computer system pick up on it first? [Trader]
The error handling has problems, delays, I'm still waiting for responses to messages [in the system]. I might have 20 messages queued somewhere, meanwhile our containers are sitting on the tarmac at Kingsford-Smith (Sydney Airport). [Trader]
A couple of weeks after ICS go-live the key seaports, Port Botany and Melbourne Port, had ground to a halt as containers piled up. Trade effectively stopped due to ICS processing delays or other problems with the new systems. Australia's trade clearance system ground to a halt from October 12 and for the next six weeks containers and cargo accumulated at ports and bonded warehouses. The situation reached crisis point when Port Botany and Port of Melbourne ran out of room to store uncleared containers. The clearance backlog came about partially because ICS throughput was just 30% of normal handling rates, this lead to excessive cargo dwell times at airports and sea ports.
I need Customs clearance to move them out of the port. [Trader]
That's the power of Customs, we are just pleased if we get their attention. [Trader]
The spectre of vessels (sea and air traffic) unable to disembark cargo was now a fact, Australia’s connections with the international trading system had seized up.
The system has fallen apart. Traders are getting seven days turnaround instead of seven minutes. In COMPILE declarations had 98% success on first time entry. In ICS, at roll-out we started with 60%, that's 40% fail! It improved slowly, 60%, 70%, 80%. [Tradegate employee]
The chairman of the Customs Industry committee said 'it was chaos', 'blood on the ground'. [Trader]
Let me put it like this, ICS was already a couple of years 'late'. Even when the pilot roll-out started in March 2005 it was obviously generating problems. Parallel running was a complete failure because no one was going to switch over until they shut down COMPILE. There was a push to switch people in July and then finally they shut down COMPILE in October and the problems grew and grew with the shipping volumes. For traders the run up to stocking Christmas shelves starts in August and goes exponential. That's when we found ourselves in a full-blown crisis; Airports 'chock-a-block', container terminals full, container ships parked off-shore unable to unload. Worse and worse, so no, we did not have a happy Christmas! [Trader]
Postscript

Press Report: "Australian Customs Service chief executive Michael Carmody has promised an 'open and frank' exchange with industry on how to fix continuing problems with its troubled Integrated Cargo System. The former Tax Commissioner said he would share with the industry the results of an independent report on how to squeeze more benefit from the system, which Customs commissioned last month from consultancy Booz Allen Hamilton. Mr Carmody said the $A200 million system was still a long way from delivering the benefits that had been expected - either for the import-export industry or Customs..."[Riley, 2006]
On 9th June 2006 a $A300,000 report by Booz Allen Hamilton on how to fix the $A200 million Customs Integrated Cargo System was handed to government. The report examined the circumstances leading up to and following the deployment of ICS, the core system controlling the movement of imports and cargo into Australia. The introduction of ICS had “caused havoc at Australia’s wharves and airports in October last year” (Mitchell 2006), with on-going implications for the Australian government such as facing “$A9 million in compensation claims, mainly for extra storage costs incurred when containers were left sitting on the wharves unable to clear customs” (Hayes 2006). Australian importers carried the cost of security shortcomings, delays, storage charges, lost business, reduced performance and increased workload.

A year later Senator Joe Ludwig (Lab), speaking to the Customs Brokers and Forwarders Council of Australia, reflected on the aftermath of the CMR programme and the ICS roll-out in particular. By this time Customs had a new CEO, Michael Carmody.
"The Cargo Management Re-engineering project, which was part of the trade modernisation agenda will be remembered as one of the all-time greatest implementation failures of a government project in Australia’s history. Its budget blew out to ten times the original estimate, it saw constant delays and, when finally switched on, it nearly brought down Australia’s trade industry." [Senator Joe Ludwig (Lab)]
References
  • Hayes, S. "Customs pushed into chaos," in: The Australian, 2006.
  • Ludwig, J. "Speech to the Customs Brokers & Forwarders - Current Issues in Trade Security," (speeches at www.senatorludwig.org) accessed Dec 2006.
  • Mitchell, S. "Customs report on IT fiasco," in: The Australian, 2006.
  • Riley, J. "Customs 'open and frank' on ICS," in: The Australian 2006.