Happy Hollowing Customs (case)

The actors

ACS: the Australian Customs Service.
Murray Harrison: (a public servant) CIO for Australian Customs, 2002-.
Chris Ellison: (Liberal politician) the Federal Government Customs Minister, 2001-2007.
Eric Roozendaal (Labour politician) Ports Minister for the state of NSW, 2006-2007. 
Joe Ludwig: (Labour politician) Senator. Opposition Shadow Minister for Ports 2004-2007
Tradegate Australia Ltd: Established in 1989 as an industry representative body to develop and implement e-commerce services for Australia's international trade and transport industry. 
EDS: Ross Perot's international IT consultancy and outsourcing supplier. 
ICS: the Integrated Cargo System (ICS), part of the CMR project.
CMR: the Cargo Management Reengineering programme initiated by the ACS. 
COMPILE: The ACS system for managing import declarations, duty/tariff transactions, quarantine and bonding.
EXIT: The ACS system for managing export declarations and clearances. 
Verisign Inc.: An internet security digital signature services company.
IBM: International Business Machines.
COBOL: One of the first, wide spread, and most successful programming languages.
C: A programming language
CICS: Customer Information Control System, mainframe transaction server.
IMS: The Information Management System (database with transaction processing capability).
P&T: Post and Telecommunications Agency (aka Plain old telephone service (POTS)).

(figure 1. Key actors and technologies prior to ICS go-live)

The ACS Expands Outsourcing Initiatives

Going back to the 70s, 80s and 90s the Australian Customs Systems Division was at the front of transactional systems development. They had decades long experience developing resilient, scalable, available tech systems over a national network employing electronic data interchange. The first systems were dial up, then dedicated leased lines from the P&T. Systems architectures ran on mainframe, using COBOL and CICS and IMS, some C development and other software.

"The switch to Java and internet in the late 90s put a huge strain on our capability, we just weren't staffed up in those areas. Stupid hiring freezes across the civil service. There were restrictions on hiring new programmers into the civil service, the Department of Finance had no problem giving permission to pay consultancies 100K or 150K a year for bodies, but point blank refused to sanction recruitment of graduate programmers at 30K (even if we could get them). The net result of the ban on being able to hire in people was to choke off our civil service tech capability in Java and Internet. [Customs]"

In 1997 EDS, the global IT services company, secured a number of lucrative multi-annual service and development outsourcing contracts with the Australian Customs Service (ACS). EDS (Electronic Data Systems, founded by Ross Perot) specialises in business process and IT outsourcing services. Prior to the outsourcing contract with EDS the ACS managed, developed, deployed and delivered its core back-end systems almost wholly in-house. After the outsourcing deal, responsibilities for the ACS's core IT systems transferred over to EDS with gradually less direct involvement by ACS internal staff.
The old system worked well and was integral to the smooth operation of the national import and export system. It delivered high availability, responsiveness, and functionality for interactive or batch operation. Around this time Tradegate executives commented that the shift of system maintenance and development responsibilities (in additional to operational services) from ACS personnel to EDS resulted in a virtual shutdown in cooperation between the technicians on the 'inside' (of ACS-EDS) and those on the 'outside' (Tradegate).

The new supplier had no interest in involving Tradegate, you might say they went out of their way to ignore them. [Tradegate member]

We're all the victims of the tendering process. It was driven by the Department of Finance, the Tax Office. It was all about finance, cost savings, and Tradgate got slated. [Customs]

One consequence of the ACS-EDS deal was that the ACS lost more and more of its own specialist Customs IT expertise as employees transferred into EDS and then gradually moved on into other areas in EDS globally. Staff transfers from ACS to EDS also altered the apparent cost structure for getting regular work done, for example the ACS paid contractor rates for the services of EDS consultants, many of whom had previously been public servants for the ACS.

Customs IT people had opportunities to join EDS, a lot of them did, a civil servant's pay isn't great, but some stayed, for different reasons. [Customs IT]

It creates tensions, side by side, people you used to work with but now they're getting paid more, a lot shipped out, our knowledge has a value elsewhere in the organisation [EDS]. [ex-Customs IT now EDS consultant]

CMR and ICS
In 1998 the ACS and EDS announced a massive $A25 million programme of IT systems renewal called CMR (Cargo Management Reengineering). The CMR was going to be the Australian Customs Service's next generation IT system; a single system for managing, recording and controlling the movement of all goods movements into and out of the country. The Minister had made commitments in Parliament and the Customs IT innovation plans had been talked about nationally and internationally for several years. The proposed Integrated Cargo System (ICS) was to be the heart the CMR project. ICS would replace the ACS's venerable EDI based COMPILE system.

They [EDS] send in the 'A-Team' to get the CMR and ICS deals. They won. The 'A-Team' gets the deal the 'D-Team' gets to deliver it... people moved, Customs was hollowed out. [Customs IT staff]

It was a huge strategic shift, going from a 'community portal' sitting on top of Customs' EDI interfaces on mainframes to Customs outsourcing development and delivery to EDS. [Industry observer]

After over five years of development, delays and an increasingly fraught relationship between the ACS and EDS, ICS finally went 'go-live' on October 12 2005. Its users were compelled to switch over to the new system from an electronic trade management system that had served them for over 15 years.
The ICS go-live, October 2005
At the time of the ICS roll-out on the 12th of October 2005, Port Botany in Sydney was handling $A100 Million in trade per day. Up to the point of ICS 'go-live' Tradegate had provided essential front-end services on top of the ACS back-end. On ICS go-live day the old systems were simply turned off!

...they brought us in for training three months before go-live. [Trader]

Only a couple of the biggest importers were involved in user acceptance, I'm not even sure they were involved at the early stages for user requirements. [Trader]

ICS has the requirements of other government departments: Border Protection, Statistics, Taxation. But no business logic worth talking about. [Trader]

The operation of the new system was marred by widespread problems. Security access flaws allowed users to access and view other Traders’ import documentation, revealing confidential information like price, supplier details, quantities and delivery dates. The system was also perceived to be unreliable, badly designed, slow and buggy.

Errors come back ages after you make a clearance request. Why didn't the computer system pick up on it first? [Trader]

The error handling has problems, delays, I'm still waiting for responses to messages [in the system]. I might have 20 messages queued somewhere, meanwhile our containers are sitting on the tarmac at Kingsford-Smith (Sydney Airport). [Trader]

A couple of weeks after ICS go-live the key seaports, Port Botany and Melbourne Port, had ground to a halt as containers piled up. Trade effectively stopped due to ICS processing delays or other problems with the new systems. Australia's trade clearance system ground to a halt from October 12 and for the next six weeks containers and cargo accumulated at ports and bonded warehouses. The situation reached crisis point when Port Botany and Port of Melbourne ran out of room to store uncleared containers. The clearance backlog came about partially because ICS throughput was just 30% of normal handling rates, this led to excessive cargo dwell times at airports and sea ports.

I need Customs clearance to move them out of the port. [Trader]

That's the power of Customs, we are just pleased if we get their attention. [Trader]

The spectre of vessels (sea and air traffic) unable to disembark cargo was now a fact, Australia’s connections with the international trading system had seized up.

The system has fallen apart. Traders are getting seven days turnaround instead of seven minutes. In COMPILE declarations had 98% success on first time entry. In ICS, at roll-out we started with 60%, that's 40% fail! It improved slowly, 60%, 70%, 80%. [Tradegate employee]

Let me put it like this, ICS was already a couple of years 'late'. Even when the pilot roll-out started in March 2005 it was obviously generating problems. Parallel running was a complete failure because no one was going to switch over until they shut down COMPILE. There was a push to switch people in July and then finally they shut down COMPILE in October and the problems grew and grew with the shipping volumes. For traders the run up to stocking Christmas shelves starts in August and goes exponential. That's when we found ourselves in a full-blown crisis; Airports 'chock-a-block', container terminals full, container ships parked off-shore unable to unload. Worse and worse, so no, we did not have a happy Christmas! [Trader]

Postscript

A year later Senator Joe Ludwig (Lab), speaking to the Customs Brokers and Forwarders Council of Australia, reflected on the aftermath of the CMR programme and the ICS roll-out in particular. By this time Customs had a new CEO, Michael Carmody.

"The Cargo Management Re-engineering project, which was part of the trade modernisation agenda will be remembered as one of the all-time greatest implementation failures of a government project in Australia’s history. Its budget blew out to ten times the original estimate, it saw constant delays and, when finally switched on, it nearly brought down Australia’s trade industry." [Senator Joe Ludwig (Lab)]

Edited narrative interview with Australia Customs IT programme manager.

"So the first wave of our outsourcing experience was a massive learning process. We chose a strategic partner, and then switched, and then switched again. Partially driven by Government policy, partially driven by necessity. Each partner was basically giving us a Rolls Royce approach. But as it turned out they were also using us as a kind of training centre for their new hires. We ended up being the first post for their graduates and internships. They'd be trained for a year or more on our systems, processes, tech skills, management skills then ship out to consultancy gigs elsewhere. [Customs]"

"Since then we've taken a different approach. We've hived off little pieces. The way we wrote our tendering would allow an incumbent to continue (rather than switching at each renew), but we also started to divide out the work. Now we split it across roles rather than systems. They (suppliers/vendors) supply so many people as system architects, business analysts, programmers. But we keep control of the systems, they're our systems, we own them. [Customs]"

"We have gradually ended up with two strategic partners and loads and loads of smaller external vendors. That has worked really well. While X are our dominant partner we have loads of small specialist partners, local, Australian. It plays out well politically but it has really paid out with a regional ecosystem of expertise. [Customs]"

"We now build, deliver, operate (and own) everything. The internal team is just 30 people, pure technical excellence, constantly training up on new tech, deep knowledge of our architecture. But it scales because we're a programme management operation. It is our own Customs system framework, we own, develop and manage it. My team work side-by-side with the vendor people. There might be ten or twenty times as many vendor people around but we're in charge. It took us 5 years to reach this point, building internal capability and having a capability for the future. [Customs]"

Keeping PACE with the local (case)

The objective stated in PACE's original business plan was to give client companies access to best-in-area programming and testing capability through offshore outsourcing to India. PACE's strategy leverages their own specialised knowledge and capabilities to manage globally distributed engineering teams. They deliver talented people, lower costs, and faster turnaround for software projects.

PACE applies a 2-stage model for client onshore/offshore transitions (figure 1); the first year of the contract is conducted onshore working closely with the client on-site after which the project transitions swiftly to an off-shore operation. The value proposition to clients is threefold and simple: project cycle time, management overhead and employee cost reductions will occur because the Indian team has access to more people, at lower cost, and working across European and American time zones. An added benefit is that project work continues around the clock if necessary.

(figure 1. PACE lifecycle for onshore/offshore transitions)

In its first 18 months of operation PACE has secured 2 European mid-sized software product development firms as clients. A further three prospective clients are nearing contract signing and they have a qualified pipeline of 12 new potential clients. Both of the active contracts are for an initial three-year duration with an option to extend. The first client contracted 60 person-years (over three years) and the second 45 (over three years). The first client company develops products that enable customers to create distributed cross-platform information systems. The second client company's products service secure point-of-sale (POS) transactions and payment validation. Both clients state that the partnership with PACE will allow them to scale; they need more engineers to grow their own business, to handle the needs of existing customers, and to create new products addressing new market opportunities. PACE's engineers are intended to service the companies' legacy products, basically they take on support, maintenance, and patch production.

PACE's business plan specifies an IRR (internal rate of return) of between 30% and 50% for its investors, a simple payback period for the original investment of just 2 years, plus annual growth potential of up to 50%. The up-front (sunk) setup cost was 100k. The firm's fixed costs included 2 full-time sales and marketing roles (both based in Europe), in addition there are 3 employees based in India tasked with sourcing and vetting candidate CVs, arranging interviews with client companies (online and via conference calls), visa processing, and other home-base activities aimed at supporting the eventual offshore transitions for projects moving to India. These on-going fixed costs (including accommodation, service charges, travel etc.) amount to around 160k per annum.

Both current clients are nearing the end of the first contract year with teams of 20 and 15 PACE engineers respectively working in onshore mode; however both companies are now expressing a desire to delay (if not a reluctance to proceed to) the next stage - transitioning to offshore operations! To complicate the situation the Indian engineers were hired with a clear expectation of returning to India; relocation to Europe is seen as a temporary assignment. Ambiguity surrounding return dates is becoming a source of dissatisfaction. Many had signed on because they wanted, eventually, to work near home, family and friends. European Winters are cold and wet, the cost of living is high and food, lifestyle, and cultural difference are a source of personal and family stress.

Delays in transitioning clients to full offshore outsourcing mode could have a major impact on PACE's business model. Simply put, the company's expected net revenue (surplus) for each contracted engineer is 5k per year while the engineer is working onshore at the client site, whereas it is 25k per engineer per year after the project transitions to full offshore mode. In onshore mode the company is barely able to meet its operating costs let alone accumulate surpluses for profit sharing (a promised employee incentive) or provide a return to investors. The company needs to transition clients to full offshore mode if it is going to do more than survive.

---

We have a problem Heuston (case)

Tom runs a product concept and design company that handles high-level web and marketing activities while outsourcing development. Although his business is located in the digital campus beside Heuston Station he claims there are just too few (affordable) software engineers in the Irish market for a small firm like his to build up its own development capability, especially as much of the work can be sourced online. He contracts software implementation and delivery to a small number of trusted outside firms that he has worked with over the years. Tom handles client facing interaction, gathering the requirements, and sourcing delivery from his contacts. He thinks of the software code merely one facet of the business, as something that just 'gets the job done';

"yes code is intellectual property but you can't be precious about code. For me it's the relationship with my client, building up a joint understanding of how we want a 'thing' to work. To be honest I couldn't care less about the code as something I own".

The CMS project was a nice earner in 2005 but now five years later, it is becoming a bit of a headache. The original developer had moved to America but passed the source code to Tom who in turn contracted a firm in India to maintain and develop it. The Indian firm was founded by Suhas and a number of partners who were IT professionals and had worked together previously in Europe. Suhas had taken on the CMS contract almost as a favor to Tom and the original developer, a personal friend. As it happened neither Suhas nor the developers on his team had ever met Tom in person or dealt directly with Tom's clients (CMS's customers and users). Tom managed all contact with India, usually through Suhas but occasionally contacting the developers assigned to the CMS by email, Skype or phone.

Pune, called the Oxford of the East by India's first Prime Minister Jawaharlal Nehru, is home to many high tech, consultancy and software companies centred around a vibrant community of universities, research institutes and colleges. Pune competes for international software outsourcing business with other locations in India and across the World. However Suhas's view has always been that Indian engineers should innovate and add value, rather than just take part in an international outsourcing race to the bottom in terms of cost or manpower. He founded the firm on principles of integrity and fairness...

"knowledge has power, but it takes time to acquire and it is only useful if it is shared and used!" 

"I always feel that we can not be threat to our clients because the end customer always prefers to have a local company to deal with which can outsource the work to outside world."

The CMS project has a history, characterised by periods of intense involvement (development and delivery work) followed by gaps of months at a time with little or no contact. The engineers who had worked on it over the years were often overwhelmed, picking up where the project had left off and working on fixes or new features with few opportunities for feedback with the end customer. A release could be delivered after which it might take Tom a week or more to report back on the quality or its acceptance. This sort of stop-start, delayed back and forwards was not typical for Suhas's other contracts. Suhas felt that the CMS was a perfectly good application that already satisfied the requirements, however Tom's point of view was that the customer was always right, if the customer was unhappy then it was a bug and it had to be fixed. For Tom, just understanding a customer's problem took time and effort, explaining it to Suhas or one of his engineers complicated things further. In spite of this Tom was concerned to keep the customer at 'arms length' from India, after all, if they dealt directly with each other what was the point of having Tom in the relationship? Suhas felt there could never be an issue of 'going behind' Tom, he reasoned that Tom owned the CMS, it was Tom's property. What Suhas wanted was for his engineers to communicate directly with the end-user so they could clarify the requirements, satisfy them completely, identify and fix bugs, get on with the job and get paid.

---

CMS: Content Management System(s)

Content management systems are applications for managing and structuring information and data accessed and delivered over the Internet. The term CMS is often synonymous with 'portal' or web site however several crucial features distinguish a CMS from other internet services including: publication workflow, fine grained user roles and privileges, extensive data/object types (event, wiki, news, chat, blog, slideshow, video, audio, forms, mail, surveys, RSS), TTW (Through The Web) administration and use, extendible via Javascript and other programming methods.
A general overview of different CMSs is available at the cmsmatrix.org. cmsmatrix.org enables feature comparisons between the many CMS product/applications available.

ICS Go-Live 2005 (case)

Disclaimer: The case as presented here is a partial account and has been edited for teaching purposes. This material has been simplified for brevity and narrative coherence. The case has been designed as a teaching aide to encourage independent investigation, learning, discussion and debate rather than as criticism, illustration of management practice, endorsement of any particular approach or product. Interviews and notes were gathered from private conversations, discussions, and data from documentation in the public domain covering the period prior to 2005 and up to current.

Introduction
The Australian Customs Service (ACS) has been at the cutting edge of electronic systems development for over 30 years, controlling trade into and out of Australia’s borders. The ACS's earliest systems were delivered using a variety of hardware, networks and applications: mainframes, direct connection (EDI), community ISP facilitated EDI, and internet based portals. The ACS's applications and service delivery mechanisms were precursors to current international initiatives involving the WCO (World Customs Organisation), to develop standards based Single Window Access (SWA) systems for Customs services. Single Window Access offers the possibility of 'interoperable integrated international Customs systems' and thus the potential for global control, risk management and regulation of international trade.

Background
The ACS has long been recognised as a centre of excellence for Customs systems and innovation. Historically technology providers have serviced government's needs for computing hardware, networks and system software. The need to match application capability with international trade systems (domain knowledge) against a backdrop of constant regulatory evolution meant that Customs IT innovation has been driven by Customs agencies rather than by the IT industry.

In 2001, the Australian Government updated the Customs and other related Acts in an effort to allow Customs IT to change more freely. The 'International Trade Modernisation' package updated the legislative framework for Customs to allow for a more flexible approach. It did not prescribe individual systems but rather introduced a kind of indirect approach by requiring "that Customs gazette its IT requirements." In effect earlier Customs Acts, the law under which Customs operates, had specified the IT systems to be used through law, therefore changes to IT systems had to be reflected in changes in the Act. Once the modernisation package was adopted, changes to the IT could then be notified by gazette in Parliament rather than being enacted through legislation. The IT still had the authority of law, but the IT did not need to be specified by the law.

Customs sees its role in the national and international trading system not as a 'tax' on industry but as a necessary control; regulating and protecting society while at the same time enabling trade and economic growth. Consequently the World's earliest electronic, real-time, high transaction volume, standards based, trade management networks, were Customs import/export systems. In many ways the demands of Customs agencies over succeeding decades has driven the IT industry agenda, first by specifying and establishing EDI standards, and second by providing the necessary business case for suppliers and service providers. The business case for Customs IT innovation has been based on controlling while facilitating traders' imports and exports at the lowest possible cost.

ACS e-Commerce Systems
From 1989 to 1999 the Australian Customs Service (ACS) designed, delivered, and operated 'back-end' electronic systems to process import and export declarations, duty payments and other essential trade services. ACS e-Commerce systems were back-end mainframes running purpose built software like the COMPILE system for imports and the EXIT system for exports. Licensed traders could access them directly or via a third party service operator, Tradegate. Tradegate was a national trader community organisation that cooperated with the ACS to provide end-user interfaces to the ACS back-end; its mission was to develop and deliver the e-commerce 'face' of the ACS's import and export clearance systems. Tradegate was established in 1989 as a not-for-profit organisation comprising the importers and exporters and the wider trading community as members. Tradegate had also been responsible for provisioning a secure national electronic network prior to the widespread availability of the Internet by contracting and managing specialist telecommunications network providers, first with Paxus then AT&T EasyLink. Traders around Australia could use Tradegate to access COMPILE and EXIT from anywhere at any time.

COMPILE was very forgiving, I enter code '32' in the box and it always goes through.[Trader]

Tradegate was funded by member fees, a per transaction charge for portal use and fixed costs for network access and the ACS held a statutory directorship on the board. Tradegate's executive, its members and its board considered themselves to be partners with the ACS, providing a stable, secure and effective service and a key enabler for Australia's international economic success.

In the old days Tradegate and Customs had indirect avenues. The Tradegate guy was probably playing golf with this guy, he could walk upstairs and have a chat. They got things done. [Trader comment]

The ACS Expands Outsourcing Initiatives
In 1997 EDS, the global IT services company, secured a number of lucrative multi-annual service and development outsourcing contracts with the Australian Customs Service (ACS). EDS (Electronic Data Systems, founded by Ross Perot) specialises in business process and IT outsourcing services. Prior to the outsourcing contract with EDS the ACS managed, developed, deployed and delivered its core back-end systems almost wholly in-house. After the outsourcing deal, responsibilities for the ACS's core IT systems transferred over to EDS with gradually less direct involvement by ACS internal staff. Around this time Tradegate executives commented that the shift of system maintenance and development responsibilities (in additional to operational services) from ACS personnel to EDS resulted in a virtual shutdown in cooperation between the technicians on the 'inside' (of ACS-EDS) and those on the 'outside' (Tradegate).

The new supplier had no interest in involving Tradegate, you might say they went out of their way to ignore them. [Tradegate member]

Referring to this period in a submission to the Australian Government in 2001 Tradegate claimed that

"Contact with the key technology and service development staff [has] virtually ceased. The ability to progress externally beneficial service improvements through regular contact and 'brain storming' sessions came to an end with all contact placed on a formal basis presumably due to the dictates of the ACS-EDS contractual arrangements."

It was apparent that the ACS-EDS deal altered to the detriment the relationship between Tradegate and ACS, that the previously close contacts were disrupted and not re-established when EDS entered the equation.

We're all the victims of the tendering process. It was driven by the Department of Finance, the Tax Office. It was all about finance, cost savings, and Tradgate got slated. [Customs]

An unexpected consequence of the ACS-EDS deal was that the ACS lost more and more of its own specialist Customs IT expertise as employees transferred into EDS and then gradually moved on into other areas in EDS globally. Staff transfers from ACS to EDS also altered the apparent cost structure for getting regular work done, for example the ACS paid contractor rates for the services of EDS consultants, many of whom had previously been public servants for the ACS.

Customs IT people had opportunities to join EDS, a lot of them did, a civil servant's pay isn't great, but some stayed, for different reasons. [Customs IT]

It creates tensions, side by side, people you used to work with but now they're getting paid more, a lot shipped out, our knowledge has a value elsewhere in the organisation [EDS]. [ex-Customs IT now EDS consultant]

CMR and ICS
In 1998 the ACS and EDS announced a massive $A25 million programme of IT systems renewal called CMR (Cargo Management Reengineering). The CMR was planned to be Australia's next generation IT system, responsible for managing, recording and controlling the movement of all goods movements into and out of the country. The Minister had made commitments in Parliament and the Customs IT innovation plans had been talked about nationally and internationally for several years. The proposed Integrated Cargo System (ICS) was to be the heart the CMR project. ICS would replace the ACS's venerable EDI based COMPILE system.

They [EDS] send in the 'A-Team' to get the CMR and ICS deals. They won. The 'A-Team' gets the deal the 'D-Team' gets to deliver it... people moved, Customs was hollowed out. [Customs IT staff]

It was a huge strategic shift, going from a 'community portal' sitting on top of Customs' EDI interfaces on mainframes to Customs outsourcing development and delivery to EDS. [Industry observer]

After over five years of development, delays and an increasingly fraught relationship between the ACS and EDS, ICS finally went 'go-live' on October 12 2005. Its users were compelled to switch over to the new system from an electronic trade management system that had served them for over 15 years.

(figure 1. Key actors and technologies prior to ICS go-live)

The ICS go-live, October 2005
At the time of the ICS roll-out Port Botany in Sydney was handling $A100 Million in trade per day. Murray Harrison (a public servant) was the CIO for Australian Customs, Chris Ellison (Lib) was the Federal Government Customs Minister and Eric Roozendaal (Lab) was Ports Minister for the state of NSW. Tradegate Australia Ltd was established in 1989 to develop and implement e-commerce services for Australia's international trade and transport industry. EDS is an international IT consultancy and outsourcing supplier. ICS is the Integrated Cargo System (ICS), a key element of the Cargo Management Reengineering (CMR) programme initiated by the ACS. The ACS is the Australian Customs Service.
Up to the point of ICS go-live Tradegate had provided essential front-end services on top of the ACS back-end. The existing system worked and was integral to the smooth operation of the national import and export system. It delivered high availability, responsiveness, and functionality for interactive or batch operation. On the ICS go-live day the old systems were simply turned off!

The 'refreshed' systems roll-out, it was called ICS. They presented it like an update of COMPILE, like it was a version upgrade but it should have been called something else. [Trader]

They were supposed to have launched in March, last year or the year before![Trader]

...they brought us in for training three months before go-live. [Trader]

Only a couple of the biggest importers were involved in user acceptance, I'm not even sure they were involved at the early stages for user requirements. [Trader]

ICS has the requirements of other government departments: Border Protection, Statistics, Taxation. But no business logic worth talking about. [Trader]

The operation of the new system was marred by widespread problems. Security access flaws allowed users to access and view other Traders’ import documentation, revealing confidential information like price, supplier details, quantities and delivery dates. The system was also perceived to be unreliable, badly designed, slow and buggy.

Errors come back ages after you make a clearance request. Why didn't the computer system pick up on it first? [Trader]

The error handling has problems, delays, I'm still waiting for responses to messages [in the system]. I might have 20 messages queued somewhere, meanwhile our containers are sitting on the tarmac at Kingsford-Smith (Sydney Airport). [Trader]

A couple of weeks after ICS go-live the key seaports, Port Botany and Melbourne Port, had ground to a halt as containers piled up. Trade effectively stopped due to ICS processing delays or other problems with the new systems. Australia's trade clearance system ground to a halt from October 12 and for the next six weeks containers and cargo accumulated at ports and bonded warehouses. The situation reached crisis point when Port Botany and Port of Melbourne ran out of room to store uncleared containers. The clearance backlog came about partially because ICS throughput was just 30% of normal handling rates, this lead to excessive cargo dwell times at airports and sea ports.

I need Customs clearance to move them out of the port. [Trader]

That's the power of Customs, we are just pleased if we get their attention. [Trader]

The spectre of vessels (sea and air traffic) unable to disembark cargo was now a fact, Australia’s connections with the international trading system had seized up.

The system has fallen apart. Traders are getting seven days turnaround instead of seven minutes. In COMPILE declarations had 98% success on first time entry. In ICS, at roll-out we started with 60%, that's 40% fail! It improved slowly, 60%, 70%, 80%. [Tradegate employee]

The chairman of the Customs Industry committee said 'it was chaos', 'blood on the ground'. [Trader]

Let me put it like this, ICS was already a couple of years 'late'. Even when the pilot roll-out started in March 2005 it was obviously generating problems. Parallel running was a complete failure because no one was going to switch over until they shut down COMPILE. There was a push to switch people in July and then finally they shut down COMPILE in October and the problems grew and grew with the shipping volumes. For traders the run up to stocking Christmas shelves starts in August and goes exponential. That's when we found ourselves in a full-blown crisis; Airports 'chock-a-block', container terminals full, container ships parked off-shore unable to unload. Worse and worse, so no, we did not have a happy Christmas! [Trader]

Postscript

Press Report: "Australian Customs Service chief executive Michael Carmody has promised an 'open and frank' exchange with industry on how to fix continuing problems with its troubled Integrated Cargo System. The former Tax Commissioner said he would share with the industry the results of an independent report on how to squeeze more benefit from the system, which Customs commissioned last month from consultancy Booz Allen Hamilton. Mr Carmody said the $A200 million system was still a long way from delivering the benefits that had been expected - either for the import-export industry or Customs..."[Riley, 2006]

On 9th June 2006 a $A300,000 report by Booz Allen Hamilton on how to fix the $A200 million Customs Integrated Cargo System was handed to government. The report examined the circumstances leading up to and following the deployment of ICS, the core system controlling the movement of imports and cargo into Australia. The introduction of ICS had “caused havoc at Australia’s wharves and airports in October last year” (Mitchell 2006), with on-going implications for the Australian government such as facing “$A9 million in compensation claims, mainly for extra storage costs incurred when containers were left sitting on the wharves unable to clear customs” (Hayes 2006). Australian importers carried the cost of security shortcomings, delays, storage charges, lost business, reduced performance and increased workload.

A year later Senator Joe Ludwig (Lab), speaking to the Customs Brokers and Forwarders Council of Australia, reflected on the aftermath of the CMR programme and the ICS roll-out in particular. By this time Customs had a new CEO, Michael Carmody.

"The Cargo Management Re-engineering project, which was part of the trade modernisation agenda will be remembered as one of the all-time greatest implementation failures of a government project in Australia’s history. Its budget blew out to ten times the original estimate, it saw constant delays and, when finally switched on, it nearly brought down Australia’s trade industry." [Senator Joe Ludwig (Lab)]

References

• Hayes, S. "Customs pushed into chaos," in: The Australian, 2006.
• Ludwig, J. "Speech to the Customs Brokers & Forwarders - Current Issues in Trade Security," (speeches at www.senatorludwig.org) accessed Dec 2006.
• Mitchell, S. "Customs report on IT fiasco," in: The Australian, 2006.
• Riley, J. "Customs 'open and frank' on ICS," in: The Australian 2006.

You've got email MAT (case)

You've got email+malware+viruses+zombies+spam+fishing+adware.
Móin Alúine Teichneolaíocht (MAT) is the brainchild of two engineers who had worked in Digital’s Galway campus in the 1990s. While MAT's customers are risk adverse, conservative multinational banks and financial services operators, they value MAT's rapid innovation and reputation for adapting quickly to technological change. ‘Prime Broker', MAT's first product, broke new ground when it was launched, based on open standards and interoperability; Web 2.0 was designed into the heart of its architecture.

Unfortunately over the last 6 months MAT had suffered a run of bad luck:

1. A small fire in the server room had destroyed hardware in the room including the router, firewall and wiring to the ISP.
2. MAT's ISP experienced 2 outages or critical network failures that affected MAT's business communications including support SLA turnaround times on bugs.
3. On no less than 6 occasions malicious bots had infected the mail server, outbound traffic became overwhelmingly bot generated and internet connectivity was reduced to a crawl. Worse, the bot traffic had red flagged MAT's IP and MX registrations, which were then 'blacklisted' on authoritative public watch listed servers. Consequently their IP and MX records were 'quarantined' by the gods of the Internet.
4. A knock-on impact of the mail server problem was that malicious spyware and bots had infected some internal workstations. MAT suspected that certain file systems had been compromised (accessed and copied).

In January 2012 the engineering and IT teams hosted an off-site day where they reviewed the company’s performance from the perspective of skills, tools, technology and trends. Among the topics discussed was the IT infrastructure (internal and external facing) among which Email and Messaging services stood out.

Email is a basic necessity for business and it has been the communication lifeblood of MAT from day 1. Newer instant communication and messaging technology like SameTime and ‘Skype’ is also extensively used by the engineers (Product Development, Test, Support, Professional Services and IT). Everyone manages three or more personal email accounts in addition to their corporate (firstname.secondname@mat.com) identities.

One proposal under consideration is to shift from the internal email service to a fully external email service from Google or another 'cloud' provider. The CEO has also asked if cloud services can replace internal systems, systems that might be better provided by an external operator.

Also discussed at the off-site were the results of a recent external ISO9001:2008 quality audit that identified shortcomings in the company's business continuity and disaster recovery plan; specifically, how core infrastructural services (email in particular) are maintained and proofed against disaster. Business continuity and disaster recovery plans are necessary to ensure that we can restore "adequate alternative arrangements for systems which need to be operated in the event of a breakdown". Critical Infrastructure is defined as the tools & structures necessary to operate essential services or carry out core activities, for example; internet access, mail server, source code control system, and the telephone system.

---

So how much could you save with Google Apps for Business?
Google's corporate email service is a key feature in its recent paid service, Google Workspace (previously Google Apps for Business). See https://workspace.google.com/pricing.html.

The Google Workspace pricing model

MX Toolbox - Email Blacklist Checks and Diagnostics (www.mxtoolbox.com)

The MX Lookup Tool can test and list "MX records for a domain in priority order. The MX lookup is done directly against the domain's authoritative name server, so changes to MX Records should show up instantly. You can click Diagnostics, which will connect to the mail server, verify reverse DNS records, perform a simple Open Relay check and measure response time performance. You may also check each MX record (IP Address) against 147 DNS based blacklists . (Commonly called RBLs, DNSBLs)"

Celtic Tiger, Chinese Dragon (case)

The company organises its 'enterprise software' development and production teams according to a 'lean development' framework. The management team's emphasis is on the top line; new sales, big deals, big customers, (even better to get big-new-customers). They had achieved success in the past by being close to the customer, understanding the customer, achieving 'fit' with the customer. The executive team and the board of directors saw bottom-line cost control as important, however close involvement with the company's customers, bringing features inside the core product had enabled continuous renewal and evolution of the product lines.
A number of consultancy projects in China were now exceeding the size of those in Europe and North America. Chinese semi-states and private corporations were investing heavily in cutting edge, large scale, IT infrastructure. Of all the developing nations China was the biggest and growing the fastest. In the end the decision to setup a centre of excellence in China was straightforward.

---

Excerpts from interviews with a Senior Software Engineer (Chinese) and the Director of Product Management (China office).

(Director of Product Management)

"...one of the reasons is [the China office] adds value to the company. If we think about the four of five possible strategies for a growing tech company, well, having a foothold in your market, plus smart people, smart customers, high profile, high value projects. It ticks all the boxes."  

"Establishing a centre of excellence in China is a dream project in many ways. When you acquire a company you have to work with a pre-established team. But setting up a green-field R&D centre, like now, it's a clean slate, new office, new hires, using our 'foreign' methodology."

"Our in-house development methods and practices are inspired by the Agile and Lean movements. Projects, operations and development use our own take on the Scrum or Kanban style of process. I have decided to use our in-house methods from the start. The local engineers might look on it as a 'foreign' approach but I don't think we'll see resistance. They are open to learning about it." 

"My view is that Agile is a strict discipline; it's more disciplined than other methods if you do it in a highly structured way, get rid of the 'ad hoc' style that seems to be associated with it. Start your stand-ups at 10am (not 5 past 10), finish strictly at 10:15am. The stand-up conversation should be structured; specific challenges approaching, specific problems you are facing, you've either run or written a test case. Programmers will like that. There's 'do or don't', 'did or didn't', no ambiguity and you need these type of conversations. Agile brings this to the front." 

"But yes, I am worried about the difference in culture, particularly problems with 'face'. How do you get engineer's admitting to difficulty or failing to achieve something?" 

"When you set up an office in another country, there's a fear among your people at the home base! You immediately create the potential for a social divide that applies when you move to an offshore site." 

"Imagine, your company sets up an office in China where the engineering costs are lower. Do you want a situation where you say 'designed in California, made in China'? What does that say about a social divide in the company? That's the choice you have to make." 

(Senior Engineer)

"In China software engineering career has quite a lot of respect." 

"In general, software engineers work long hours, often seven days a week, starting early, leaving late." 

"International companies might have more formal policies, leave work at 7pm, come back in the morning. But they might not be as flexible as smaller companies that expect working around a project. If an emergency comes in a small company they might have to work all night."

"Yes, there is a lot of employee turnover. Conditions can be bad, the overtime, bad management, working away from home." 

"Now Agile is well known in China. In the real implementation there are some cultural differences that might make it feel different to an implementation in the US. But a company, should not admit some failure to its customers." 

"Let me put it this way; In western countries, people like to treat work like playing games. But here in China, a manager here is something like the parent, not childlike." 

"I don't accept the childish approach. Engineers think 'we are not kids, we are adults'. So Agile can inspire engineers in the West because it is playful, but in China, I don't know?"

"If your company commits to something your business lives or dies by it. But that's business in China, completely different to the West. Completely different."

---

Related Data

• Agile 101 - Methodologies, Learn more about Scrum, XP, DSDM, FDD and Lean software development (www.infoq.com).
• Understanding Chinese business culture: potential starting points for developing an understanding of guanxi (relational) and mianzi (face/status/rank) on wikipedia.org or google search.
• Issues around manufacturing in China (sources: Irish Times Thu, Dec 29, 2016, 17:17; Seattle Times December 31, 2016; original article New York Times)

Relocating knowledge (case)

Jo had just returned from an assignment at Nordic's (a pseudonym) headquarters in Scandinavia nicknamed the Mothership. It was a planned six month rotation that had extended over a year, intended as a secondment for professional development, part of her career path to senior engineer that couldn't be achieved at Nodic's base in Dublin. While there the team she had been attached to had designed a software block for smartphone services, a crucial part of the company's technology roadmap and when one of the engineers took paternity leave her assignment was extended to cover his absence. This software block was an element in the next generation platform delivering advanced data services over mobile networks which were expected to grow exponentially(*1). After the release Jo had stayed on to help with the first update and was then offered an indefinite post, but 13 months away from home, friends, and family was enough; Mothership was nice but it wasn't home.

"So Jo, tell me what it was like in the Mothership?" said Ian.

"Great, yeah. You know, like the 'one company' slogan, it was great." said Jo.

"Right, yes, 'one company.' Look, I want you to be frank, don't hold back, I need to understand what it was really like, off the record." Ian replied.

"My impressions?"

"Your impression of the differences between the two sites, between the groups in Dublin and the Mothership, how things really work."

"Ok, since you've asked" Jo said. "So you're familiar with the organisational structure here in Dublin..."

Jo started to draw a sketch of Nordic’s organisational hierarchies in Dublin and at Mothership in Scandinavia (Figure 1 below).

Ian started: "In Dublin the feature set (FS) teams report into one of the the SysArch committees who report into the Release Div (Unified Systems Release Division) and Subsystem Group."

It's a bit flatter at HQ" Jo continued. "SubSys (Subystem) teams report into the local Subsystem Group, the Release Div there seems to work more like a service group. The Product Architecture Board sits at the top of it all of course, same as here."

Figure 1 Organisational arrangements at Mothership and Dublin

Once she had started talking Jo found she had a lot to say about the differences, between working in Dublin and at Mothership.

(design)

Something I didn't expect while I was there; the SubSys teams, although they're like our FS teams, they manage whole code blocks rather than just down to feature level. Mothership's SubSys teams manage the feature deltas so they also manage old versions within the team. My experience of the approach in Dublin couldn't have been more different. In Dublin SysArch keep the feature design deltas to themselves, SysArch hand down design documents with feature work specified from 'soup to nuts', engineers have nothing to do, SysArch have done it all - "here's how you prep, here's how you deliver, here's how you pass the exam!" Let me put it this way... On my team at the Mothership, the design document might be a 'use case' diagram and a flow chart, a picture and two sentences. I had to talk to the others to get anything done, there was so little detail to the design documents it was scary! Contrast this with Dublin, a design document for feature deltas is massive, 300 pages. One of Dublin's SysArch told me "I'm proud of our design docs, I really am," but 300 pages for one subsystem? You can't read it!

(support and bug tracking)

In Dublin some FS teams can avoid getting involved in supporting old versions at all, Support chase down the engineer who touched the code, who then hands over to the build group and then Test; the process is managed by Support using the issue tracking system. Mothership use the same issue tracking system but SubSys engineers rotate on bug fixing and the assigned engineer follows a problem through to its resolution. In Mothership the bug 'stays on you,' but of course the SubSys team is there to back you up because it's ultimately the team's responsibility. The way it goes in Dublin - the bug bounces from one person to the next, from one team to the next - Dublin's strength is the strength of the system process.

(test and support engineers)

What I thought worked well at Mothership was having a test engineer and a support engineer working on rotation in the SubSys team. In Dublin the support and test groups maintain an almost clinical separation between themselves and the engineering teams; it leads to a bit of finger-pointing at times. In contrast at Mothership the rotation gives the engineering team a 'tame' tester and support person, there's no delay in getting feedback. I also heard from the test and support guys that they learnt more this way, about new features, or if there were problems later on they'd have a direct line into the engineering group.

(release)

There's no appreciable difference between the sites in how product is released. To me it felt like the exact same process globally.

(overall)

Overall I think Mothership's SubSys teams have a lot more autonomy, that's not to say it's better. Sometimes it's just plain scary, it feels like you've got complete authority but the responsibility can be daunting. They relied on you going out and figuring it out for yourself. On the other hand, Dublin's FS teams often don't have a lot to do because the guys in SysArch have done all the design thinking for you. I think it's this way because we hire so many graduates straight out of college, inexperienced. At Mothership the teams are definitely older, more mature; a team of 20 might have 5 junior or graduate engineers whereas in Dublin it would be 10 or more. Sharing information, but also learning, is always the problem. In smaller groups I think it works better, knowledge growth. In Scandinavia we had set it up, that mix of senior and newbie, smaller groups to learn in.

(the 'Ask')
Jo paused, surprised at how much she had to say on the topic and at how different the two sites were in spite of them being the same company; 'one company' as the corporate vision described it.

"So Ian, why the interest?" asked Jo.

"Well... this is just a 'heads-up' as they say Jo but next quarter expect to hear head office announce that your team's code block at Mothership, plus a few others, will be relocated here, to Dublin." said Ian.

"Outsourcing to Dublin?"

"In as many words."

"But the teams at Mothership, what'll happen to them, they'll think I knew all along? And how's it going to work in Dublin?" asked Jo.

"Don't you worry about Scandinavia," said Ian, "but as for Dublin, you're going to be in charge!"

"Me? Responsible for bringing a large complex feature set from head office to Dublin?"

---

Notes:
*1. To illustrate the growth of data services over mobile consider Vodafone's 2010 Q3 mobile data revenue figures which exceeded SMS/MMS for the first time (article on The Register)
Nordic's global operations employ anywhere between 100,000 to 150,000 people around the world.

Questions/problems:
Transferring knowledge, how?
Is it necessary to reconcile the very different styles of working at the two sites?
Will the Scandinavian style of working translate to other sites?
How should they organise transplantation of product team/responsibility/activity from one site to another?
How should they manage transition?
Is cultural distance a serious issue or just a background characteristic or something else?
Are other sites viable?
Why Dublin?
Is this a near-shore sourcing arrangement?
The transfer seems to have high operation risk. Is this a fair statement?
What is the benefit of going from Scan to Dub
The transfer has low/moderate structural risk - non-codifiable process but same organisational family.
Are these really in fact complex and problematic business process? Or not? Justify...
Does identity affiliation and professional standards make transfer easier?
What skills, attitudes, structural, cultural aspects work to enable the process?
Problem of practices (culture 500 pgs) use radical exasketch?
Knowledge transfer problem may be overcome through tacit transfer occurring through interchanging team members.
The differences between Dub/Scan aren't between right/wrong; both work well in their context.
Does code need people?
Is the process model excessive?
Should the company consider open source approach?
Is the painting by numbers approach a misconception or a brilliant method for deconstructing difficult technical work?