The MGS Blog

Monday, January 24, 2022

DataCamp 2022 Outsourcing

This class has educator access for six months to DataCamp.

DataCamp for Classrooms access equates to full Professional access for a period of six months for all members of the account. Students and teachers have access to every piece of content on DataCamp (except for courses in Tableau, Power BI, and Oracle).

Access to self-directed and self-paced learning with "DataCamp's growing curriculum of expert content—designed for students of all data skills and levels."

Saturday, January 15, 2022

Cases - analysis/learning/synthesis

Case Analysis Exercises - Homework (mini cases)

Response organised as follows:
  1. Identify up to three (max) challenges/problems evident in this case.
  2. State knowledge/skills you currently possess relevant to these challenges/problems.
  3. Identify up to three (max) personal knowledge gaps related to any aspect of this case.
  4. Provide evidence of key learning sources you identified to address your personal knowledge gaps.
  5. Propose strategic/remedial actions that may address the three (max) challenges/problems evident in this case.

Friday, January 14, 2022

Exercise: Global Pharma (case)

Exercise: 15"

Read the case (again): 5"

In groups assess the suitability of one of the following areas for a pharmaceutical firm's sourcing strategy...
  • Email
  • ERP
  • MRP
  • HR
  • LIMS
  • Plant control systems
  • Public/market/marketing content management websites
  • Internal documentation and content management intranets
  • Storage
  • Collaboration technology
  • Logistics (3PL)
  • Regulatory compliance
Applying one or more of the instruments from chapter 2 to this example.

Debrief: 15"
Report to the class…

  • Service categories: General Business / Unit or Site Specific (HR, Finance etc) / Industry Specific

  • Strategy recommendation?

  • Cloud? / In-House? / On-Shore? / Off-Shore? / 3rd Party? /

Managing Global Local (sourcing case)

Interview with Declan, Project Programme Manager - Dublin.

"Bringing this project up to the starting line is a story in itself however I can say we had a great start. A really compelling prototype, demonstrated all the key functionality; web access, cloud data, simple workflow, fine grained ownership with security, reporting, detailed versioning, multilingual. We demonstrated the capacity to delivery a key internal service to the global company. You don't really need to know too much about the detail."

"The initial prototype was a joint production between Santa Monica (US) and Dublin, we pitched the project at our quarterly Knowledge Exchange event. In the company we call this kind of bidding process 'outsourcing inside'. We ended up with bids from three other groups, from our offices in Cairo (Egypt), Pune (India) and Langfang (China). Some of them signed up as 50% 'own time', the others were full-time. Cairo provided full-time developers, Langfang and Pune gave us full-time testers (figure 1)."
cloballocal
Figure 1. Global-Local initial project team

"We applied formal project planning from the start. The project is outsourced to multiple sites so a number of different cultures are involved. From our video calls I can say that the physical environments at all sites appear to be open plan cubicle formats, or 'tube-farms'. It is a tribute to the whole team that a passion for quality and for doing the job well has remained throughout and that is common across all geographies."

"The working cultures have little differences from office to office but I can say that's a strength. The vendor development team in Cairo appear to think little of working 24 hours a day, 7 days a week if they feel it necessary. Some of the flexibility shown in Cairo has its cost. After a 'spike' of effort they need to recover before reverting to normal working hours, inevitably your life outside (or lack of sleep) catches up with you. Those of us in Dublin and the US, we need our sleep and breaks just a little more but we're nearly always on hand to check mail, and, if necessary, work at weekends too. The test teams in India and China are very structured, very productive, and seem to always work the same hours, sometimes little later on busy weekdays, very rarely at weekends. In contrast, the part-time nature of the involvement of some of the players contributes to a certain lack of accountability, after all this isn't their 100% day job. You might load a work-task in Project at 50% but they aren't really able to allocate 50% all the time."

"With the number of locations involved in conference calls we've found that verbal communications can be a big problem. The mixture of accents in each office is compounded by differing audio quality across the sites. Audio and video quality degradation is just unavoidable. We're a global technology leader and only use best-in-class systems and still it's difficult to have perfectly clear group discussions. I think it has been a factor in generating some miscommunication."

"But the situation we're dealing with now... The formal project management approach requires an initial design process, very structured, heavy on documentation and long-winded. Crucially we agreed on multiple customers with different requirements, so every possible feature requirement was included in the plan from the start. You could say, with three customers the project might lack a certain direction. The customers (and I suspect architects) were essentially free to add their own personal dreams into the requirements and specifications. The requirements are also 'moving targets' because the goals of the different customer groups have shifted over time."

"We followed formal project planning practice, a one year schedule with three major milestones: alpha, beta, general release. Each feature had a feature spec written, subjected to a review and sign-off meeting. Feature spec review and sign-off ends up being quite lengthy and it has to happen for each feature. After the feature specification is nailed down the developers create a detailed design spec. This hammers out the finer details of the features. The test teams use the design spec to write a test spec and start generating 'white box' tests. The developers take the design specs and implement them. "

"It was pretty obvious that the formal approach takes time, but it can also lead to over engineered feature specifications. This causes problems elsewhere because of integration, features have to work with each other, and the interdependencies are complicated in unexpected ways. The alpha milestone was the point where first draft features were integrated. Debugging these interactions and interdependencies takes more time and generates further instabilities."

"It quickly became clear that while the features had been highly designed, perhaps even 'over engineered', their integration with each other was troublesome. While it had been considered during design it was obvious that each feature's designer should have studied the interfaces with the others far more closely. Another factor was that the customers had, in some cases, quite different requirements; the outputs of features customised for one customer regularly caused problems as inputs for features customised for another. Debugging the interactions between the features has pushed the schedule out by months."

"We've been on a 'death march' to reach the beta milestone. Management are beginning to question the project's funding. We have a management meeting next week, the project is close to being canned, but a lot of blood, sweat and tears has been invested. I think everyone wants to give it a chance."

"Starting with a concept, prototyping, and making the funding pitch is the base of a good technology recipe. How to succeed with distributed development teams? That's the secret sauce."

Core Banking (sourcing case)

Edited transcript of narrative style interview with Peter Forman, Executive CIO of International European Bank (IEB). SB (State Bank, Ireland) had been purchased by IEB one year previously and had recently rebranded to IEB and achieved full systems integration with its parent. IEB had also recently acquired Lokalny Bank Polska (LBP) and LBP's Baltic branch network. Identities and names anonymised.

"IEB is a leading player in Northern Europe; we reached this place through organic growth and well chosen acquisitions over the past ten years. What is interesting thing about us is we really don’t have an IT organisation. We have a development organisation! And my responsibility isn’t really IT, it’s product, process and systems so on.

Some things in the lower stack of IT operations have become a commodity, like IT operations, like how you handle a notebook like this (pointing to his laptop computer). We’re not differentiating ourselves on that. However, financial services are one of the most digitalised industries of all. What we are… The services we are selling, they are digital! I know you can still have credit card, and there are some notes, still, even in Ireland there are cheques still... so, of course there is paper as well, but really all our new service products are digital, so we really compete very much based on our abilities to use IT both in our business development but also in our productivity. So IT is very much the heart of that. Not the IT! But the development utilising IT!
SinglePlatformSOA
For us IT does matter! We compete based on our abilities of using IT to bring the best products and processes to the market. Technology is our production engine and information technology development is our business heart. And 'single platform' is at the heart of our business model, a single banking platform, a single way of doing things, across all our banking brands and customer services. All our banking 'brands', including SB, will and do use the same platform, 'Single Platform', and the customer is at the core of this. 'Single Platform' is built on a SOA, a Service Oriented Architecture.
CustomerCentred
I can announce today that we reached agreement on our newest acquisition, our biggest and most complex yet, Lokalny Bank Polska (LBP) and its Baltic brands. How will we achieve this, our largest integration so far? That’s why we have our offshore development and shared services centre partnership with one of India's fastest growing IT companies. They will assist some of the projects. We chose the partner that we ‘liked’, that had the same ‘model’ in mind as we had ourselves. And also of course an aspect, that we wanted a partner where we use two, three, four hundred developers but we would still be very important to them. If we needed to scale and that was really important for us but we had chosen a bigger company like Infosys or Tata, then maybe it wouldn’t be as important for Infosys or Tata.
IEB_developers
We started off in India in October last year and the plan was to steadily increase the number of employees, first ten, twenty, thirty, forty. But now the LBP opportunity develops, for this big project we have to do twenty hundred man-years of development in one year alone. We are currently twelve hundred developers across Northern Europe. They, I should say 'we' now have two hundred developers in India and a hundred or so in our Head Office, and we’ve added a lot of external contractors as well. Our partner, they’re doing the actual hiring of the people for us. And they are running Customer satisfaction call centre services as well, they actually have the Customer responsibility in some way, and actually, we treat them just like employees. They are developing using our development process, totally the same thing. Of course the training cost to bring them in, is a little bit higher, we use CMMI, that’s what they’re used to doing as well, so our development process is not different.

The development model also it’s the same development model for all persons. We are running CMMI projects, and it’s very much based on CMMI and focusing on CMMI disciplines, because, you cannot run too many different systems at the same time. Our development model is now compliant to CMMI level three. Many of our projects are close to being level three. But on average it’s only a little bit on top of level 2. In the past, we’ve had always quite good people, so we’ve been able to succeed with our projects, but sometimes, actually we’ve been able to succeed with a project even though the processes were not very good. So of course there’s a lot of advantage in being better at doing the work, but I think the most important thing, is if we, with the governance process, can ensure that we actually do the right work.

Selected quotes from an interview with Stuart, VP Digital of International European Bank (IEB), on sourcing, supplier selection and governance. Identities and names anonymised.
"We consider ourselves to be a technology driven boutique operation. We control the banking service-bus, the middleware. Control of the middleware, our API, is the heart of the bank." 
"Engagement over the tendering process was crucial; and what clinched it for us was culture matching. The culture match between us was crucial. Plus, we wanted a supplier that wasn't too large, one in which it mattered to them, that we were special, that we are big enough to appear on their corporate radar, that we matter." 
"We can outsource services, but we can't outsource our obligations. Because of that it cannot be a 'black box', you can never really step away from it. My approach is to 'see' but don't touch. Let the SLAs and KPIs work for you but it shouldn't be a black box, the box should be transparent. I want to see inside, if it's steam coming from a working engine or smoke from a fire."

We need warm bodies! (case)

Andrew (the firm's COO) has been having discussions with a specialist outsourcing firm from India and is meeting with Chris (CEO) and Charlie (VP Engineering) this morning. Chris has just returned from a state trade mission to China, and Charlie has been researching software development capabilities in the US, South America and Eastern Europe.

Profits are 40% of revenue, enough to plough investment back into numerous new product development and R&D projects. The ‘cash cow’ is a third-generation (mature) application that continues to generate substantial new sales and deliver a reliable stream of lucrative annual support fees. The firm's strategy so far has been to drive sales in new markets. They have achieved this by being close-to-the-customer, putting skilled engineers and sales people on site or in-country.

The first item on today's agenda is hiring. Andrew, Chris and Charlie reviewed a proof copy from HR of the job advertisement for next Friday's job section (see below) but are undecided about the value of 'yet another' advertising led recruitment initiative. For all the effort of the last six months of advertising, rounds of interviews and eventual job offers little has changed; not enough software engineers for the volume of work that needs to be done. In some ways the problem has become worse, of the last 25 job offers; 18 had rejected, preferring to take up positions in start-ups or larger American multinationals. The problem of attracting, hiring, and retaining talented software engineers is a major issue at all the firm's offices (New York, Boston, Houston, San Francisco, Sydney, Perth, Tokyo, Munich, Geneva, Dublin). Salary inflation reflects a tightening of the supply of qualified personnel and increasing costs are having an impact on the firm’s  profitability and stock price.

In-house engineer turnover and burnout is also a big issue. The fact is that the firm is struggling with a limited supply of IT professionals in local markets and is finding it difficult to convince the firm's own engineers to work on essential product porting and maintenance, or support on existing products. Turnaround and resolution times for escalated support issues has grown and the quality of patches and service packs released to customers is suffering as more experienced senior engineers prefer to work on 'more interesting' new-product projects.



Proof Copy: Job Advertisement for FIRM

Area: Engineering/Product Development
A Web 2.0+ engineering team focused on solving tough problems delivering highly usable solutions that build and leverage social networks.

Role & Responsibilities: Software Engineer
  • You will help to design, implement, and improve our Ruby-on-Rails based web platform, developing data intensive responsive cloud hybrid web applications.
  • You will be working with a modern toolkit on really exciting projects that you will help define, refine and deliver.
  • You love learning, learning with others, teaching and learning, listening (learning) and creating things.
  • You like to get things done, up and running and finished, with just enough polish.
  • You take responsibility for your own work, help others, and willingly receive help.
  • You will (as does everyone else) also contribute to essential porting, maintenance, and support for existing applications on platforms like OS/390, some mobile devices and ancient legacy systems.
  • You will work with Git, svn, or other cvs for code management; lighthouse for issue tracking and collaboration/communication tools such as Skype, Basecamp, blogging, and especially *wiki for team communication and project management.
Requirements:
  • Experience! (or BSc or Masters in Computing, Informatics, Engineering etc.)
  • Excellent team skills working with other engineers, non-engineers, and especially users/customers.
  • Open-source participation and server admin know-how (*nix and MS) a big plus.
  • Ruby-on-Rails, interfacing apps with 3rd party services like Twitter API via OAuth
  • Postgresql but also accept MySQL or similar framework and DB system.
  • Comfortable with agile software development practices but appreciative of waterfall rigor for deployment and delivery.

Symantec moves off-shore (case)

"The problem with success" thought Gordon Eubanks “is you need to keep changing the game to keep winning".
Gordon had achieved outstanding success in the software market in the US in a host of software product and utility categories (programming languages, antivirus, disk management and more). The company had developed in-house products and acquired competitors such that it was now the dominant supplier of these products in the US. What was left was the rest of the world; the question was how?

The Symantec board was meeting to review plans for the next stage in the company's expansion. Their strategy was to transform Symantec from being a national leader into the global brand for computer utilities and productivity applications. The corporate target was to reach sales revenues of over 100 million a quarter within three years. International sales already accounted for around 5 million dollars a quarter over the last financial year and its value was growing but not fast enough. International customers wanted software in their own languages, not just English. This meant localised versions with translated or enabled software interfaces AND help systems, user guides, packaging, marketing etc. It also meant that the software had to be tested on the types of computers and operating systems in use in those markets.

Some thought it odd that a company with its roots in LA and San Francisco had yet to produce a Spanish-language version of any of its products. But computing was a science and industry that had long neglected the needs of non-English language users. Gordon wasn't even sure if his products could be localised without more or less radical rewrites and redesign. Software with hard-coded strings and assumptions about character sets were embedded throughout most of the products. Then there was the Manufacturing challenge. The company was going to need boxed versions in German, French, Spanish, Italian, Swedish; they might even need to release a version of each product in UK English if the sales justified the cost or customers demanded it. And then there was ‘double byte’, the Asian markets Japan and China.

The company currently manufactures around 200 SKUs (stock-keeping units) for products in American English sold in the US and to the rest-of-world (ROW). With localised versions of all product lines they would end up with over 1,000 new SKUs to manage and maintain in the first year alone. Gordon considered his options; he could outsource manufacturing to 'turn–key' providers, manage the job 'in-house', or leave it to resellers and distributors to create localised versions and handle the manufacturing for each national market. The current reseller agreement with Softbank in Japan was just like that. Softbank did localisation and translation but it also got to keep most of the sales revenue. Japanese language versions of their products attracted a lucrative premium. OEM (original equipment manufacturer) deals with non-US PC manufacturers bundling 'lite' versions could also bring more money while at the same time reaching new customers who might then upgrade to 'full' versions or pay for monthly updates.

Should a single division be given responsibility for all language and manufacturing or should the work be farmed out to each national office? The problem with leaving it to resellers or country sales offices was one of quality control, marketing and message management. Can you imagine having one 'country version' translating the product name one way and another choosing to go with the English title? Should Symantec establish an international base and if so where? Gordon's erstwhile competitor Mitch Kapor, the creator of 'Lotus Notes' had established a software lab in Dublin, Ireland, but Israel and India also had growing software services and manufacturing sectors. Having a headquarters based inside the European Economic Community might avoid import tariffs if products were manufactured and had value added in Europe. There were also favourable corporate tax rates on profits in some countries.
Gordon pondered, “does success follow the money... or is it that money followed success?"



Why Ireland?
The Irish Development Agency (IDA) promotes Ireland as a 'pro-business' environment (www.idaireland.com - Investment Ireland). The IDA summarises the benefits of Ireland as follows:
  • a favourable tax regime,
  • a young and talented workforce,
  • a critical mass of relevant supporting industry and infrastructure,
  • a unique political and social commitment to supporting FDI and multinationals,
  • and excellent managerial talent.
The 2008-2012 Business Environment Ranking of the Economist Intelligence Unit placed Ireland 11th globally out of 82 countries.


Invest in Israel
Israel's Ministry of Industry, Trade and Labor (and www.investinisrael.gov.il/) promotes Israel as an investment destination based on the following criteria:
  • a positive business climate (access to venture capital, access to international markets),
  • an exceptional workforce (educated, entrepreneurial and multi-lingual),
  • investment incentives (grants, supports, structures)
Israel invests 4.5% of its GDP in R&D, which is the highest ratio of any country in the world (IMD World Competitiveness Yearbook 2008).


India: Why Maharashtra?
The Maharashtra Industrial Development Corporation (www.midcindia.org) - MIDC - of Maharashtra State (capital Mubai, regional centres Pune and others) claims the state as a proactive driver of inwards investment and support for entrepreneurs. It "continues to attract the largest quantum of investments, both domestic and foreign". Maharashtra's strengths include:
  • high literacy,
  • good infrastructure (power, ports, airports, rail, and road),
  • global players already based there,
  • access to an educated workforce,
  • surrounded by India's most prestigious universities and research institutes,
A joint survey of leading Indian States conducted by the World Bank along with the CII, found that Maharashtra has the best investment climate.

Happy Hollowing Customs (case)

The actors

ACS: the Australian Customs Service.
Murray Harrison: (a public servant) CIO for Australian Customs, 2002-.
Chris Ellison: (Liberal politician) the Federal Government Customs Minister, 2001-2007.
Eric Roozendaal (Labour politician) Ports Minister for the state of NSW, 2006-2007. 
Joe Ludwig: (Labour politician) Senator. Opposition Shadow Minister for Ports 2004-2007
Tradegate Australia Ltd: Established in 1989 as an industry representative body to develop and implement e-commerce services for Australia's international trade and transport industry. 
EDS: Ross Perot's international IT consultancy and outsourcing supplier. 
ICS: the Integrated Cargo System (ICS), part of the CMR project.
CMR: the Cargo Management Reengineering programme initiated by the ACS. 
COMPILE: The ACS system for managing import declarations, duty/tariff transactions, quarantine and bonding.
EXIT: The ACS system for managing export declarations and clearances. 
Verisign Inc.: An internet security digital signature services company.
IBM: International Business Machines.
COBOL: One of the first, wide spread, and most successful programming languages.
C: A programming language
CICS: Customer Information Control System, mainframe transaction server.
IMS: The Information Management System (database with transaction processing capability).
P&T: Post and Telecommunications Agency (aka Plain old telephone service (POTS)).


ACS_CMR_ICS
(figure 1. Key actors and technologies prior to ICS go-live)

The ACS Expands Outsourcing Initiatives

Going back to the 70s, 80s and 90s the Australian Customs Systems Division was at the front of transactional systems development. They had decades long experience developing resilient, scalable, available tech systems over a national network employing electronic data interchange. The first systems were dial up, then dedicated leased lines from the P&T. Systems architectures ran on mainframe, using COBOL and CICS and IMS, some C development and other software.
"The switch to Java and internet in the late 90s put a huge strain on our capability, we just weren't staffed up in those areas. Stupid hiring freezes across the civil service. There were restrictions on hiring new programmers into the civil service, the Department of Finance had no problem giving permission to pay consultancies 100K or 150K a year for bodies, but point blank refused to sanction recruitment of graduate programmers at 30K (even if we could get them). The net result of the ban on being able to hire in people was to choke off our civil service tech capability in Java and Internet. [Customs]"
In 1997 EDS, the global IT services company, secured a number of lucrative multi-annual service and development outsourcing contracts with the Australian Customs Service (ACS). EDS (Electronic Data Systems, founded by Ross Perot) specialises in business process and IT outsourcing services. Prior to the outsourcing contract with EDS the ACS managed, developed, deployed and delivered its core back-end systems almost wholly in-house. After the outsourcing deal, responsibilities for the ACS's core IT systems transferred over to EDS with gradually less direct involvement by ACS internal staff.
The old system worked well and was integral to the smooth operation of the national import and export system. It delivered high availability, responsiveness, and functionality for interactive or batch operation. Around this time Tradegate executives commented that the shift of system maintenance and development responsibilities (in additional to operational services) from ACS personnel to EDS resulted in a virtual shutdown in cooperation between the technicians on the 'inside' (of ACS-EDS) and those on the 'outside' (Tradegate).
The new supplier had no interest in involving Tradegate, you might say they went out of their way to ignore them. [Tradegate member]
We're all the victims of the tendering process. It was driven by the Department of Finance, the Tax Office. It was all about finance, cost savings, and Tradgate got slated. [Customs]
One consequence of the ACS-EDS deal was that the ACS lost more and more of its own specialist Customs IT expertise as employees transferred into EDS and then gradually moved on into other areas in EDS globally. Staff transfers from ACS to EDS also altered the apparent cost structure for getting regular work done, for example the ACS paid contractor rates for the services of EDS consultants, many of whom had previously been public servants for the ACS.
Customs IT people had opportunities to join EDS, a lot of them did, a civil servant's pay isn't great, but some stayed, for different reasons. [Customs IT]
It creates tensions, side by side, people you used to work with but now they're getting paid more, a lot shipped out, our knowledge has a value elsewhere in the organisation [EDS]. [ex-Customs IT now EDS consultant]
CMR and ICS
In 1998 the ACS and EDS announced a massive $A25 million programme of IT systems renewal called CMR (Cargo Management Reengineering). The CMR was going to be the Australian Customs Service's next generation IT system; a single system for managing, recording and controlling the movement of all goods movements into and out of the country. The Minister had made commitments in Parliament and the Customs IT innovation plans had been talked about nationally and internationally for several years. The proposed Integrated Cargo System (ICS) was to be the heart the CMR project. ICS would replace the ACS's venerable EDI based COMPILE system.
They [EDS] send in the 'A-Team' to get the CMR and ICS deals. They won. The 'A-Team' gets the deal the 'D-Team' gets to deliver it... people moved, Customs was hollowed out. [Customs IT staff]
It was a huge strategic shift, going from a 'community portal' sitting on top of Customs' EDI interfaces on mainframes to Customs outsourcing development and delivery to EDS. [Industry observer]
After over five years of development, delays and an increasingly fraught relationship between the ACS and EDS, ICS finally went 'go-live' on October 12 2005. Its users were compelled to switch over to the new system from an electronic trade management system that had served them for over 15 years.
The ICS go-live, October 2005
At the time of the ICS roll-out on the 12th of October 2005, Port Botany in Sydney was handling $A100 Million in trade per day. Up to the point of ICS 'go-live' Tradegate had provided essential front-end services on top of the ACS back-end. On ICS go-live day the old systems were simply turned off!
...they brought us in for training three months before go-live. [Trader]
Only a couple of the biggest importers were involved in user acceptance, I'm not even sure they were involved at the early stages for user requirements. [Trader]
ICS has the requirements of other government departments: Border Protection, Statistics, Taxation. But no business logic worth talking about. [Trader]
The operation of the new system was marred by widespread problems. Security access flaws allowed users to access and view other Traders’ import documentation, revealing confidential information like price, supplier details, quantities and delivery dates. The system was also perceived to be unreliable, badly designed, slow and buggy.
Errors come back ages after you make a clearance request. Why didn't the computer system pick up on it first? [Trader]
The error handling has problems, delays, I'm still waiting for responses to messages [in the system]. I might have 20 messages queued somewhere, meanwhile our containers are sitting on the tarmac at Kingsford-Smith (Sydney Airport). [Trader]
A couple of weeks after ICS go-live the key seaports, Port Botany and Melbourne Port, had ground to a halt as containers piled up. Trade effectively stopped due to ICS processing delays or other problems with the new systems. Australia's trade clearance system ground to a halt from October 12 and for the next six weeks containers and cargo accumulated at ports and bonded warehouses. The situation reached crisis point when Port Botany and Port of Melbourne ran out of room to store uncleared containers. The clearance backlog came about partially because ICS throughput was just 30% of normal handling rates, this led to excessive cargo dwell times at airports and sea ports.
I need Customs clearance to move them out of the port. [Trader]
That's the power of Customs, we are just pleased if we get their attention. [Trader]
The spectre of vessels (sea and air traffic) unable to disembark cargo was now a fact, Australia’s connections with the international trading system had seized up.
The system has fallen apart. Traders are getting seven days turnaround instead of seven minutes. In COMPILE declarations had 98% success on first time entry. In ICS, at roll-out we started with 60%, that's 40% fail! It improved slowly, 60%, 70%, 80%. [Tradegate employee]
Let me put it like this, ICS was already a couple of years 'late'. Even when the pilot roll-out started in March 2005 it was obviously generating problems. Parallel running was a complete failure because no one was going to switch over until they shut down COMPILE. There was a push to switch people in July and then finally they shut down COMPILE in October and the problems grew and grew with the shipping volumes. For traders the run up to stocking Christmas shelves starts in August and goes exponential. That's when we found ourselves in a full-blown crisis; Airports 'chock-a-block', container terminals full, container ships parked off-shore unable to unload. Worse and worse, so no, we did not have a happy Christmas! [Trader]

Postscript

A year later Senator Joe Ludwig (Lab), speaking to the Customs Brokers and Forwarders Council of Australia, reflected on the aftermath of the CMR programme and the ICS roll-out in particular. By this time Customs had a new CEO, Michael Carmody.
"The Cargo Management Re-engineering project, which was part of the trade modernisation agenda will be remembered as one of the all-time greatest implementation failures of a government project in Australia’s history. Its budget blew out to ten times the original estimate, it saw constant delays and, when finally switched on, it nearly brought down Australia’s trade industry." [Senator Joe Ludwig (Lab)]

Edited narrative interview with Australia Customs IT programme manager.
"So the first wave of our outsourcing experience was a massive learning process. We chose a strategic partner, and then switched, and then switched again. Partially driven by Government policy, partially driven by necessity. Each partner was basically giving us a Rolls Royce approach. But as it turned out they were also using us as a kind of training centre for their new hires. We ended up being the first post for their graduates and internships. They'd be trained for a year or more on our systems, processes, tech skills, management skills then ship out to consultancy gigs elsewhere. [Customs]"
"Since then we've taken a different approach. We've hived off little pieces. The way we wrote our tendering would allow an incumbent to continue (rather than switching at each renew), but we also started to divide out the work. Now we split it across roles rather than systems. They (suppliers/vendors) supply so many people as system architects, business analysts, programmers. But we keep control of the systems, they're our systems, we own them. [Customs]"
"We have gradually ended up with two strategic partners and loads and loads of smaller external vendors. That has worked really well. While X are our dominant partner we have loads of small specialist partners, local, Australian. It plays out well politically but it has really paid out with a regional ecosystem of expertise. [Customs]"
"We now build, deliver, operate (and own) everything. The internal team is just 30 people, pure technical excellence, constantly training up on new tech, deep knowledge of our architecture. But it scales because we're a programme management operation. It is our own Customs system framework, we own, develop and manage it. My team work side-by-side with the vendor people. There might be ten or twenty times as many vendor people around but we're in charge. It took us 5 years to reach this point, building internal capability and having a capability for the future. [Customs]"

Keeping PACE with the local (case)

The objective stated in PACE's original business plan was to give client companies access to best-in-area programming and testing capability through offshore outsourcing to India. PACE's strategy leverages their own specialised knowledge and capabilities to manage globally distributed engineering teams. They deliver talented people, lower costs, and faster turnaround for software projects.

PACE applies a 2-stage model for client onshore/offshore transitions (figure 1); the first year of the contract is conducted onshore working closely with the client on-site after which the project transitions swiftly to an off-shore operation. The value proposition to clients is threefold and simple: project cycle time, management overhead and employee cost reductions will occur because the Indian team has access to more people, at lower cost, and working across European and American time zones. An added benefit is that project work continues around the clock if necessary.
onshore-offshore
(figure 1. PACE lifecycle for onshore/offshore transitions)
In its first 18 months of operation PACE has secured 2 European mid-sized software product development firms as clients. A further three prospective clients are nearing contract signing and they have a qualified pipeline of 12 new potential clients. Both of the active contracts are for an initial three-year duration with an option to extend. The first client contracted 60 person-years (over three years) and the second 45 (over three years). The first client company develops products that enable customers to create distributed cross-platform information systems. The second client company's products service secure point-of-sale (POS) transactions and payment validation. Both clients state that the partnership with PACE will allow them to scale; they need more engineers to grow their own business, to handle the needs of existing customers, and to create new products addressing new market opportunities. PACE's engineers are intended to service the companies' legacy products, basically they take on support, maintenance, and patch production.

PACE's business plan specifies an IRR (internal rate of return) of between 30% and 50% for its investors, a simple payback period for the original investment of just 2 years, plus annual growth potential of up to 50%. The up-front (sunk) setup cost was 100k. The firm's fixed costs included 2 full-time sales and marketing roles (both based in Europe), in addition there are 3 employees based in India tasked with sourcing and vetting candidate CVs, arranging interviews with client companies (online and via conference calls), visa processing, and other home-base activities aimed at supporting the eventual offshore transitions for projects moving to India. These on-going fixed costs (including accommodation, service charges, travel etc.) amount to around 160k per annum.

Both current clients are nearing the end of the first contract year with teams of 20 and 15 PACE engineers respectively working in onshore mode; however both companies are now expressing a desire to delay (if not a reluctance to proceed to) the next stage - transitioning to offshore operations! To complicate the situation the Indian engineers were hired with a clear expectation of returning to India; relocation to Europe is seen as a temporary assignment. Ambiguity surrounding return dates is becoming a source of dissatisfaction. Many had signed on because they wanted, eventually, to work near home, family and friends. European Winters are cold and wet, the cost of living is high and food, lifestyle, and cultural difference are a source of personal and family stress.

Delays in transitioning clients to full offshore outsourcing mode could have a major impact on PACE's business model. Simply put, the company's expected net revenue (surplus) for each contracted engineer is 5k per year while the engineer is working onshore at the client site, whereas it is 25k per engineer per year after the project transitions to full offshore mode. In onshore mode the company is barely able to meet its operating costs let alone accumulate surpluses for profit sharing (a promised employee incentive) or provide a return to investors. The company needs to transition clients to full offshore mode if it is going to do more than survive.


onshore-offshore_costmodel

We have a problem Heuston (case)

Tom runs a product concept and design company that handles high-level web and marketing activities while outsourcing development. Although his business is located in the digital campus beside Heuston Station he claims there are just too few (affordable) software engineers in the Irish market for a small firm like his to build up its own development capability, especially as much of the work can be sourced online. He contracts software implementation and delivery to a small number of trusted outside firms that he has worked with over the years. Tom handles client facing interaction, gathering the requirements, and sourcing delivery from his contacts. He thinks of the software code merely one facet of the business, as something that just 'gets the job done';
"yes code is intellectual property but you can't be precious about code. For me it's the relationship with my client, building up a joint understanding of how we want a 'thing' to work. To be honest I couldn't care less about the code as something I own".
The CMS project was a nice earner in 2005 but now five years later, it is becoming a bit of a headache. The original developer had moved to America but passed the source code to Tom who in turn contracted a firm in India to maintain and develop it. The Indian firm was founded by Suhas and a number of partners who were IT professionals and had worked together previously in Europe. Suhas had taken on the CMS contract almost as a favor to Tom and the original developer, a personal friend. As it happened neither Suhas nor the developers on his team had ever met Tom in person or dealt directly with Tom's clients (CMS's customers and users). Tom managed all contact with India, usually through Suhas but occasionally contacting the developers assigned to the CMS by email, Skype or phone.

Pune, called the Oxford of the East by India's first Prime Minister Jawaharlal Nehru, is home to many high tech, consultancy and software companies centred around a vibrant community of universities, research institutes and colleges. Pune competes for international software outsourcing business with other locations in India and across the World. However Suhas's view has always been that Indian engineers should innovate and add value, rather than just take part in an international outsourcing race to the bottom in terms of cost or manpower. He founded the firm on principles of integrity and fairness...
"knowledge has power, but it takes time to acquire and it is only useful if it is shared and used!" 
"I always feel that we can not be threat to our clients because the end customer always prefers to have a local company to deal with which can outsource the work to outside world."
The CMS project has a history, characterised by periods of intense involvement (development and delivery work) followed by gaps of months at a time with little or no contact. The engineers who had worked on it over the years were often overwhelmed, picking up where the project had left off and working on fixes or new features with few opportunities for feedback with the end customer. A release could be delivered after which it might take Tom a week or more to report back on the quality or its acceptance. This sort of stop-start, delayed back and forwards was not typical for Suhas's other contracts. Suhas felt that the CMS was a perfectly good application that already satisfied the requirements, however Tom's point of view was that the customer was always right, if the customer was unhappy then it was a bug and it had to be fixed. For Tom, just understanding a customer's problem took time and effort, explaining it to Suhas or one of his engineers complicated things further. In spite of this Tom was concerned to keep the customer at 'arms length' from India, after all, if they dealt directly with each other what was the point of having Tom in the relationship? Suhas felt there could never be an issue of 'going behind' Tom, he reasoned that Tom owned the CMS, it was Tom's property. What Suhas wanted was for his engineers to communicate directly with the end-user so they could clarify the requirements, satisfy them completely, identify and fix bugs, get on with the job and get paid.


CMS: Content Management System(s)

Content management systems are applications for managing and structuring information and data accessed and delivered over the Internet. The term CMS is often synonymous with 'portal' or web site however several crucial features distinguish a CMS from other internet services including: publication workflow, fine grained user roles and privileges, extensive data/object types (event, wiki, news, chat, blog, slideshow, video, audio, forms, mail, surveys, RSS), TTW (Through The Web) administration and use, extendible via Javascript and other programming methods.
A general overview of different CMSs is available at the cmsmatrix.org. cmsmatrix.org enables feature comparisons between the many CMS product/applications available.

ICS Go-Live 2005 (case)

Disclaimer: The case as presented here is a partial account and has been edited for teaching purposes. This material has been simplified for brevity and narrative coherence. The case has been designed as a teaching aide to encourage independent investigation, learning, discussion and debate rather than as criticism, illustration of management practice, endorsement of any particular approach or product. Interviews and notes were gathered from private conversations, discussions, and data from documentation in the public domain covering the period prior to 2005 and up to current.

Introduction
The Australian Customs Service (ACS) has been at the cutting edge of electronic systems development for over 30 years, controlling trade into and out of Australia’s borders. The ACS's earliest systems were delivered using a variety of hardware, networks and applications: mainframes, direct connection (EDI), community ISP facilitated EDI, and internet based portals. The ACS's applications and service delivery mechanisms were precursors to current international initiatives involving the WCO (World Customs Organisation), to develop standards based Single Window Access (SWA) systems for Customs services. Single Window Access offers the possibility of 'interoperable integrated international Customs systems' and thus the potential for global control, risk management and regulation of international trade.

Background
The ACS has long been recognised as a centre of excellence for Customs systems and innovation. Historically technology providers have serviced government's needs for computing hardware, networks and system software. The need to match application capability with international trade systems (domain knowledge) against a backdrop of constant regulatory evolution meant that Customs IT innovation has been driven by Customs agencies rather than by the IT industry.

In 2001, the Australian Government updated the Customs and other related Acts in an effort to allow Customs IT to change more freely. The 'International Trade Modernisation' package updated the legislative framework for Customs to allow for a more flexible approach. It did not prescribe individual systems but rather introduced a kind of indirect approach by requiring "that Customs gazette its IT requirements." In effect earlier Customs Acts, the law under which Customs operates, had specified the IT systems to be used through law, therefore changes to IT systems had to be reflected in changes in the Act. Once the modernisation package was adopted, changes to the IT could then be notified by gazette in Parliament rather than being enacted through legislation. The IT still had the authority of law, but the IT did not need to be specified by the law.

Customs sees its role in the national and international trading system not as a 'tax' on industry but as a necessary control; regulating and protecting society while at the same time enabling trade and economic growth. Consequently the World's earliest electronic, real-time, high transaction volume, standards based, trade management networks, were Customs import/export systems. In many ways the demands of Customs agencies over succeeding decades has driven the IT industry agenda, first by specifying and establishing EDI standards, and second by providing the necessary business case for suppliers and service providers. The business case for Customs IT innovation has been based on controlling while facilitating traders' imports and exports at the lowest possible cost.

ACS e-Commerce Systems
From 1989 to 1999 the Australian Customs Service (ACS) designed, delivered, and operated 'back-end' electronic systems to process import and export declarations, duty payments and other essential trade services. ACS e-Commerce systems were back-end mainframes running purpose built software like the COMPILE system for imports and the EXIT system for exports. Licensed traders could access them directly or via a third party service operator, Tradegate. Tradegate was a national trader community organisation that cooperated with the ACS to provide end-user interfaces to the ACS back-end; its mission was to develop and deliver the e-commerce 'face' of the ACS's import and export clearance systems. Tradegate was established in 1989 as a not-for-profit organisation comprising the importers and exporters and the wider trading community as members. Tradegate had also been responsible for provisioning a secure national electronic network prior to the widespread availability of the Internet by contracting and managing specialist telecommunications network providers, first with Paxus then AT&T EasyLink. Traders around Australia could use Tradegate to access COMPILE and EXIT from anywhere at any time.
COMPILE was very forgiving, I enter code '32' in the box and it always goes through.[Trader]
Tradegate was funded by member fees, a per transaction charge for portal use and fixed costs for network access and the ACS held a statutory directorship on the board. Tradegate's executive, its members and its board considered themselves to be partners with the ACS, providing a stable, secure and effective service and a key enabler for Australia's international economic success.
In the old days Tradegate and Customs had indirect avenues. The Tradegate guy was probably playing golf with this guy, he could walk upstairs and have a chat. They got things done. [Trader comment]

The ACS Expands Outsourcing Initiatives
In 1997 EDS, the global IT services company, secured a number of lucrative multi-annual service and development outsourcing contracts with the Australian Customs Service (ACS). EDS (Electronic Data Systems, founded by Ross Perot) specialises in business process and IT outsourcing services. Prior to the outsourcing contract with EDS the ACS managed, developed, deployed and delivered its core back-end systems almost wholly in-house. After the outsourcing deal, responsibilities for the ACS's core IT systems transferred over to EDS with gradually less direct involvement by ACS internal staff. Around this time Tradegate executives commented that the shift of system maintenance and development responsibilities (in additional to operational services) from ACS personnel to EDS resulted in a virtual shutdown in cooperation between the technicians on the 'inside' (of ACS-EDS) and those on the 'outside' (Tradegate).

The new supplier had no interest in involving Tradegate, you might say they went out of their way to ignore them. [Tradegate member]

Referring to this period in a submission to the Australian Government in 2001 Tradegate claimed that
"Contact with the key technology and service development staff [has] virtually ceased. The ability to progress externally beneficial service improvements through regular contact and 'brain storming' sessions came to an end with all contact placed on a formal basis presumably due to the dictates of the ACS-EDS contractual arrangements."
It was apparent that the ACS-EDS deal altered to the detriment the relationship between Tradegate and ACS, that the previously close contacts were disrupted and not re-established when EDS entered the equation.
We're all the victims of the tendering process. It was driven by the Department of Finance, the Tax Office. It was all about finance, cost savings, and Tradgate got slated. [Customs]
An unexpected consequence of the ACS-EDS deal was that the ACS lost more and more of its own specialist Customs IT expertise as employees transferred into EDS and then gradually moved on into other areas in EDS globally. Staff transfers from ACS to EDS also altered the apparent cost structure for getting regular work done, for example the ACS paid contractor rates for the services of EDS consultants, many of whom had previously been public servants for the ACS.
Customs IT people had opportunities to join EDS, a lot of them did, a civil servant's pay isn't great, but some stayed, for different reasons. [Customs IT]
It creates tensions, side by side, people you used to work with but now they're getting paid more, a lot shipped out, our knowledge has a value elsewhere in the organisation [EDS]. [ex-Customs IT now EDS consultant]
CMR and ICS
In 1998 the ACS and EDS announced a massive $A25 million programme of IT systems renewal called CMR (Cargo Management Reengineering). The CMR was planned to be Australia's next generation IT system, responsible for managing, recording and controlling the movement of all goods movements into and out of the country. The Minister had made commitments in Parliament and the Customs IT innovation plans had been talked about nationally and internationally for several years. The proposed Integrated Cargo System (ICS) was to be the heart the CMR project. ICS would replace the ACS's venerable EDI based COMPILE system.
They [EDS] send in the 'A-Team' to get the CMR and ICS deals. They won. The 'A-Team' gets the deal the 'D-Team' gets to deliver it... people moved, Customs was hollowed out. [Customs IT staff]
It was a huge strategic shift, going from a 'community portal' sitting on top of Customs' EDI interfaces on mainframes to Customs outsourcing development and delivery to EDS. [Industry observer]
After over five years of development, delays and an increasingly fraught relationship between the ACS and EDS, ICS finally went 'go-live' on October 12 2005. Its users were compelled to switch over to the new system from an electronic trade management system that had served them for over 15 years.
ACS_CMR_ICS
(figure 1. Key actors and technologies prior to ICS go-live)
The ICS go-live, October 2005
At the time of the ICS roll-out Port Botany in Sydney was handling $A100 Million in trade per day. Murray Harrison (a public servant) was the CIO for Australian Customs, Chris Ellison (Lib) was the Federal Government Customs Minister and Eric Roozendaal (Lab) was Ports Minister for the state of NSW. Tradegate Australia Ltd was established in 1989 to develop and implement e-commerce services for Australia's international trade and transport industry. EDS is an international IT consultancy and outsourcing supplier. ICS is the Integrated Cargo System (ICS), a key element of the Cargo Management Reengineering (CMR) programme initiated by the ACS. The ACS is the Australian Customs Service.
Up to the point of ICS go-live Tradegate had provided essential front-end services on top of the ACS back-end. The existing system worked and was integral to the smooth operation of the national import and export system. It delivered high availability, responsiveness, and functionality for interactive or batch operation. On the ICS go-live day the old systems were simply turned off!
The 'refreshed' systems roll-out, it was called ICS. They presented it like an update of COMPILE, like it was a version upgrade but it should have been called something else. [Trader]
They were supposed to have launched in March, last year or the year before![Trader]
...they brought us in for training three months before go-live. [Trader]
Only a couple of the biggest importers were involved in user acceptance, I'm not even sure they were involved at the early stages for user requirements. [Trader]
ICS has the requirements of other government departments: Border Protection, Statistics, Taxation. But no business logic worth talking about. [Trader]
The operation of the new system was marred by widespread problems. Security access flaws allowed users to access and view other Traders’ import documentation, revealing confidential information like price, supplier details, quantities and delivery dates. The system was also perceived to be unreliable, badly designed, slow and buggy.
Errors come back ages after you make a clearance request. Why didn't the computer system pick up on it first? [Trader]
The error handling has problems, delays, I'm still waiting for responses to messages [in the system]. I might have 20 messages queued somewhere, meanwhile our containers are sitting on the tarmac at Kingsford-Smith (Sydney Airport). [Trader]
A couple of weeks after ICS go-live the key seaports, Port Botany and Melbourne Port, had ground to a halt as containers piled up. Trade effectively stopped due to ICS processing delays or other problems with the new systems. Australia's trade clearance system ground to a halt from October 12 and for the next six weeks containers and cargo accumulated at ports and bonded warehouses. The situation reached crisis point when Port Botany and Port of Melbourne ran out of room to store uncleared containers. The clearance backlog came about partially because ICS throughput was just 30% of normal handling rates, this lead to excessive cargo dwell times at airports and sea ports.
I need Customs clearance to move them out of the port. [Trader]
That's the power of Customs, we are just pleased if we get their attention. [Trader]
The spectre of vessels (sea and air traffic) unable to disembark cargo was now a fact, Australia’s connections with the international trading system had seized up.
The system has fallen apart. Traders are getting seven days turnaround instead of seven minutes. In COMPILE declarations had 98% success on first time entry. In ICS, at roll-out we started with 60%, that's 40% fail! It improved slowly, 60%, 70%, 80%. [Tradegate employee]
The chairman of the Customs Industry committee said 'it was chaos', 'blood on the ground'. [Trader]
Let me put it like this, ICS was already a couple of years 'late'. Even when the pilot roll-out started in March 2005 it was obviously generating problems. Parallel running was a complete failure because no one was going to switch over until they shut down COMPILE. There was a push to switch people in July and then finally they shut down COMPILE in October and the problems grew and grew with the shipping volumes. For traders the run up to stocking Christmas shelves starts in August and goes exponential. That's when we found ourselves in a full-blown crisis; Airports 'chock-a-block', container terminals full, container ships parked off-shore unable to unload. Worse and worse, so no, we did not have a happy Christmas! [Trader]
Postscript

Press Report: "Australian Customs Service chief executive Michael Carmody has promised an 'open and frank' exchange with industry on how to fix continuing problems with its troubled Integrated Cargo System. The former Tax Commissioner said he would share with the industry the results of an independent report on how to squeeze more benefit from the system, which Customs commissioned last month from consultancy Booz Allen Hamilton. Mr Carmody said the $A200 million system was still a long way from delivering the benefits that had been expected - either for the import-export industry or Customs..."[Riley, 2006]
On 9th June 2006 a $A300,000 report by Booz Allen Hamilton on how to fix the $A200 million Customs Integrated Cargo System was handed to government. The report examined the circumstances leading up to and following the deployment of ICS, the core system controlling the movement of imports and cargo into Australia. The introduction of ICS had “caused havoc at Australia’s wharves and airports in October last year” (Mitchell 2006), with on-going implications for the Australian government such as facing “$A9 million in compensation claims, mainly for extra storage costs incurred when containers were left sitting on the wharves unable to clear customs” (Hayes 2006). Australian importers carried the cost of security shortcomings, delays, storage charges, lost business, reduced performance and increased workload.

A year later Senator Joe Ludwig (Lab), speaking to the Customs Brokers and Forwarders Council of Australia, reflected on the aftermath of the CMR programme and the ICS roll-out in particular. By this time Customs had a new CEO, Michael Carmody.
"The Cargo Management Re-engineering project, which was part of the trade modernisation agenda will be remembered as one of the all-time greatest implementation failures of a government project in Australia’s history. Its budget blew out to ten times the original estimate, it saw constant delays and, when finally switched on, it nearly brought down Australia’s trade industry." [Senator Joe Ludwig (Lab)]
References
  • Hayes, S. "Customs pushed into chaos," in: The Australian, 2006.
  • Ludwig, J. "Speech to the Customs Brokers & Forwarders - Current Issues in Trade Security," (speeches at www.senatorludwig.org) accessed Dec 2006.
  • Mitchell, S. "Customs report on IT fiasco," in: The Australian, 2006.
  • Riley, J. "Customs 'open and frank' on ICS," in: The Australian 2006.

You've got email MAT (case)

You've got email+malware+viruses+zombies+spam+fishing+adware.
Móin Alúine Teichneolaíocht (MAT) is the brainchild of two engineers who had worked in Digital’s Galway campus in the 1990s. While MAT's customers are risk adverse, conservative multinational banks and financial services operators, they value MAT's rapid innovation and reputation for adapting quickly to technological change. ‘Prime Broker', MAT's first product, broke new ground when it was launched, based on open standards and interoperability; Web 2.0 was designed into the heart of its architecture.

Unfortunately over the last 6 months MAT had suffered a run of bad luck:
  1. A small fire in the server room had destroyed hardware in the room including the router, firewall and wiring to the ISP.
  2. MAT's ISP experienced 2 outages or critical network failures that affected MAT's business communications including support SLA turnaround times on bugs.
  3. On no less than 6 occasions malicious bots had infected the mail server, outbound traffic became overwhelmingly bot generated and internet connectivity was reduced to a crawl. Worse, the bot traffic had red flagged MAT's IP and MX registrations, which were then 'blacklisted' on authoritative public watch listed servers. Consequently their IP and MX records were 'quarantined' by the gods of the Internet.
  4. A knock-on impact of the mail server problem was that malicious spyware and bots had infected some internal workstations. MAT suspected that certain file systems had been compromised (accessed and copied).
In January 2012 the engineering and IT teams hosted an off-site day where they reviewed the company’s performance from the perspective of skills, tools, technology and trends. Among the topics discussed was the IT infrastructure (internal and external facing) among which Email and Messaging services stood out.

Email is a basic necessity for business and it has been the communication lifeblood of MAT from day 1. Newer instant communication and messaging technology like SameTime and ‘Skype’ is also extensively used by the engineers (Product Development, Test, Support, Professional Services and IT). Everyone manages three or more personal email accounts in addition to their corporate (firstname.secondname@mat.com) identities.

One proposal under consideration is to shift from the internal email service to a fully external email service from Google or another 'cloud' provider. The CEO has also asked if cloud services can replace internal systems, systems that might be better provided by an external operator.

Also discussed at the off-site were the results of a recent external ISO9001:2008 quality audit that identified shortcomings in the company's business continuity and disaster recovery plan; specifically, how core infrastructural services (email in particular) are maintained and proofed against disaster. Business continuity and disaster recovery plans are necessary to ensure that we can restore "adequate alternative arrangements for systems which need to be operated in the event of a breakdown". Critical Infrastructure is defined as the tools & structures necessary to operate essential services or carry out core activities, for example; internet access, mail server, source code control system, and the telephone system.



So how much could you save with Google Apps for Business?
Google's corporate email service is a key feature in its recent paid service, Google Workspace (previously Google Apps for Business). See https://workspace.google.com/pricing.html.


The Google Workspace pricing model



MX Toolbox - Email Blacklist Checks and Diagnostics (www.mxtoolbox.com)

The MX Lookup Tool can test and list "MX records for a domain in priority order. The MX lookup is done directly against the domain's authoritative name server, so changes to MX Records should show up instantly. You can click Diagnostics, which will connect to the mail server, verify reverse DNS records, perform a simple Open Relay check and measure response time performance. You may also check each MX record (IP Address) against 147 DNS based blacklists . (Commonly called RBLs, DNSBLs)"



Celtic Tiger, Chinese Dragon (case)

The company organises its 'enterprise software' development and production teams according to a 'lean development' framework. The management team's emphasis is on the top line; new sales, big deals, big customers, (even better to get big-new-customers). They had achieved success in the past by being close to the customer, understanding the customer, achieving 'fit' with the customer. The executive team and the board of directors saw bottom-line cost control as important, however close involvement with the company's customers, bringing features inside the core product had enabled continuous renewal and evolution of the product lines.
A number of consultancy projects in China were now exceeding the size of those in Europe and North America. Chinese semi-states and private corporations were investing heavily in cutting edge, large scale, IT infrastructure. Of all the developing nations China was the biggest and growing the fastest. In the end the decision to setup a centre of excellence in China was straightforward.


Excerpts from interviews with a Senior Software Engineer (Chinese) and the Director of Product Management (China office).

(Director of Product Management)
"...one of the reasons is [the China office] adds value to the company. If we think about the four of five possible strategies for a growing tech company, well, having a foothold in your market, plus smart people, smart customers, high profile, high value projects. It ticks all the boxes."  
"Establishing a centre of excellence in China is a dream project in many ways. When you acquire a company you have to work with a pre-established team. But setting up a green-field R&D centre, like now, it's a clean slate, new office, new hires, using our 'foreign' methodology."
"Our in-house development methods and practices are inspired by the Agile and Lean movements. Projects, operations and development use our own take on the Scrum or Kanban style of process. I have decided to use our in-house methods from the start. The local engineers might look on it as a 'foreign' approach but I don't think we'll see resistance. They are open to learning about it." 
"My view is that Agile is a strict discipline; it's more disciplined than other methods if you do it in a highly structured way, get rid of the 'ad hoc' style that seems to be associated with it. Start your stand-ups at 10am (not 5 past 10), finish strictly at 10:15am. The stand-up conversation should be structured; specific challenges approaching, specific problems you are facing, you've either run or written a test case. Programmers will like that. There's 'do or don't', 'did or didn't', no ambiguity and you need these type of conversations. Agile brings this to the front." 
"But yes, I am worried about the difference in culture, particularly problems with 'face'. How do you get engineer's admitting to difficulty or failing to achieve something?" 
"When you set up an office in another country, there's a fear among your people at the home base! You immediately create the potential for a social divide that applies when you move to an offshore site." 
"Imagine, your company sets up an office in China where the engineering costs are lower. Do you want a situation where you say 'designed in California, made in China'? What does that say about a social divide in the company? That's the choice you have to make." 

(Senior Engineer)
"In China software engineering career has quite a lot of respect." 
"In general, software engineers work long hours, often seven days a week, starting early, leaving late." 
"International companies might have more formal policies, leave work at 7pm, come back in the morning. But they might not be as flexible as smaller companies that expect working around a project. If an emergency comes in a small company they might have to work all night."
"Yes, there is a lot of employee turnover. Conditions can be bad, the overtime, bad management, working away from home." 
"Now Agile is well known in China. In the real implementation there are some cultural differences that might make it feel different to an implementation in the US. But a company, should not admit some failure to its customers." 
"Let me put it this way; In western countries, people like to treat work like playing games. But here in China, a manager here is something like the parent, not childlike." 
"I don't accept the childish approach. Engineers think 'we are not kids, we are adults'. So Agile can inspire engineers in the West because it is playful, but in China, I don't know?"
"If your company commits to something your business lives or dies by it. But that's business in China, completely different to the West. Completely different."

Related Data
  • Agile 101 - Methodologies, Learn more about Scrum, XP, DSDM, FDD and Lean software development (www.infoq.com).
  • Understanding Chinese business culture: potential starting points for developing an understanding of guanxi (relational) and mianzi (face/status/rank) on wikipedia.org or google search.
  • Issues around manufacturing in China (sources: Irish Times Thu, Dec 29, 2016, 17:17; Seattle Times December 31, 2016; original article New York Times)

Relocating knowledge (case)

Jo had just returned from an assignment at Nordic's (a pseudonym) headquarters in Scandinavia nicknamed the Mothership. It was a planned six month rotation that had extended over a year, intended as a secondment for professional development, part of her career path to senior engineer that couldn't be achieved at Nodic's base in Dublin. While there the team she had been attached to had designed a software block for smartphone services, a crucial part of the company's technology roadmap and when one of the engineers took paternity leave her assignment was extended to cover his absence. This software block was an element in the next generation platform delivering advanced data services over mobile networks which were expected to grow exponentially(*1). After the release Jo had stayed on to help with the first update and was then offered an indefinite post, but 13 months away from home, friends, and family was enough; Mothership was nice but it wasn't home.
"So Jo, tell me what it was like in the Mothership?" said Ian.
"Great, yeah. You know, like the 'one company' slogan, it was great." said Jo.
"Right, yes, 'one company.' Look, I want you to be frank, don't hold back, I need to understand what it was really like, off the record." Ian replied.
"My impressions?"
"Your impression of the differences between the two sites, between the groups in Dublin and the Mothership, how things really work."
"Ok, since you've asked" Jo said. "So you're familiar with the organisational structure here in Dublin..."
Jo started to draw a sketch of Nordic’s organisational hierarchies in Dublin and at Mothership in Scandinavia (Figure 1 below).
Ian started: "In Dublin the feature set (FS) teams report into one of the the SysArch committees who report into the Release Div (Unified Systems Release Division) and Subsystem Group."
It's a bit flatter at HQ" Jo continued. "SubSys (Subystem) teams report into the local Subsystem Group, the Release Div there seems to work more like a service group. The Product Architecture Board sits at the top of it all of course, same as here."
NordicOrgChart
Figure 1 Organisational arrangements at Mothership and Dublin
Once she had started talking Jo found she had a lot to say about the differences, between working in Dublin and at Mothership.

(design)
Something I didn't expect while I was there; the SubSys teams, although they're like our FS teams, they manage whole code blocks rather than just down to feature level. Mothership's SubSys teams manage the feature deltas so they also manage old versions within the team. My experience of the approach in Dublin couldn't have been more different. In Dublin SysArch keep the feature design deltas to themselves, SysArch hand down design documents with feature work specified from 'soup to nuts', engineers have nothing to do, SysArch have done it all - "here's how you prep, here's how you deliver, here's how you pass the exam!" Let me put it this way... On my team at the Mothership, the design document might be a 'use case' diagram and a flow chart, a picture and two sentences. I had to talk to the others to get anything done, there was so little detail to the design documents it was scary! Contrast this with Dublin, a design document for feature deltas is massive, 300 pages. One of Dublin's SysArch told me "I'm proud of our design docs, I really am," but 300 pages for one subsystem? You can't read it!
(support and bug tracking)
In Dublin some FS teams can avoid getting involved in supporting old versions at all, Support chase down the engineer who touched the code, who then hands over to the build group and then Test; the process is managed by Support using the issue tracking system. Mothership use the same issue tracking system but SubSys engineers rotate on bug fixing and the assigned engineer follows a problem through to its resolution. In Mothership the bug 'stays on you,' but of course the SubSys team is there to back you up because it's ultimately the team's responsibility. The way it goes in Dublin - the bug bounces from one person to the next, from one team to the next - Dublin's strength is the strength of the system process.
(test and support engineers)
What I thought worked well at Mothership was having a test engineer and a support engineer working on rotation in the SubSys team. In Dublin the support and test groups maintain an almost clinical separation between themselves and the engineering teams; it leads to a bit of finger-pointing at times. In contrast at Mothership the rotation gives the engineering team a 'tame' tester and support person, there's no delay in getting feedback. I also heard from the test and support guys that they learnt more this way, about new features, or if there were problems later on they'd have a direct line into the engineering group.
(release)
There's no appreciable difference between the sites in how product is released. To me it felt like the exact same process globally.
(overall)
Overall I think Mothership's SubSys teams have a lot more autonomy, that's not to say it's better. Sometimes it's just plain scary, it feels like you've got complete authority but the responsibility can be daunting. They relied on you going out and figuring it out for yourself. On the other hand, Dublin's FS teams often don't have a lot to do because the guys in SysArch have done all the design thinking for you. I think it's this way because we hire so many graduates straight out of college, inexperienced. At Mothership the teams are definitely older, more mature; a team of 20 might have 5 junior or graduate engineers whereas in Dublin it would be 10 or more. Sharing information, but also learning, is always the problem. In smaller groups I think it works better, knowledge growth. In Scandinavia we had set it up, that mix of senior and newbie, smaller groups to learn in.
(the 'Ask')
Jo paused, surprised at how much she had to say on the topic and at how different the two sites were in spite of them being the same company; 'one company' as the corporate vision described it.
"So Ian, why the interest?" asked Jo.
"Well... this is just a 'heads-up' as they say Jo but next quarter expect to hear head office announce that your team's code block at Mothership, plus a few others, will be relocated here, to Dublin." said Ian.
"Outsourcing to Dublin?"
"In as many words."
"But the teams at Mothership, what'll happen to them, they'll think I knew all along? And how's it going to work in Dublin?" asked Jo.
"Don't you worry about Scandinavia," said Ian, "but as for Dublin, you're going to be in charge!"
"Me? Responsible for bringing a large complex feature set from head office to Dublin?"



Notes:
*1. To illustrate the growth of data services over mobile consider Vodafone's 2010 Q3 mobile data revenue figures which exceeded SMS/MMS for the first time (article on The Register)
Nordic's global operations employ anywhere between 100,000 to 150,000 people around the world.

Questions/problems:
Transferring knowledge, how?
Is it necessary to reconcile the very different styles of working at the two sites?
Will the Scandinavian style of working translate to other sites?
How should they organise transplantation of product team/responsibility/activity from one site to another?
How should they manage transition?
Is cultural distance a serious issue or just a background characteristic or something else?
Are other sites viable?
Why Dublin?
Is this a near-shore sourcing arrangement?
The transfer seems to have high operation risk. Is this a fair statement?
What is the benefit of going from Scan to Dub
The transfer has low/moderate structural risk - non-codifiable process but same organisational family.
Are these really in fact complex and problematic business process? Or not? Justify...
Does identity affiliation and professional standards make transfer easier?
What skills, attitudes, structural, cultural aspects work to enable the process?
Problem of practices (culture 500 pgs) use radical exasketch?
Knowledge transfer problem may be overcome through tacit transfer occurring through interchanging team members.
The differences between Dub/Scan aren't between right/wrong; both work well in their context.
Does code need people?
Is the process model excessive?
Should the company consider open source approach?
Is the painting by numbers approach a misconception or a brilliant method for deconstructing difficult technical work?