|Previous page||Table of Contents||Next page|
Professor of Software Systems Engineering
University College London
|2 Terms of Reference||157|
|7 Outline of Project||160|
|10 Living with CAPSA||171|
1.1 Implementing a financial information system in a large and complex organisation is a notoriously difficult and risk prone process. It requires a delicate collaboration in which resources, technical expertise, project management, organisational reengineering and change management must be fused into a harmonious whole. Even with total organisational commitment and using best practice this is not always achievable.
1.2 In the case of the implementation of CAPSA the University did not display the requisite whole hearted commitment. Basic good practice was not adhered to by the University, and in many cases by the external consultants it engaged. No appropriate managerial safeguards were in place. By these means a necessarily risky proposition was transformed into a surefire failure.
1.3 Most of the lessons from CAPSA are simple and it is well within the capacity of the University to learn and act on them. The lessons relate to clarity of responsibilities; well organised project management and technical processes; properly qualified and trained staff; unswerving strategic focus; careful control of external consultants; in depth scrutiny of project proposals.
1.4 CAPSA has caused much pain and inconvenience. It has cost a lot of money, damaged the integrity of the University's financial processes and soured relationships between academics and administration. Its operational successor, CUFS, is likely to continue to do so for some time. If, however, appropriate resources are devoted to it this should be on a decreasing basis.
2.1 This report has been prepared in response to formal terms of reference supplied by the Audit Committee and approved by Council. These terms of reference are reproduced in Appendix I.
2.2 This report covers Part A of the review of CAPSA - the 'University's new on-line accounting system project' and its implementation, that is the 'technical' component of the review. It provides an account of the conduct of the project. It looks at key decisions and the processes by which these were reached. The report reaches beyond the scope of Part A in making some comment on how the problems with CAPSA arose and making recommendations for the technical and managerial conduct of future large software-intensive system projects within the University.
2.3 The report is intended to cover 'CAPSA' to date, that is to the submission of the report. Its scope goes therefore beyond CAPSA covering CUFS (Cambridge University Finance System) the operational system resulting from the CAPSA project and to a certain extent CAPSA's predecessors. CAPSA and its implementation, the title of the report, is thus a convenient shorthand and I will often use the term CAPSA in an embracing sense.
2.4 The report should be read in conjunction with an associated report, prepared by Prof. Michael Shattock, that examines the interplay between the broader systems of University management, policy making and governance, and the conduct of the project.
2.5 Though this report is, in the first instance, addressed to the Audit Committee, the Board of Scrutiny have also played an important role in seeking a review. The report is intended to inform their discussions and act as a basis for their report to the Regent House. Many of the problems encountered by the University of Cambridge and many of the mistakes made are, I suspect, common to the Higher Education sector as a whole and perhaps more general lessons can be learned.
3.1 This report has a very simple structure. The method by which its findings have been reached is briefly outlined (Sect 4). An account is given of assumptions which have shaped the review and underpin the analysis (Sect 5). A background section is given (Sect 6). This sets the context for the outline of the project (Sect 7). The outline of the project includes a, necessarily condensed, narrative of processes and decision making, and a discussion of the operation and use of the system. Section 8 presents an analysis that attempts to pick out 'where things went wrong' and to identify their principal causes. The recommendations (Sect 9) indicate the steps the University should take in order to reduce the risk of similar projects having similar outcomes to those of CAPSA. The report concludes with a discussion of the current state of CAPSA and considers its future (Sect 10).
3.2 The proper place to start a report of this kind is with an account of why it was necessary in the first place. In other words why the Audit Committee and Council felt that such a review should take place. I state these reasons baldly.
That CAPSA cost significantly more than had initially been envisaged.
That the implementation of CAPSA resulted in significant disruption to the working of the University.
That individual staff of the University were placed under intolerable pressure and were unduly burdened by the need to use CAPSA and manage financial transactions in the context of CAPSA.
That staff charged with, and responsible for, the management of finance within the University were unable to fulfil those responsibilities as a result of CAPSA.
That the integrity of the University's accounting and hence its legal responsibilities were placed at significant risk.
That, in respect of significant areas of functionality, CAPSA fails to meet the requirements that the University has of it.
That CAPSA is of low quality, being both unreliable and difficult to use.
That the loss of key staff, difficulties with external consultants, evident low morale, and lack of coordination among those directly involved with CAPSA are strongly suggestive of managerial failings.
That key 'stakeholders', including key administrative and clerical staff, Heads of Departments, managers of important research units are strongly dissatisfied and are, or are threatening to, opt out of control by CAPSA.
That the University wishes to engage in further projects of a similar nature, student records being the most immediate and most strategic in scope, and is unable to do so until the failings of CAPSA and its implementation have been properly addressed.
3.3 In each case there is support for these concerns justifying the requirement for a thorough review. This support is both in evidence gathered during the course of the review, see below, and plainly made manifest in the proceedings of the Regent House.
4.1 There is no established practice for reviews of this form and there are relatively few models which can be followed. The methodology for conducting such a review is essentially a qualitative one. It rests on careful examination of source materials, in depth interviews with eye witnesses or key informants and critical analysis. Central to such a methodology is 'triangulation', that is seeking to establish a position with respect to other reference points.
4.2 During the course of the review I have undertaken a detailed examination of many hundreds of documents. Some of these documents are maintained in paper archives maintained by Finance Division at The Old Schools and the implementation team of the Management Information Services Division (MISD) at their offices in the Royal Greenwich Observatory building (RGO). These archives, though incomplete, have provided the backbone to the review. MISD also maintains an electronic archive which I have had access to. A further electronic archive of documents was supplied by an external consultant from their project records. Where crucial documents were missing many of them have been supplied by individuals, though it has not always proved possible to follow complete document trails.
4.3 Invaluable support to the review has been provided by the contributions of individual members of the University, and others, in response to a published invitation that appeared in the Cambridge University Reporter on 31st May 2001. These contributions provided evidence, opened up new lines of enquiry, made recommendations, and, crucially, indicated the strength of feeling which many of the issues discussed below have engendered.
4.4 I have conducted in depth interviews with almost all the key players engaged in CAPSA and its implementation. Some of these interviews have been conducted by the joint review team. Some interviewees have been seen on several occasions. Of particular importance have been interviews with external organisations and professional advisers, many of whom are no longer in a contractual relationship with the University. Similarly interviews have been conducted with former employees to whom I am particularly grateful for their time and forbearance. While the principal outcome of the review is this report, the interviews themselves have I believe been beneficial. In many cases they have provided an opportunity for the protagonists to step back, reflect critically on the events discussed below, and to learn from them.
4.5 Some other institutions within the higher education sector with direct experience relating to that of Cambridge have also been seen by the review team. This has provided useful confirmation of much learned in the review, however it is to be clearly understood that the immediate findings of the review are entirely restricted to Cambridge University. I have examined other analogous projects within the University to lend further support to the analysis.
4.6 In the report I have not attributed quotations to individual interviewees without specific permission and have undertaken to maintain confidentiality with respect to comments about individuals. Almost all the individuals and organisations giving evidence to the review have given permission for their names to be listed in Appendix II.
4.7 Since much of the analysis in the report draws on established engineering good practice, it is not necessary in most cases to provide additional justification or support. A useful prompt, and a helpful analytical tool, in preparing these recommendations have been the normative models of good practice provided by 'capability-maturity' models and associated international standards.
4.8 The report concentrates heavily on processes and procedures. In the time available to this review it has not always been possible to revisit particular technical decisions. Where a decision has been made, and there is evidence that due process has been followed, appropriate expertise deployed and sound technical evidence marshalled, I have not sought to subject it to further analysis.
5.1 This report takes a 'systems' perspective. This needs little explanation to the engineer but requires a brief description for the lay reader. A system is an organised whole comprised of a set of interrelated components. CAPSA is a system whose components are other systems. These systems include software and computing hardware - but are not restricted to it. The CAPSA system includes for example the procedures by which the software is used, the organisation that provides ongoing support, the materials for training, the plans for disaster recovery, and so on. It may even, in a broad interpretation, include the users of the computing system. The CAPSA system resides within a management information system, again not to be interpreted as purely a technical system, and ultimately within the University, a system in its own right.
5.2 The implications of taking this view are straightforward. In implementing a 'system' the responsibility is to deliver the whole, the combination of parts, people, procedures and technology, set in such relationship to each other that the goals of the system are achieved.
5.3 In his seminal account of technological failure [Perrow, 19851] Charles Perrow writes 'systems fail systemically'. What may at first appear a truism provides a useful insight, it is usually difficult to ascribe the failure of a complex system to a single root, rather it is the interactions between components of the system and the complex relationships between the parts that lead to failure. Systems fail as systems, CAPSA is no exception.
5.4 Much of the discussion which has surrounded CAPSA has revolved around who 'owned' the system. A debate of this form evidences a lack of systems thinking. In a large organisation with a complex mission, critical systems are not owned by individuals or even groups. It is best to think of a system as addressing a large and dynamic set of stakeholders with interlocking, and occasionally conflicting, objectives. All the stakeholders have requirements of the system and have responsibilities for aspects of its operation. Slurring the casual distinction between provider and user, individuals can be both part of the system and a stakeholder for it. This is, it should be emphasised, not to argue that the implementation of a complex system does not require coherent management control or that financial decision making should not be carefully organised but rather to emphasise that responsibility and ownership are ultimately shared.
6.1 It is assumed that the readership of this report is largely within the University of Cambridge and that it is unnecessary to enter into a review of its very complex and, during the period under examination, fluid governance and management structures. Some of this is in any case examined in the associated report by Prof. Michael Shattock. Nevertheless some broader points need to be made.
6.2 The higher education sector, like others in receipt of substantial government funding, has been subject to increasing scrutiny. This scrutiny extends beyond the assessment processes, such as research assessment, with which all academics are familiar, and within which Cambridge excels. There is a strong demand for higher standards of financial accountability and 'transparency'. Such demands are only set to increase.
6.3 The University of Cambridge has a strongly decentralised system of financial management. This system is not unique within the sector, though the scale, diversity in size and nature of the constituent units, and the organisational structures through which it operates mean that the system is particularly complex and unwieldy. The decentralised system is valued by all concerned, fits the 'culture' of the institution and makes a strong and positive contribution to its governance. The responsibility for ensuring accountability and transparency within this system rests with central administration, particularly with the Finance Division, and ultimately with the Principal Officers of the University. In short the viability of federated management and decentralised finance are critically dependent on a strong and effective central administration and finance function.
6.4 Though it is difficult to establish an accurate basis for comparability, the central administrative functions in Cambridge are very 'lean'. As the account below will show the Finance Division and MISD were significantly under resourced to deal with the challenges that the implementation of a new accounting system entailed. Further, salary scales and appointment procedures meant that staff retention and recruitment were extremely difficult. It is evident that not only were the critical areas understaffed throughout the period but that staff, including those appointed to key posts, were given tasks which they were insufficiently qualified or experienced to handle.
6.5 In a decentralised management setting significant responsibility devolves upon the constituent units. In Cambridge there are a great variety of these though obviously Departments under various guises are the majority. Within these units finance is the responsibility of independently appointed clerical and administrative staff owing their primary allegiance to the academic structures within which they work. Working on the review I have met many of these staff. Doing so I have been unfailingly impressed by their dedication to the University and their sympathetic understanding of the work of the academics with whom they are associated. They undoubtedly contribute much to the institutional success of Cambridge. It is clear however, that many of these staff lack the professional training and experience to cope with the necessarily increased levels of complexity which modern accounting systems demand. There is a readily discernible air of 'amateurism' about Departmental arrangements in respect of finance further compounded by heavy workloads and stretched resources at all levels. As the narrative below will show, this played an important part in the problems encountered with CAPSA and its implementation.
6.6 In setting the context it is necessary to comment on a particular and striking feature of institutional 'culture'. Many of the discussions with members of the University have been strongly coloured by the opposition between 'them' - 'the Old Schools', that is the University central administration and 'us' - the Departments. Though this is not an unfamiliar feature of University life elsewhere, its vehemence and prevalence are a surprise. It is certainly shored up by the tone and accusatory style which characterises some University debate. This issue is more fully discussed in the associated report and is briefly covered again in the recommendations below. I believe the mistrust that these attitudes have engendered has contributed to the problems encountered with CAPSA and its implementation.
7.1 In order to set CAPSA and its implementation in its proper context. It is necessary to take a long view. The University obtained its first computer-based accounting system in 1966. This system comprised bespoke software written in COBOL and using flat file storage for data. This system remained operational, with the usual maintenance and some minimal evolution in functionality until June 1993 when it was decommissioned. From inception and through its life the system was managed by Mr John Bray who held the post of Head of the Administrative Computing Unit (ACU).
7.2 By at least 1990 it was apparent that the original bespoke system was aging and that maintaining both the software and the hardware platform it required was increasingly costly and difficult. In May 1993 a new system went live, intended to replace it. During the first months of the operation of this new system its predecessor was run in parallel as a precaution. Perceptions differ but in general terms the transition to the new accounting system is regarded as having been relatively painless.
7.3 The new system was substantially different in terms of technology. It was built on top of an Oracle relational database management system and, though much of the application software remained bespoke, significant parts were based on components provided by external software suppliers, Financial Solutions and Crown Systems. These differences masked the fact that, at the insistence of Finance Division the system cloned the 'workflows', accounting procedures, and 'data structures', financial information, of the original system. In particular, though off-the-shelf-components were available, at relatively low cost, to provide commitment accounting no move in this direction was made.
7.4 From May 1993 onwards this system, now commonly referred to as Legacy, was subject to continuous evolution. In particular, in 1995 there was a substantial reengineering of much of the application software. Changes in functionality were also introduced. In August 1997 commodity codes were introduced and in November 1997 a budget facility was provided. These changes attracted considerable controversy and proved difficult to introduce. Legacy remained in operation until August 2000, CAPSA go-live.
7.5 By early 1995, if not before, an understanding had built up, shared among both academics and administrators, that a unified approach to support for commitment accounting was necessary. This was reinforced by the recruitment to Finance Division, at a relatively junior level, of a professional management accountant, Mr Chris Lang. In late 1995 an investigation group - the Commitment Accounting Working Group (CAWG) - was established to look into commitment accounting. The group was led by Mr Lang and included representatives from Internal Audit, Purchasing and the ACU.
7.6 By late 1996 a preferred route had been identified by the CAWG which entailed the in-house development of a lightweight front-end system based on Microsoft Access. It was to be loosely modelled on Hyperdata, an accounting system used in some large Departments that provides basic support for commitment accounting. The system proposed would synchronise, in a manner not clearly specified, with the Legacy system. Initial proposals included plans for a highly incremental enhancement of the system starting with limited reporting and a very simple bridge between the legacy back-end and the new local front-end.
7.7 Using funding from the Allocations Committee an external consulting organisation, Tate Bramald, was engaged to complete a study of the requirements for, and the feasibility of, a 'standardised' commitment accounting system. They were asked in their report to place particular emphasis on the preferred route that CAWG had identified. Tate Bramald are an established supplier of finance system software with a significant presence in the voluntary sector and an expertise in supporting medium sized Microsoft based accounting solutions. They were also a potential supplier of components to the proposed system. The Tate Bramald report was completed at the end of 1996. It is relatively brief and in essence amounts to unqualified support for the CAWG preferred system proposal.
7.8 In early 1997 a formal proposal was made to the Resources Committee and to the Joint Committee on Central Administrative Computing, the committee charged at that time with oversight of the ACU. This proposal covers the justification for a commitment accounting system, an outline of the preferred system proposal, a costing (£377,000) and a cost-benefit analysis.
7.9 The proposal, prepared and fronted by Mr Lang, was approved and by mid 1997 two additional analyst programmers were recruited to the ACU and charged, together with some existing staff members drawn from the Legacy team, with the development of the new system which I shall call Local for convenience.
7.10 At approximately the same time, that is mid-1997, members of Finance and the ACU attended a sales pitch for SAP/R3, a large enterprise resource planning system, prepared by Apex Systems a supplier of corporate software solutions. Though this is not directly germane to the narrative, it suggests at this early point some waning of the enthusiasm for Local.
7.11 Work on Local however proceeded through the second half of 1997 and at some stage the name CAPSA was adopted. A capsa is a Roman book storage box, the name has the merit of including the initial letters CA for commitment accounting, embracing an appropriate metaphor and being suitably arcane for a University audience. Tate Bramald were engaged to provide the commitment modules of the system.
7.12 By late 1997 sufficient confidence in the development of Local had been obtained and a 'CAPSA User Group Meeting', with the principal objective of engaging the users in the process, was held. Early in 1998 a further meeting to expose progress and to obtain feedback from Departmental Computing Officers (DCOs) was held. This meeting with the DCOs proved to be critical to the project and indeed to the entire history of the development of CAPSA.
7.13 In essence Local was subjected to sustained technical criticism. This criticism exposed a number of key issues that had remained unresolved since its inception. These were primarily, though not exclusively, concerned with non-functional requirements such as performance, security, connectivity and platform constraints which are central to the architecture and operation of a distributed system of the type envisaged.
7.14 The criticisms were a significant blow to the project team and to their 'sponsors' within Finance and added to an underlying, though unexpressed, worry that Local, an essentially in-house solution, might be unable to satisfy the need for external transparency of accounting information and processes. Following on from the meeting an internal ACU review of Local was convened which raised further doubts, calling special attention to the low involvement of the prospective users. With confidence at a low ebb it was decided to seek an external review of Local.
7.15 Details of the appointment of the consultant who was to review Local have proved difficult to obtain, but some partial picture can be pieced together. At the suggestion of Mr Lang, and through a 3rd Party recruitment agency, Atlas/Barley Fisher, with which he had contact, Ms Sue Baron a Director of Bayles Consultancy was approached.
7.16 Bayles Consultancy is a small consultancy offering business systems consulting and project management services. It works with a number of contract consultants and is developing an overseas business in system development outsourcing. It is based in Newmarket, Suffolk. Ms Baron is a Director and works as senior consultant for the firm. In offering herself for the work Ms Baron cited relevant qualifications in both computing and finance as well as significant relevant business and financial systems experience, both as a consultant and a senior corporate manager. At that time she had no experience in the higher education sector.
7.17 Ms Baron was interviewed in March 1998 by a panel chaired by Mr Lang, supported by Mr Dave Sandham, a representative of ACU, and Mr Lloyd Retallick, a systems developer from within ACU who had led technical work on both Legacy and Local. Subsequent to this interview she was engaged with a brief to review the Local project. It has not been possible to track down the formal details of the appointment but it is safe to assume that it was subject to the standard Supply of Services Agreement issued by Bayles Consultancy and that it required the Local statement of requirements and the technical architecture to be examined in detail. Ms Baron was to report to Mr Lang.
7.18 It is unclear to what extent those principally involved expected Ms Baron to 'rubber stamp' the Local proposal, albeit with some minor amendments. In any case this was not the outcome. Immediately after being appointed Ms Baron set to work with considerable energy. During the period April/May 1998 a very substantial requirements gathering effort was undertaken. A questionnaire was issued, opening a communication line with many key stakeholders in Departments and Schools. This was supported by a large number of face-to-face meetings with staff in Departments and Schools intended to establish working practices.
7.19 At the end of May 1998 an interim report covering the work to date was issued by Ms Baron and presented to the Treasurer, Mrs Joanna Womack. The Treasurer was also, at that time, in overall charge of ACU. The interim report was written from the standpoint of the Departments and Schools. It strongly suggested that Local would be unable to meet the requirements as determined up to that point.
7.20 It is not surprising that the Treasurer reacted with shock and surprise to the interim report. Up to this point she had minimal involvement with the development of Local, having delegated the responsibility to Mr Lang, and was unaware of the difficulties that Local had encountered. Nevertheless, further work was authorised to evaluate 'third party products' and to examine alternative solutions to Local. This can be seen as a major milestone in the history of CAPSA. It marks the point at which Local ceased to be the preferred solution and joined a range of contesting options.
7.21 Immediately following the presentation of the interim report, at the beginning of June 1998 work on Local was 'suspended', presumably pending the results of the further work, in fact never to resume. At approximately this time a CAPSA Management Committee was formed. It consisted of representatives of ACU and other members of Finance Division. Nominally chaired by the Deputy Treasurer it was in effect chaired by Mr Lang.
7.22 During June, and through much of July, work by Mr Lang and Ms Baron continued. The offerings of major software suppliers in the enterprise resource planning sector - SAP, Oracle and Peoplesoft - were subject to a preliminary analysis, and other contending industrial solutions considered. Discussions were initiated with other institutions in the higher education sector using these systems.
7.23 By late June 1998 a further draft report critical of Local, of the handling of the project and of ACU, was issued internally. This report with some excisions and minor additions was to be issued in July 1998 to the Joint Committee on Administrative Computing.
7.24 The report recommended that the University should acquire an integrated, off-the-shelf 'industry-standard' accounting package. It concluded that Local, characterised as essentially patching up the old system would not meet the requirements of audit nor satisfy the pressing demands for improved financial transparency. On the basis of this report a request was made to Finance Committee for funding to support the development of a 'functional and technical specification'. This request was approved though work continued on providing further costing information.
7.25 The approval marks again a new phase in the project, now 'CAPSA-proper'. Through the summer of 1998, for a period stretching to October 1998, work proceeded with the clear objective of acquiring a suitable software package. The work included the formation of requirements 'working groups', notably a technical working group of DCOs. These groups acted as fora for particular interests and provided a further substantial input into the requirements. In addition a phased implementation plan was sketched that provided for a gradual roll out of a solution across the University with introduction of new functionality being also phased.
7.26 It should be noted that an undercurrent of disquiet could already be discerned suggesting that insufficient attention had been paid to the central finance requirements on the new system. Further, somewhat inconclusive work to address this was launched.
7.27 As a first step in selection the extensive list of potential suppliers identified earlier in the process was boiled down to a smaller set of key suppliers. These key suppliers, that is SAP and Oracle, provided demonstrations of their offerings within the University. Peoplesoft, previously regarded as a possible supplier, opted not to participate in the selection process for reasons which remain disputed and I have, in the course of the review, been unable to establish firmly. These demonstrations, including large scale public presentations, were attended by a wide range of University staff. In essence the demonstrations were structured 'sales pitches' by the main contenders, with 'canned' workflows and user dialogues drawn from the standard business use of the systems.
7.28 To support the demonstrations, visits by members of key working groups, were made to reference sites offered by SAP and Oracle. None of the visits were successful. In the case of SAP because no truly comparable reference appeared to be available. A 48 hour visit to see Oracle Financials at California Institute of Technology (CalTech) was arranged. As the system was not actually fully operational little was learned from the visit. Members of the working party were however struck by the vast effort devoted to the business process mapping activity being undertaken by CalTech staff and by warnings about the research grants module. Unfortunately these critical observations got lost in the jet-lagged confusion of the visit. A short visit was also made to Imperial College who were undertaking a similar selection exercise to that of Cambridge.
7.29 An important step in the proposed process was for a direct test of both SAP and Oracle's solutions to take place in selected Departments. It was conceived that both suppliers would offer 'generic' systems making available the core functionality, and that these would be tested with Departmental data. Oracle were to provide a system to the Department of Engineering and The Fitzwilliam Museum while SAP would provide a system to the Department of Pathology and the Programme for Industry Office. It is clear that neither SAP nor Oracle provided a demonstration system of the type envisaged. Oracle did give access to a remote system and staff in the Departments were able to see detailed demonstrations though not as far as I can determine with extensive University data. SAP, not unexpectedly, balked at the significant effort that providing a test system would entail and offered further demonstrations of industrial implementations of the SAP suite. Surprisingly the Departmental users of both systems gave the OK and both Oracle Financials and SAP moved to the next selection stage.
7.30 As the selection process proceeded both SAP and Oracle were asked to review the requirements and provide an undertaking that they could support them. Both provided a 'line-by-line' match between the processes supported by their system and those implied by the requirements. On this basis the suppliers were asked to quote for supplying a system to correspond with their mapping of the requirements.
7.31 By the end of October 1998 a lengthy report setting out the requirements and a functional specification, directly corresponding to those requirements, as well as providing a standard weighted checklist assessment of the competing solutions, was submitted to the Joint Committee on Central Administrative Computing (JCAC). JCAC were the committee charged at that time with the oversight of ACU. This report came down in favour of Oracle Financials. Elements which appeared to weigh heavily in the selection process were Oracle's more open and positive attitude and their commitment to the education market specifically demonstrated by the Research Grants module. Oracle had in any case provided substantially the lower quotation. JCAC recommended: a further 3-6 month investigation with a view to detailing the specification in certain areas; that Oracle be nominated preferred supplier; that an implementation date - for initial roll out - of August 1st 2000 be targeted (a step back from the more adventurous phased implementation plan which had some roll out in summer 1999); and, that 'senior members of the Unified Administration should be closely involved in determining and monitoring the management issues arising out of this project'.
7.32 It is somewhat difficult to untangle the precise sequence of the ensuing events and it is probably not necessary to do so. Based on the presumption that Oracle Financials was to be chosen, a hardware selection exercise was undertaken and the use of a Sun platform recommended. Some significant work was devoted to identifying the infrastructure requirements that running Oracle Financials would entail, though it should be noted that these primarily referred to 'server-side' (that is service provider) requirements rather than 'client-side' (user) requirements.
7.33 In December 1998 the Planning and Resources Committee allocated £4.3m to the project though this was to be subsequently revised as, inexplicably, VAT liability had not been correctly determined. Appendix III gives a spreadsheet and associated notes summarising expenditure on CAPSA kindly prepared by Mr John Bray. It is the most accurate available. Some additional notes are also included covering other items of expenditure though these have not been checked and cannot be relied upon.
7.34 At around this time there was a significant hiccup in the process. SAP were unhappy at not being the preferred supplier, a view they expressed with considerable force. They observed that the procurement process breached EU regulations, which might have left the University open to legal challenge. As a result the process was rerun with repeat tenders considered under a framework agreement managed by a 3rd Party procurement agency. The result of this process was the same, with Oracle as the selected supplier, though clearly this reflected poorly on the University and resulted in additional cost.
7.35 The beginning of January 1999 saw the arrival of Mr Nick Robinson, recruited from a middle management position in the property services industry, to head a new Management Information Services Division (MISD). MISD was to take over the functions of the ACU and was charged with responsibility for taking a 'strategic' view of the University's management information requirements. Within the unified administrative structure MISD reported to the Registrary.
7.36 At the time of the appointment of the Director the most significant 'strategic' item of work for MISD was the CAPSA project. It was expected by the Principal Officers of the University that the active management of the CAPSA project from the University standpoint would be undertaken by him. In retrospect it is less clear whether the new appointee understood this to be the case or, if he did, fully grasped the consequences.
7.37 With scarcely a pause for breath to mark this important change, CAPSA rolled on. The outline implementation plan was revised and further work was undertaken by Oracle mapping their product to the functional specification. By the end of February 1999 contractual exchange was underway.
7.38 Coincidentally, also at the end of February 1999 the JCAC held its last meeting and in early March 1999 was formally dissolved by Finance Committee. Oversight of the CAPSA project was then vested in a newly appointed CAPSA Steering Committee (CAPSA-SC) reporting to the Principal Officers meeting and through the Treasurer to Finance Committee. The CAPSA-SC was chaired by Prof. Malcolm Longair, a senior academic and highly distinguished scientist. The other members included the Treasurer, Director of MISD, CAPSA project manager, representatives of the acting internal auditors - Robson Rhodes, a senior representative of Oracle, and assorted Faculty representatives with a known interest in accounting issues. The CAPSA-SC held approval rights over items of expenditure in excess of £50,000.
7.39 The beginning of March 1999 is thus the start of a new phase in the CAPSA project - implementation. Ms Baron continued, under an extension of the previous arrangements with Bayles Consultancy, to manage this part of the process.
7.40 Oracle have a well established process for supporting large scale implementation projects. This process is embedded in the Application Implementation Methodology (AIM). In essence AIM provides templates for high level planning and management, generic workflow definitions, documentation and other standards, as well as testing frameworks. It was decided to use this methodology as the main 'technical' driver of the process. In conjunction with AIM, it was intended that Prince (PRojects IN a Controlled Environment), a UK Government standard for IT project management, be adhered to and that the project be managed in conformance with the Prince 2 project management method.
7.41 An initial CAPSA team was recruited consisting of 5 staff from MISD and between 8 and 12 seconded from other University Departments on a full or part-time basis. This team was supplemented by consultants, of which more below. It is notable that the project records of staffing of the CAPSA project by internal and contracted staff are incomplete, though from the fragmentary picture I have been able to piece together it is very evident that the staff commitment on the part of Finance Division was negligible.
7.42 To compound this, in February 1999, a Special Early Retirement Scheme was announced. Under this scheme highly favourable terms were offered to longstanding University employees. A number of key Finance Division staff indicated that they wished to take advantage of the scheme. This included Mr David Whiting the Principal Assistant Treasurer, who is widely considered to have had the most significant expertise in the operation of Legacy and associated accounting procedures. Other losses, notably that of the Deputy Treasurer, Mr Peter Mardles, temporarily weakened the overall operating capacity of Finance Division, already understaffed, with knock-on consequences for CAPSA. These early retirements were effective from the end of July 1999. The resignation of Ms Ann Searle who had provided much of the Finance Division input into the early stages of acquisition further worsened the situation.
7.43 Coinciding with the start of implementation the CAPSA team moved to the Royal Greenwich Observatory building (RGO), a new building some way from both MISD and the Finance Division. As might be expected, moving a nascent team into office space without computers, network, telephone lines, provision for housing equipment etc. was not an ideal foundation for the strong start the project needed. Despite heroic efforts by some MISD staff, the project lost momentum and morale before a beginning had even been made. It took nearly 5 months before the premises were fully operational.
7.44 In addition to staff from within the University, the project saw a large variety of consultancy staff occupied for varying amounts of time on CAPSA. This was, as will be seen below, a continuing feature of the conduct of the project. At this stage, the Bayles Consultancy team was supplemented by staff closely associated with Ms Baron to handle project secretarial services and some of the day to day routine of project management. Oracle provided an implementation team consisting of Oracle project management and technical staff including 'functional specialists' with a detailed knowledge of particular Oracle Financials modules. The team was somewhat different to that which had hitherto led Oracle's engagement with the University. Over the next few months additional technical consultants were engaged by Ms Baron to cover technical shortfalls. These were mostly independent contractors with experience of Oracle implementations.
7.45 It seems clear that personal relations between all the parties involved, Bayles Consultancy, Oracle staff, seconded University staff and external consultants declined, slowly at first,and then with increasing rapidity, from the beginning of implementation. A failure to resolve issues of managerial responsibility and control led to conflicts between Oracle project management and Bayles Consultancy. This in turn contributed to a nose dive in morale which was in any case not high to start with. Seconded University staff encountered a management style and working environment significantly different from that with which they were familiar.
7.46 Technical progress on CAPSA from this point till June 1999 was sporadic. While a number of technical issues were worked on, notably system interfaces, arrangements for parallel running, stock coding etc. it is difficult from the documentation to gain any sense of orderly progress. Existing documentation is incomplete, most documents are in draft versions, and so far as I can judge of generally poor technical quality.
7.47 What does stand out is the emerging realisation that the Chart of Accounts, the account coding scheme on which a general ledger is based and the most basic building block of any accounting system, would require revision. It had been an assumption, on which the implementation plan was based, that the Chart of Accounts would be unchanged in the new system. This assumption was false, a fact which had severe knock-on consequences, technically, managerially and organisationally.
7.48 To tackle the issue, in or around May 1999, an Accounting Policy sub-committee of the CAPSA-SC (also called Accounting Policy Working Group - APWG) was established, chaired by Mr Richard Collet-Fenson, Head of Accounts in the Department of Engineering and influential in the early testing of the Oracle solution, the representative of the General Board on the CAPSA-SC. This group was charged with identifying and determining accounting policy issues of relevance to CAPSA, subject to Finance Committee approval.
7.49 While the technical progress of the project was poor; some project management discipline was maintained. More open to question is the precise relationship between these plans and the work being actually carried out. Though in theory any significant items of expenditure were controlled by Mr Robinson in his capacity as Director of MISD in reality the resources were under the control of the project manager. Project 'burn', the rate at which resources were being consumed by the project, was only loosely controlled through a control of consultant commitment. The records of the CAPSA-SC shed very little light on the progress of the project during this stage.
7.50 In what is, by any standards, a complex and eventful history a further milestone is reached with the arrival at the end of June 1999 of Ms Adrienne Rudkin, appointed to the new position of Director of Finance. Ms Rudkin brought to this position experience in University finance from outside Cambridge. Realising that CAPSA was central to the operation of Finance Division and hence to her role she took immediate steps to establish the state of CAPSA. Acting with some speed, coupled with considerable energy and determination, she built a schedule of the information required to do this. When the information was not immediately forthcoming she actively intervened, working with the Director of MISD, Oracle and others to try and get an accurate picture of the project and its progress.
7.51 The Director of MISD was, it also appears, unhappy with progress, in particular with the possibility of a significant overspend. In June 1999 KPMG, the external auditors, were asked, primarily at his instigation, to review the project control systems. Their report was highly critical and its criticisms extended beyond the immediate project control issues, covering the lack of Finance buy-in and the flawed assumption concerning the Chart of Accounts. In short the report clearly identified both failing project management and, on the part of the University, serious errors compounded by lack of managerial and technical engagement with the project. It may be surmised that the incoming Director of Finance was in sympathy with these conclusions giving considerable extra force to her intervention.
7.52 By mid-August it was clear that the position of the Project Manager was untenable and that the Directors of MISD and Finance were unsatisfied with the further information they had received. In a joint report the Directors highlighted problems with: record keeping; communications; management style; contract consultants; clarity of deliverables; financial monitoring; and decisions about accounting policy (that is the replacement of the Chart of Accounts).
7.53 Ms Baron resigned and her engagement was to be concluded with a final report summarising the current state of the project. It should be noted that the handling of the resignation on the part of the University, the delicate business of ensuring the integrity of project records, continuity in the management of the project and maintaining morale, was dealt with in a cack handed manner by those immediately concerned. The final report sheds little light on project progress but does illuminate the considerable difficulties which the lack of Finance involvement had entailed.
7.54 With the position of project manager effectively vacant, Mr Dave Sandham, the senior MISD staff member associated with the project and at that time responsible for dealing with the hardware and infrastructure issues, took over. The project entered a period of stasis. Project burn, however, continued as expensive external consultants twiddled their thumbs. This was not to last too long before further external consultants entered the frame.
7.55 The Directors of Finance and MISD, with support from both the Treasurer and the Registrary, determined to appoint the KPMG Oracle practice to manage the project. KPMG were experienced system implementors with a well established technology partnership with Oracle and considerable exposure in Higher Education. They had connections to the University through the audit practice and this connection may well have weighed heavily in decision making. Above all KPMG were large and reputable, a 'safe bet' for a troubled project.
7.56 The KPMG team were in place by early September. The key members of that team were Ms Beverly Bryant acting as project manager, Mr Simon Street as technical manager and Ms Amanda Konishi as change manager. The reporting structure remained, in principle, unchanged that is to the Director of MISD and the CAPSA-SC. In reality however the managerial structure changed significantly. The Directors of MISD and Finance together with the KPMG project manager acted as a managerial troika, meeting regularly and making decisions jointly. The 'customer' or principal stakeholder to which the project addressed itself was the APWG, within which the Director of Finance took an increasing role, eventually chairing.
7.57 Ms Bryant acted rapidly, dismissing all the external technical consultants to reduce the burn. The project team were reorganised and a firm and directive management regime established. An effective project control system was established. It might have appeared that CAPSA was, after its wobble, on the way to success.
7.58 All the key players had firmly in their minds the 'absolute necessity' of delivering the system by August 2000, a date which had by this point become fixed. How precisely this date became so entrenched is difficult to pin down, certainly the commitments to Finance Committee and CAPSA-SC as well as requests by the Audit Committee played a role. The recovery of VAT, a sum of some £4m being said to be outstanding, also loomed large in the minds of those involved.
7.59 To deliver on this schedule some tough decisions had to be made. Ms Bryant, with support from the Directors of MISD and Finance and as appropriate from APWG, therefore set aside the phased roll-out, opting instead for a 'big-bang' delivery. An axe was taken to the large number of external system interfaces that had been projected. The web user interface and lightweight access for occasional users which had formed a large part of the Departmental requirements for the system was also dropped.
7.60 Work continued on the part of APWG to deliver a revised Chart of Accounts, though within its wider remit as surrogate customer it was called on to make other technical decisions. This task was complicated by an inability to determine the impact of those decisions, either on the requirements or on the technical progress of the project.
7.61 In November 1999, with a view to further simplifying the task that lay ahead, the KPMG team proposed that the University grouped, for accounting purposes, smaller units together under the umbrella of School offices. They also suggested setting money aside to cover the costs of CAPSA implementation likely to fall on Departments and Faculties. There is no evidence that this message was heeded.
7.62 There is little to say about technical progress of the project. The pace of work was rapid and pressure high, both within the CAPSA team and in Finance. The evidence of the work is embedded in a large slew of detailed documentation covering the configuration of Oracle Financials. Despite this progress it was becoming obvious that the timescales were very compressed, and without a phased plan there was no way of relieving pressure other than trimming functionality. The first evidence of disquiet can be determined in the proceedings of the CAPSA-SC.
7.63 By the end of 1999 rumours were already circulating within the University about progress on CAPSA. In late January 2000 these rumours gained common currency through the CAPSA roadshow. The roadshow, intended to demonstrate CAPSA functionality and capture interest seems to have had precisely the reverse effect. To a large audience of junior staff and many of the influential Departmental Computing Officers, who would be required to ensure that the 'client-side' of CAPSA operated, a static set of screendumps was presented, few questions were answered. Most attendees left with a strong sense of disquiet, many commented on the absence of the senior University staff who might have been able to lend credibility to the project.
7.64 Through February 2000 and onwards, work continued on CAPSA though a definite change in emphasis can be discerned. Work on configuring Oracle Financials was throwing up bugs, both minor and major. An ever increasing proportion of time was required to identify these bugs, track them from the project side and apply patches. Much of the technical expertise from the Oracle consultants was being sucked into bug fixing or its immediate consequences. Though Oracle devoted substantial effort and resources to the problem, from at least February till May 2000 the overall volume of unresolved bugs (subject to the precise method of categorisation) grew. More seriously the ability of the CAPSA team to deal with this became stretched, so that patches were dropped or misapplied and the sense of a positive trajectory so vital to an effective software project was lost.
7.65 To cope with the debugging and its knock on consequences, the schedule slipped. Most notably the testing phase became more and more compressed. Without a stable set of baselines across the project it became almost impossible to apply effective integration testing, and the rigorous regression testing that a changing software system requires was out of the question. Because of the fluidity of the system the team were unable to perform volume testing. The CAPSA team were working exceptionally long hours and the hammer was down.
7.66 Meanwhile the DCOs, the nemesis of Local, had determined to establish for themselves the precise state of CAPSA and to get a measure of the work which it would require of them. Early in May 2000 the technical environment was set up for a presentation by the CAPSA team, showing the operation of CAPSA in a typical Departmental setting. The demonstration was a failure. A number of important shortcomings in the CAPSA strategy for handling heterogeneous client platforms were revealed. More seriously the DCOs, zealous guardians of the requirements of their Departments, became aware for the first time of the functionality dropped by the Project Manager. Most embarrassingly it was apparent that the full extent of these cuts had not been effectively communicated within the CAPSA team.
7.67 It was from this point onwards that CAPSA began its final descent into the unmitigated disaster that was go-live. Certainly there is evidence that the confidence of the Director of Finance had been dented. In April project meetings were held with a view to identifying a fall back position in the event that CAPSA could not go-live on the projected date. This work however, did not continue and faith was placed in Oracle's ability to fix the bugs. By late May 2000 influential and indeed expert voices external to the project were being raised, calling into question the viability of the August 2000 go-live. Project reports from within the CAPSA team were signalling a high risk that functionality would not meet user expectations and of bugs affecting critical functionality.
7.68 In June 2000 Robson Rhodes, as the internal auditors, reported to the Audit Committee in significantly negative terms concerning the state of the project. Their report highlighted the lack of adequate contingency plans in the event that CAPSA was unable to go-live. The lack of such contingency plans was, of course, only explicable in the light of the absolute determination of the project management troika to go-live, come what may.
7.69 Despite having pushed back testing, by the end of June there appeared to be growing confidence in the CAPSA team that Oracle were 'on-top' of the bugs. Certainly Oracle were devoting substantial effort and showing clear commitment to dealing with the outstanding issues. Steps were therefore taken to engage in some user acceptance testing, an activity second only to requirements gathering in overall importance in projects of this nature. Naturally, because of the compressed timescale this testing, originally scheduled for March - late in any event, had to be radically curtailed. In particular it proved difficult to 'populate' the system with adequate or appropriate data. In the event user acceptance testing was never completed and in none of the trial areas did the users 'sign-off' on the system.
7.70 It was time to start user training. Oracle training consultants were engaged and a programme assembled. Early summer was not the best time in the academic year to hold training courses for hard-pressed Departmental administrative staff but there was, nevertheless, a substantial enrolment for the courses, and still some will on the part of attendees to 'make CAPSA work'.
7.71 For most staff the training was, in retrospect, unsuitable. The courses took little account of the particular nature of University work or of the specialised workflows which this work entailed. The trainers were familiar with the core Oracle Financials functionality but not with the particular Cambridge instantiation of it. Highly computer literate staff were mixed in with staff that had little or no computer experience. No attention whatsoever was paid to the massive conceptual leaps which those individuals without an accounting background needed to make in order to use the system. Many of the example transactions used in the training were either overly simplistic or unrepresentative, commonly both. The technical environment for training was itself unstable, with lock-ups and crashes giving an unwanted taster of what was to come. The result of the training programme was that mixture of bewilderment and unjustified confidence that generally accompanies poor education. In the minds of the CAPSA team once user training had started there was no going back, August go-live was now wholly inevitable.
7.72 Astonishingly, apart from a brief report some months previously, very little attention had been paid to the matter of continuing support. Nobody had considered August 2nd! It was only in mid-July, in what might appear an after thought, that the existing understaffed MISD help-desk function were asked to step into the breach. There was no room, no desks, no phones, no materials. At this point support could only be nominal. A basic help structure was set in place, temporary clerical staff were hired to provide first level support, that is answer immediate technical questions, but they were only able to provide call-logging. The CAPSA implementation team were intended to provide second level support, answer more technically demanding issues that required long term resolution. In the event they were to have their hands full. Later in the day Oracle provided some second level support directly, further evidence of their strong commitment to the project. Nevertheless, for the duration, and despite the best efforts of MISD staff, the help-desk was a sink - problems went in but little came out.
7.73 At the end of July 2000 notwithstanding the poor stability and inadequate testing the weekly project meeting decided that the acceptance criteria had been met and approval was given for go-live. At the beginning of August 2000 CAPSA went live, without parallel running. The Oracle technical consultants who had been working on implementation left and the longer term Oracle support scheme kicked in. In mid August 2000 the KPMG project management team formally, and in accordance to the plan, handed the project over to the University.
7.74 The pressure of the project had already told on University staff attached to the CAPSA team, and the Director of MISD was ill during this period. He was later to leave University employment on the grounds of ill health. The Director of Finance, a strong manager with a positive can-do attitude, took the helm.
7.75 A lengthy report could be made covering all the problems encountered by CAPSA users but little purpose would be served by doing so. Instability; the complexity of straightforward and frequently occurring transactions; the poor support for some hardware platforms; the obscure accounting terminology; the wholly inadequate reporting; the peculiar arrangements for printing; the difficulty in matching the underlying security model onto the University organisational arrangements; are among the most common complaints. The inevitable, and perhaps timely, jolt that commitment accounting was going to provide to the existing administrative arrangements within the University was amplified by the technical inadequacy of the system. Perhaps the most difficult thing for users to understand was the distance between the CAPSA they were given and the requirements they had set out in 1998. This was however only slowly to dawn on CAPSA users, as a further crisis intervened.
7.76 Immediately after go-live the system slowed down to a crawl. This should not have been unexpected. Getting large database-centric systems to perform in a distributed setting is a well known problem, they require careful tuning. Given that no volumetric testing had taken place and that a fragile system was to be hit immediately by a large number of inexperienced users, the problems were inevitable. As it turned out the system had not been configured for the right number of concurrent users and once this had been sorted out it was hit by a bug arising out of the interaction between sub-ledger security arrangements and caching. Problems of this nature are par for the course, and proper volumetric testing combined with phased roll-out and parallel running would have ensured they never reached a crisis point. In the case of CAPSA however, for a critical 6 weeks the system was all but unuseable.
7.77 Effective technical work by Oracle and a sub-contracted database administration service SysAO largely resolved the performance problems. Unfortunately the relatively small amounts of slack that users were prepared to cut for CAPSA had been rapidly consumed. The psychological effects, in terms of dented confidence, were perhaps more severe than the technical ones. In the CAPSA team desperate work to fix bugs and improve the system continued, there seemed to be no option available but 'to hold the line'. A formal health and safety warning concerning stress and working conditions was issued. By mid-October 2000 the combination of these working conditions and poor pay had resulted in the loss of 50% of the staff of MISD.
7.78 Over the summer and as the new academic year began, patience on the part of the users was exhausted. A series of increasingly angry meetings was held. Senior academic staff, having seen their administrators struggling with CAPSA and the stress and effort that this entailed, spoke forcefully about the problems. Dissatisfaction reached such a pitch that the Vice-Chancellor stepped in to Chair an open meeting and to calm the situation.
7.79 From the beginning of October 2000 the Director of Finance fell ill and Mr Sandham took the lead as Acting Director of MISD. Work on CAPSA bug-fixing continued, albeit slowly. There was now insufficient expertise and resource within the CAPSA team to cope with the fixes that Oracle were able to make. This situation was to last until the appointment of an experienced, determined and highly competent Acting Director of Finance, Mr Melvyn Keen.
7.80 Mr John Kenworthy, Interim Director of MISD following the departure of Mr Robinson, introduced a system for prioritising application issues. Mr Keen asked Oracle to assess the state-of-play. The result of this assessment was pessimistic - it would require at least 11 people for a further 6 months to deal with the critical system issues and stabilise CAPSA. In the event he was able to secure some additional effort, approximately 25% of that required, from Oracle at substantially discounted rates and this contributed significantly to improving the system in the period October 2000 to February 2001.
7.81 The CAPSA-SC held its final meeting in December 2000 and Prof. Longair provided a final report signed by the Treasurer and the academic members of the committee.
7.82 At the instigation of Mr Kenworthy, CAPSA was moved from an implementation project onto an ongoing operational basis. To reflect this, the name of CAPSA was changed to Cambridge University Financial System (CUFS) and a permanent operations team established with significantly improved help desk arrangements. In February 2001 at the direct suggestion of the Registrary, with whom he had previously worked, Mr Roy Bent, an independent consultant with a very substantial background in the acquisition and operation of University administrative computing systems was hired and assumed the role of Operations Manager. A new oversight arrangement was put in place, the Financial Systems Management Committee (FSMC). This committee is chaired by Prof. Longair and shares some of its membership with the late CAPSA-SC though it has a wider membership drawn from among those with expertise and professional involvement with CUFS.
8.1 Below I try to pick out some key areas 'where things went wrong' and to identify the principal causes. Most of the points flow directly from the outline above. Running through them all are the consequences of the constant changes in managerial arrangements and personnel that characterised CAPSA.
8.2 It is fair to date the start of CAPSA's problems to 1993. The failure to lay the foundations of commitment accounting at this point was a serious error. Further, the failure to pick up on the difficulties which the relatively conservative enhancements to Legacy caused, and to see these as indicative of wider problems within the University's handling of its accounting and financial processes suggests that this important area was receiving insufficient management attention.
8.3 Responsibility for a critical aspect of the University's financial management, the determination of strategy and the development of new systems, was placed in the hands of a junior member of staff. The Treasurer failed to exercise adequate management control or oversight over the work that was undertaken.
8.4 It could be argued that a lightweight 'federated' system of the form envisaged in Local was architecturally the most appropriate solution for the University. ACU's 'instincts' were correct. The development of Local, however was not undertaken in a manner likely to lead to success and ACU possessed neither the skills nor the resources to carry it through. In particular ACU had not developed the technical basis in distributed heterogeneous systems necessary to construct Local. The criticisms that the system received were justified and the time spent on Local was wasted.
8.5 The engagement of Tate Bramald is one of a series of external appointments of consultants and technical experts made without competition and without a proper process. The appointment of Bayles Consultancy and KPMG are further cases. Such appointments are risky, open to unfair influence and may not be in the best financial interest of the University. It is also clear that these consultants were not in each case provided with a clear written brief, though some provided their own terms of reference, with respect to which the University could manage the relationship.
8.6 During the early phases of the Bayles Consultancy tenancy of CAPSA a good job was done in gathering requirements and in involving the academic units of the University. There was however, no systematic or complete business process analysis - an essential accompaniment to an exercise of this type. No attention was paid to business process reengineering, nor to managing the expectations of users in the light of a package acquisition. In general the requirements were translated directly into the functional specification and the process was configured as for a 'green-field build' rather than 'buy-in and tailor'.
8.7 From the inception of the project and until the arrival of Ms Rudkin as Director of Finance, staff within the Finance Division had inadequate involvement in the project. As a result serious technical mistakes were made, for example the false assumption concerning the Chart of Accounts.
8.8 There is no reason, given the decision to acquire a homogeneous centralised accounting system, to doubt that the decision to purchase Oracle Financials was sound. The manner in which the selection process was managed was however, inept. Further, the lack of a serious examination of appropriate reference sites and extended user testing of alternatives are serious strikes against the process.
8.9 The difference between the actual costs incurred and those originally projected are startling. This can in significant part be attributed to the poor quality of the financial case supporting the acquisition. No proper attempt was made to estimate the cost of implementation nor of the recurrent costs. The failure to consider in any detail the costs that might be incurred by the system users is inexplicable. Not only is the cost significantly greater than had been estimated or approved, but even today it is almost impossible to obtain a full and accurate account of it. It is surprising that the University's financial oversight mechanisms permitted a case for expenditure of such poor quality to be approved.
8.10 The appointment of a Director of MISD should have allowed the University to fulfil its management responsibilities with respect to CAPSA. From the time of the appointment he was the senior manager directly responsible for ensuring the good order of the project on behalf of the University. The appointee was an IT professional with a direct reporting line to the Registrary. It is impossible to avoid the conclusion that he failed to fulfil this responsibility, adopting instead a hands-off approach. I have seen no evidence that he monitored progress, saw to the management of MISD staff attached to the project, or provided guidance to the external project managers. It is reasonable for the Registrary to have relied upon him to play that role.
8.11 The Byzantine governance arrangements underpinning the appointment of the CAPSA-SC are beyond the scope of this report. It should be remarked however, that continuity is vital to effective oversight and that the disbandment of the JCAC may not have assisted matters. During its life the CAPSA-SC failed to fulfil its oversight role. It often appeared uncertain if it was managing the project, acting as a customer representative for the purpose of supplying requirements, or acting as a watchdog. In the event it fulfilled none of these roles. Despite good intentions the members of the CAPSA-SC were unqualified to provide the management control that the Director of MISD appeared unwilling to exercise; insufficiently representative or financially expert to act as customer representatives; and too dependent on the external project managers to act as effective watchdogs.
8.12 For the entirety of the project there were no clear lines of responsibility. External consultants were given too much autonomy.
8.13 Though the implementation was ostensibly to be controlled using a well recognised project management method, Prince 2, there is little evidence that this was properly established, or used with the rigour that such methods demand. The implementation method, AIM, is powerful and provides the skilled implementor with a large number of valuable resources. The method is however, driven by document templates which need to be used with considerable care. I am not confident that in the early stages of the project AIM was properly used. During the entire project and throughout every phase, though large amounts of detailed documentation were produced, there is little by way of high-level overview. Both project and technical managers appear to have had difficulty in abstracting away from the immediate technical minutiae of the project to establish the big picture necessary for sound decision making and control.
8.14 It is difficult at this distance to draw detailed conclusions about the quality of the technical consultants engaged on work within the project. It is probably fair to say that this quality was highly variable, indeed more variable than the University might reasonably have expected. The project suffered severely from 'consultant churn', the net effect of independent consultants, Oracle and KPMG staff turning over with some rapidity. This gave rise to a learning overhead on the project as new consultants familiarised themselves with the University; reduced the effectiveness of knowledge transfer between University staff and consultants; damaged morale; and, in a very fluid setting, impeded technical continuity.
8.15 There were insufficient University staff attached to the project. Many of those that were attached were inadequately qualified, lacking a formal background in either computing or accounting, for the tasks that they were asked to perform. University staff, owing their principal loyalty outside the project, were withdrawn, or withdrew themselves, at short notice. The small number of MISD staff working on the project were largely concerned with peripheral tasks and lacked the skills or determination to assume a significant role in the technical direction of the project. In general terms MISD was understaffed and lacking in suitable technically qualified staff. The early retirement scheme was applied without due regard either to current staffing levels or to the preservation of critical expertise.
8.16 The Director of Finance worked hard to deliver CAPSA. However, despite concerns over the state of the project she supported the decision to go-live in August 2000. There appeared to be strong reasons for making this decision. She could have expected that the experienced University professional advisers, KPMG, would have guided her. Nevertheless, the reasons for supporting go-live were clearly outweighed by the problems the system might engender. The decision must be considered an error.
8.17 KPMG project and technical management made a series of serious mistakes. On taking over, they failed to pick up the threads of the project and to use the work that had already been performed. They failed to manage to the documented requirements or to assess the consequences of making changes to those requirements, a breach of the discipline of requirements traceability. They failed to identify or manage relationships with many of the key project stakeholders. They paid only lip service to risk management. They steered the project to a big-bang launch, without due regard to the risks this entailed. They failed to set up a process with proper stage gates so that when the schedule slipped, critical activities, notably testing, were sacrificed. They failed to put in place proper contingency arrangements in the event that go-live in August was not possible. They allowed the system to go-live without sign-off on user acceptance testing. They failed to pay attention to continuing support. They maintained a blinkered insistence on the August 2000 go-live date founded on a mistaken understanding of the interests of the University.
8.18 Oracle supplied a product which was of poor quality. In particular the Research Grants module is only marginally fit for purpose. Such poor quality is a feature of the software industry as a whole; it is however, little comfort that Oracle Financials is industry-standard sub-standard. Some responsibility for the problems encountered by CAPSA are attributable to poor configuration control and undisciplined issuing of patches on the part of Oracle. A proportion of the problems are also attributable to the fact that Oracle 'sold' functionality somewhat in advance of its capacity to deliver it. The poor quality is however, partially offset by the considerable efforts that were made by Oracle to rectify the situation.
8.19 The CAPSA project was hobbled by poor communications. Poor communications between project team and Finance Division, as a result of understaffing and lack of commitment. Poor communications between project team and Departments, as a result of an instinctive distrust between 'the Old Schools' and the academics they serve. Poor communications within the project team itself, as a result of an unnaturally tight schedule and an occasionally abrasive management style. Poor communications between the project manager and the CAPSA-SC, as a result of the desire to always present matters in the best light possible. The poor communications were exacerbated by the cumulative effects of continued changes in organisational arrangements, personnel and reporting lines.
In the light of the findings of this review I recommend that:
[i] MISD be merged with The University Computing Service to ensure an integrated IT service with the technical strength in-depth sufficient to manage and deliver on major information systems development projects. Such a merged service would be able to offer posts with sufficient variety, opportunities for skill development, prestige and pay that might attract high calibre staff with a technology background to work on management information systems projects.
[ii] The University appoint a senior engineer, with appropriate formal qualifications, and substantial experience in delivering on complex IT projects to lead the development of its management information systems.
[iii] The University prepare an 'Information Systems Strategy' covering the development of management information systems within the University. That this strategy, signed-off by the Principal Officers of the University, is used as the primary vehicle for strategic management of IT.
[iv] The University establish, within the unified Computing Service, a project office that can monitor major IT projects. That this office is responsible for setting processes, methods and standards for such projects, for ensuring compliance to these standards and for quality assurance. The office can act as an independent advisor to committees set up for the purpose of oversight.
[v] In any future IT projects the proposal is accompanied by a clear statement of the responsibilities to be undertaken in the direct technical and management conduct of the project and also on the part of the 'users' of the projected system in engaging with the development of the system. That organisational changes and early retirement programmes are evaluated against the responsibilities agreed in these statements.
[vi] Guidelines are laid down for the submission of financial proposals for major IT systems projects. These guidelines should include what costs are to be included and how the expenditure is to be monitored.
[vii] Guidelines are laid down for the hiring of external consultants ensuring proper tendering, a detailed brief and arrangements for management control from within the University.
[viii] The University establishes and maintains a complete and up-to-date Accounting Policy Manual documenting its policies and procedures.
[ix] The University devotes additional resources to the staffing of the Finance Division and in particular to the recruitment of professionally qualified accounting staff. Urgent steps are taken to put in place a programme that upgrades the skills of the Departmental staff concerned with finance.
[x] A training audit covering the Principal Officers of the University and those senior academics likely to be engaged in oversight committees should be undertaken. In particular skills in the strategic management of information systems and finance should be carefully examined.
[xi] Immediate and forceful steps should be taken to establish if the University can obtain recompense from the external consultants it engaged.
[xii] The University should not pursue any further substantial management information systems projects until at least recommendations ii, iii, iv, v, vi, vii are in place. Further, in the light of this report other ongoing projects should be subject to a careful internal audit.
[xiii] The University as a whole considers how best to improve relationships and build trust between its administration and academics.
Recommendations concerning the future of CAPSA are given below. In the spirit of shared learning, the external consultants engaged by the University may wish to review their practice in the light of the findings above.
10.1 CUFS is still not performing acceptably. It is fragile with significant risk of failure in handling day-to-day operations. Usability remains poor, workflows are clumsy with many simple operations requiring multiple screens. Reporting is particularly bad with inadequate and poorly structured reports that are difficult to interpret. Flaws remain in security. System response time is still patchy and frequently impedes progress. Down time is greater than that appropriate for a mission critical enterprise system. At the time of writing the year end procedures have entailed significant disruption. There is a general and warranted lack of confidence in the accuracy of the financial information maintained by CUFS. The research grants module is only marginally fit for purpose. The broad experience of University staff remains that CUFS is a 'sponge' which soaks up time and effort that are not matched by what it produces.
10.2 A pressing issue for the University and one which has been subject to some heated discussion is whether to continue with Oracle Financials. Ultimately I believe that this is the only viable option. Clearly many of the reasons that motivated the acquisition of a system of this type, and the selection of Oracle Financials specifically, remain valid.
10.3 Very substantial investment has been made in equipment and software but most significantly by individuals at all levels, in learning to make the system work. To write off this investment would, I judge, be an error.
10.4 System stability has increased substantially since go-live and the magnitude and complexity of the outstanding issues have decreased. The situation is no longer chaotic and there is a good basis for moving forwards. For this, the operations group led by Mr Bent and, in particular, Mr Keen on behalf of the Finance Division deserve credit.
10.5 Though new versions and the addition of functionality may destabilise the system in the short term, in the longer term the University can expect some further improvement in terms of stability and 'system quality'. Similarly I would anticipate improvements in the experience of the system as Oracle improve their interface, core technologies for supporting heterogeneous clients improve and the University improves reporting. Indeed the latest versions of Oracle Financials may actually deliver in a reasonable form the functionality that Oracle 'sold' to the University.
10.6 In this case Oracle have appeared to rely heavily on users to test and debug their systems, a practice that I regard as unacceptable despite its prevalence in the software industry. The less heavily a component of a system is used the longer it will take for it to become relatively bug free. For modules such as research grants, with a small user base, in relation to the overall Oracle suite, this period is necessarily extended.
10.7 Though it is difficult to make an appropriate estimate, with a permanently established and well resourced operations group, without significant technical discontinuities, at the current rate of progress, the University should look to a period of at least 2 years before all the major outstanding issues are eliminated.
10.8 Oracle Financials is a 'modular' system, though the extent to which this modularity is a marketing, as distinct from a technical, feature might be open to discussion. It has been suggested that it may be possible to step back from some specific modules of Oracle Financials opting instead to acquire or have them bespoke developed. Research Grants and Inventory are the obvious candidates. Research Grants was not originally developed by Oracle and is an adapted project ledger. Inventory was designed for a manufacturing setting and is not well suited to the particular nature of University stores. In the case of Inventory is not difficult to envisage a lightweight solution that synchronised with Oracle Financials; indeed the system currently in use in Engineering might act as a 'loose' proof of concept. Research grants is altogether a more complex proposition as it is more tightly bound to the General Ledger and has widespread use, it is nevertheless worth serious evaluation. It should be noted however, that the horizon for any developments of this form is 2 years and beyond. It should also be understood that this is a suggestion which requires further evaluation in the context of the University Information Systems Strategy, see recommendations above.
10.9 As indicated above, no thorough analysis of the University's business processes has taken place. By reducing the number of units exposed to CUFS, by centralising routine processing (some invoice processing and payment of suppliers) and by a slow but steady reduction in the diversity of the processes which are adopted, it may be possible to make life with CUFS easier. In any case, long term management of the University's information systems requires that these processes are well understood and open to improvement. I would therefore recommend that the University undertake a careful business process analysis with a view to making sensitive and incremental changes in the way it handles financial transactions.
I would like to acknowledge the invaluable assistance of Alan Winter who has acted as Secretary to the Review and has provided discrete and efficient support. Iris Hunter has also provided material support of which I am most appreciative. I am grateful to Ross Anderson for suggesting my name to the University. I have very much enjoyed working with Mike Shattock, he has taught me a lot. Thanks finally to my colleagues and students at UCL to whose work I should have been paying more attention.
1 Perrow, C. (1985); Normal Accidents: Living with High Risk Technologies; Basic Books.
|Previous page||Table of Contents||Next page|
Cambridge University Reporter, 2 November
Copyright © 2011 The Chancellor, Masters and Scholars of the University of Cambridge.