[Previous] [Next]
[Previous] [Next]
Since a vendor cannot do a full-scale production test of the product at your institution, members of the selection team should make visits to sites of similar production scale that have successfully implemented the product. Extensive performance benchmarks should be run on every product under actual user conditions, if possible, before choosing a financial software product, because all such products will have some performance limitations.
The selection team can create detailed evaluation reports to compare suppliers based on the above evaluation activities. It is also helpful to construct a list of key benefits that would be provided by each solution being evaluated (or ask the respondents to provide this for your scoring and use).
The selection team can also develop an internal measurement and evaluation process to calculate the extent of the gap between your institution's identified strategic, business, and technology needs (level of expectation) and the products and services offered by potential suppliers in their responses to the RFP. The evaluation is very unlikely to result in an exact match, even if the project is undertaken with the purpose of building a system from scratch by internal and/or external partnered resources. The key issue is whether the extent of the gap represents an acceptable solution and what efforts, between internal and external resources, can be brought to bear to minimize the gap.
With this in mind, there are many models by which the evaluation of alternative proposals (and resulting gaps) can be measured. Your institution can develop its own model or modify an existing model to reflect the relative value of quantitative and qualitative factors. In general terms it is worthwhile to consider an overall evaluation structure that will allow for specific evaluations of functional issues, provide flexibility to measure the relative values across applications and technical areas, and also allow for subjective judgments with respect to strategic issues and cost factors. Unless a strict cost is associated with the project, it may be worthwhile to exclude that factor from the initial stages of the evaluation.
The initial evaluation should address the functional issues related to application requirements and technical requirements. These items together will identify a system approach that will meet your institution's specified needs. The business and technological requirements detailed in the RFP will offer tangible evidence of the needs for proposed systems to match current functional requirements with the future direction of the institution in the business reengineering or technological areas.
The structure shown in Figure 5 was used by Sinclair Community College to provide an overall evaluation process for its integrated systems project. Note in this case that the weights for functional and technical requirements evaluation are different. This approach reflected a particular philosophy that business requirements should have a higher weighted value than the technical requirements. In a different environment, these values could be reversed or both areas could receive the same value, depending on where an institution placed itself on the futuristic chart.
Beyond the initial level of functional and technical requirements, the team will need to compare the different applications to the technical requirements. There is obviously more than one approach that can be taken to suit the needs of individual institutions based on strategic and tactical directions as well as the culture of the organization. Appendix E includes a detailed model based on the Sinclair illustration above, as well as an alternative model of how measures could be defined for use by the selection team to assist in the evaluation of proposals. A land mine associated with taking such an approach is that it is possible to get too caught up in this kind of weighted evaluation, to the extent of getting the right system on paper but not in fact. If such a device is used, it will be important to use a two-phase process to allow some reevaluation and flexibility in the weights.
[Previous] [Next]