Technology Selection: What You Need to Know to Make a Decision
Jeffrey Herron, Beaconfire Consulting, USA
Once an organization decides to launch an Internet technology project, the next step is selecting the right technology and the right vendor. Although today's marketplace for nonprofit technologies is flooded with generally mature and capable Web-based tools that provide functionality in the most common nonprofit business areas, the selection process is daunting. Given the availability and viability of so many tools, why does technology selection continue to be a struggle? Shouldn't technology selection get easier as more viable technologies become available? In part, the problem is that nonprofits are faced with making a decision without understanding that technology is not always the most important element of the decision making process. The key challenge, then, for nonprofits is to determine what factors are central to making the best-fit selection for the organization. This paper will provide some guidelines to help organizations make sense of their options and will outline the pros and cons of the many solution categories so each technology selection can be made from an informed perspective and in the organization's best interests.
Keywords: technology selection, non-profit strategies, planning
Since about the middle of the last decade, organizations have been turning to the Internet as a new outlet for their business. Nonprofits in general were a bit slower to adopt Web-based tools and technologies as another way to fulfill their missions. Today, however, thousands of nonprofits use the Web as a core part of their activities - whether for marketing, information dissemination, program delivery, engaging constituents or raising funds - and employ any number of technologies, tools and solutions to achieve results on-line. Over time the solutions that nonprofits have used have generally become much more complex and feature-rich, more powerful and easier to implement, and less expensive and less risky for the organization. Put simply, the market for nonprofit technology has matured.
Today, nonprofits can select software products from among dozens of vendors focused solely on the unique needs of nonprofit organizations. Many of the common nonprofit functions have spawned categories of software that cater to these specific needs. Vendors offer products to help nonprofits with on-line fundraising, membership management, content management, e-mail marketing, advocacy and community building among others.
As a consulting company that helps nonprofits to identify and select software to meet their on-line needs, we have found many of the products available today to be generations ahead of the custom tools that pioneering nonprofits created, often at great expense, in the late 1990s. In fact, a number of vendors have repackaged the software created for these innovative and forward-thinking clients into solutions now available as products for fractions of the cost. Additionally, the nonprofit solutions have also benefited from the concepts, innovations and technologies available for years in the corporate world and now available in these packages.
Against this backdrop of mature software products, nonprofits would seem to be in a 'golden age' of Web-based software with easy evaluation and selection to be made. Given the availability and viability of so many tools, why does technology selection continue to be a struggle for nonprofits? Shouldn't technology selection become easier and more straightforward as more viable technologies are available?
This paper seeks to help nonprofit technology decision makers to understand why despite an increased number of viable product options, technology selection and decision making has not gotten correspondingly simpler and more straightforward.
The premise of this paper is that technology selection and decision-making is difficult for organizations for many reasons, but only partially due to the complexities of technology. The fact is that organizations do not evaluate fully all the factors that impact the best fit for the organization. This paper follows a general outline to present the challenges, considerations and options that organizations must consider before making a selection.
Challenges: We start our discussion with the organizational context explaining why technology decision-making can easily fail to yield positive results.
Getting Started: After identifying and exploring some of the barriers that make technology selection processes challenging, we discuss the high-level steps that lead to the point of selecting a technology.
Considerations: In order to make an effective selection, there are key factors to consider during the review. We suggest that there are three types of considerations: Functional, Business and Technical.
Decision Making Criteria: We'll discuss the need for and value of clearly defined decision-making criteria.
Technology Options: Part of the challenge that organizations face is making sense of the many viable technology options. Generally, the products differ most in how they are sold and implemented. If the product is an off-the-shelf product that is purchased (licensed) upfront and then installed, this impacts the organization differently than if the product is leased for a monthly fee over time. We will explore the impacts of each of these ownership models on the decision-making process.
Selection Processes: We will review the overall steps of the selection process and explain the various formal and informal processes that organizations can use. We describe the Request for Information (RFI), Request for Proposal (RFP), and Request for Quote (RFQ). We also briefly describe a process called Software Evaluation as another option.
Why is Technology Selection So Hard?
It is important to start with the organizational context to understand why technology decision-making can easily fail to yield positive results. Part of this context is the barriers that may or may not exist internally to prevent solid decision making, some of which is specific to technology and some an organizational challenge.
Our experience suggests that there are three general categories of challenges that can derail technology selection efforts: Management, Process, and Technology. In some cases (Management challenges), the selection process can be doomed even before it gets started.
Decision by Committee: For those leading a technology selection process, there may be a lack of clarity about who has the ultimate decision-making authority, particularly when a decision-making committee is involved. Committees are very good for getting multiple perspectives and organizational input, but can be a barrier to clear and effective decision-making if all members do not share the same understanding of the ground rules. For the leader of the selection process, understanding the dynamics will go a long way toward ensuring a successful selection.
Authority: It is common that those who use technology (think Marketing or Communications) are not typically the same as those who have budgetary responsibility (think IT or Development). This situation is particularly common when selecting Web technologies since the Web is often not centrally owned by any one part of the organization. In this case, does the authority for decision-making rest with the department with the budget, or with the department whose requirements the tool should satisfy? The answer to this question should be clear before work begins.
Buy-in: Another management challenge involves ensuring that all the key stakeholders participate in the selection process. Buy-in occurs when a project has proper support from all or most of the key stakeholders, those critical to the success of the project. When this doesn't happen, stakeholders who are left out can sabotage the adoption of the project or sometime try to stonewall the decision. Example: When technology teams are not involved in the evaluation and decision-making, they are put in a tough position when asked to review a decision after the fact. If a department has already selected a tool, then the options the technology team has are limited. They could:
Buy-in results when stakeholders, departments and staff feel they have input, can present feedback, and are sure their concerns will be addressed.
Business Case / Lack of Support: Some selection processes fail because the full business case has not been made for the project. This challenge manifests itself at two points in the project. The first is when a team wants to begin a technology selection project and early on the project does not materialize due to lack of resources. If a winning business case had been created, then the resources would likely have been allocated. The second situation is more damaging and wasteful. This situation occurs after the project has started but is stopped because the leadership of the organization does not support it. For whatever reason, lack of a clear business case to demonstrate that the costs for the project are outweighed by new opportunities cost the project high-level support. Having a clear business case provides the following benefits:
Many organizations feel that selecting technology projects is difficult because of the way technology companies market their products. Here are some of the more common challenges that we hear from clients involved in technology selection projects:
'There are many solutions to choose from. How can I even begin to know what is right for my organization?' The downside to the large number of viable solutions is that there are lots to choose from. The key is to use criteria to narrow the field.
With all of the different types of product and functionality, it is difficult to compare products in a meaningful way.' This is the most common challenge with evaluating vendors and products. Often getting to the point of an apples to apples& comparison is the hardest part of any process.
It is hard to tell which features are included in a product or solution and which are extra.
The solutions seem to be either too generic or too complex.
'Will I get stuck with a product that is obsolete, proprietary, or tied to a particular vendor?'
There may be little that project leaders can do about internal power struggles, lack of communications and buy-in, and difficult budget choices. However, understanding these and other challenges outlined above before the project begins gives the team the ability to prepare for them and adjust to the situation as it unfolds.
We recognize that technology selection projects do not materialize out of the blue. Rather, they occur as the culmination of other efforts to define business problems including Strategy projects, program planning, internal technology audits or other planning efforts. Our starting point for discussion assumes that the organization has identified a business problem and determines that a project team should be formed to solve it. From here, we believe that the project unfolds with a number of key initial steps that will determine if there is a technology need and if this project will involve a technology selection.
Definition of the Project
Ideally, the first two steps above will lead to the creation of a business case for the project. As noted earlier, the business case consolidates much of the rationale for a project along with a quantitative and qualitative analysis of its merits.
Reviewing the Role of Technology
Since many projects, including non-technology projects, begin with the above steps, there must be some effort to determine whether and then how technology could be a part of the solution. In reviewing the entire solution, it is important to keep the role of technology in perspective as it may be a small part of the solution or it may be the entire solution.
In order to make a best-fit technology selection, there are many questions that should be asked and many perspectives to consider including the organization's unique context for making the decision. If the questions facing organizations when making technology selections were solely related to the functionality of the tools, then the decision-making would be more straightforward as the product options mature and the risk of making a poor choice diminishes.
In the last few years, our experience has been that nonprofits have been lulled into thinking that the selection of a product is easy because most technologies provide much of what organizations think they want. Organizations believe that most of the solutions are good; therefore there are few bad selections, and those are generally easy to spot. Consequently, organizations believing the decision-making simple do not consider all the issues that are central to whether they are making a best-fit selection, not just a good selection. The difference is that a best-fit selection is good for the organization across a range of factors; it is not simply a tool do what the project requires.
We posit that there are three categories of considerations that impact the selection of a best-fit tool: Functional, Business and Technical considerations. We'll deal with each one of them in the section below.
Functional considerations (project considerations)
The functional considerations are the project-specific issues that organizations typically have a good handle on. The specific features and functions that the project is trying to address are usually quite apparent to the team. Certainly, this is the case after to the team identifies the business problem and the role technology could play. The functional considerations are more self evident to the entire project team than the other two considerations.
For example, in a project to select a Content Management System, there are dozens of features that the system will provide, and they can be compared with the organization's list of requirements. In this example, here are some of the primary features that might come into play.
Diagram 1: Example of Features List for a content management system.
In each of these feature categories, there could be numerous additional features that more fully describe what the organization wants. At a minimum the team's listing of requirements will serve as the functional considerations. Selecting a technology product without even the bare minimum of functional considerations will likely end in a sad and frustrating waste of time.
Business considerations influence the project decision-making but have roots outside of the project and may be not solely be the purview of the project team. For example, the risk tolerance of the organization (one business consideration) might lead the team to consider safe and simple solutions if the tolerance is low. The reality is that this consideration reflects the organization as a whole and not simply the view of the organization from the perspective of the project.
Business considerations are also more often the purview of senior management or the IT staff, and are therefore at risk of being left out of the decision-making process if these parts of the organizations are not represented on the team. For example, without the representation and perspective of senior management, the team may not be able to factor in the costs and overall impact to the organization of a new technology. Understanding and considering the change management issues in technology selection is important to finding a best-fit selection.
Here is the full list of the Business considerations that we recommend:
Time to Implement - The time by which the project must be complete is a prime factor. If the time to implement is a non-negotiable deadline then this factor will drive many of the decisions for the project. Since large technology projects usually impact the entire organization, the time to implement consideration is also driven by organizational calendars.
Budget - Budget is usually a primary factor. In the context of the project or department, the cost is usually well understood and can be compared across vendors. However, there is a larger perspective on costs than the budget for the project: Total Cost of Ownership.
Total Cost of Ownership - This concept refers to the analysis used to measure not just the initial costs but also the cost over a longer period of time, when variations in cost can drastically shift the perspective of overall value. Total Cost of ownership often includes a number of factors:
Change Management - CM refers to the management and understanding of the impact of technology on people and processes within the organization. When organizations undertake projects that may improve or change current processes as an outcome, management of the impacts is critical to ensuring success in yielding the expected improvements or benefits. Change management also seeks to identify cultural or other non-tangible barriers to the project's successful adoption.
Business Opportunities / Strategic Goals - Some projects are undertaken to address a strategic need or take advantage of an opportunity. In this situation, a strategic opportunity provides context as to why the project should move forward, and in some cases provides justification for budget expenditures that otherwise may not be possible. If the project is meant to satisfy strategic goals, then this consideration may be a primary decision-making factor when the technology is selected.
Internal Resources - Another factor in selecting one technology over another involves the availability and capability of internal resources required to support, manage and use the technology. Lack of internal resources with skills suggests either training or outsourcing, both of which may add to the cost and risk of ownership.
Maturity of Technologies - Maturity refers to a measurement of the risk associated with adoption a tool that is not well supported or from a company that does not have a track record of success. This evaluation involves a review of a company's technology in comparison to the market they serve to determine if the applications are stable, reliable and have gained the loyalty of a legion of developers and programmers. The loyalty of the developer community is particularly important for assessing the viability of open source solutions which are often not supported by a corporate entity and could therefore pose increased risks for organizations that do not have technical skills on staff.
Risk Tolerance - Understanding the amount of risk an organization is willing to take on is critical. If the project puts key business areas (fundraising, customer service, user experience) at risk, this may influence the options to be considered. Risk can also refer to the pace and nature of technology change within an organization. If the organization is already undertaking another large project that involves lots of change, a technology that adds to this upheaval scores lower than one that offers lesser change. Risk tolerance is usually paired with one of the other considerations (budget, change management, resources etc) but is important on its own accord.
The following section describes the technical considerations and their importance.
Security - Encompasses both data and application security.
Performance - The system or application's general ability to meet day-to-day expectations of users, site visitors and administrators.
Reliability - The system or application's ability to consistently perform simple and complex tasks under optimal and adverse conditions.
Scalability - The system's ability to meet exponential growth in traffic and transactions either with or without additional hardware.
Portability - The ability of application and data to be migrated to other infrastructures or applications as needed indicates that it is portable.
Interoperability – The ability of systems and applications to connect, communicate, and exchange data with each other. Interoperability includes:
Compatibility: Standards for Coding and Data Storage, Standard Communication Protocols, Web Services
Integration: Ability to integrate with other systems using widely accepted protocols and practices like API, XML Schemas, and Web Services
Data Exchange: Ability to collect, aggregate, configure, and transfer data based on flexible sets of data exchange standards
Support - Good support indicates that there is industry-wide support for the system, core technologies, or application from a variety of vendors and developers. Look for a mature system with a stable development community.
Ease of Use or Usability - Ease of use from the perspective of different users - end-users, support staff, and system administrators
Key Decision Criteria
In the course of the decision-making process, all considerations are not equal, nor do all considerations always have a role in the decision. Before any selection can be made, decision criteria must exist. The most critical part of the selection process is definition of and agreement on the decision-making criteria. The most common problem for selection teams is lack of clear, documented and agreed-upon criteria for selection.
So what are the most important criteria? How many should there be? It depends. But here are some general thoughts.
What happens when you don't have agreed-upon decision-making criteria? How many times have you been on a selection committee where after all of the interviews, RFPs, demos etc, the team members have a different opinion based on different criteria? Think of the job candidate who is hired because the interviewer thought he/she was 'nice' or because someone 'liked' him/her the best. The fact that someone is nice and is likable was probably not on the list of factors that were set out going into the hiring process. If these two factors were not on the list, and yet they were important to the decision, then the team didn't prepare its true decision-making criteria and therefore was at risk of making a decision that was not in the best interests of the organization.
Regardless of the considerations that go into the decision, without an agreed-upon and well thought out list of decision-making criteria, the team will struggle to reach consensus and may not serve the best interests for the organization.
Each project is driven by situational factors that determine which of the considerations we've discussed come into play. Consider the case of a recent Beaconfire nonprofit client who was looking to enhance an on-line presence for the purposes of raising money and engaging constituents. This organization had many years of successful off-line direct mail fundraising and possessed a well known brand, yet on-line its fundraising was virtually nonexistent. Our strategy work resulted in a series of recommendations to deploy on-line fundraising tools along with a new Web presence to engage those interested in its mission. As part of the strategy, we considered a number of technical options and went about the process of helping the organization determine which option was a best fit.
Here were the relevant factors that created the decision-making context
From this context, the key decision making criteria emerged:
Before continuing this client example by describing the comparison of the specific solution options, it is important to explore the technology marketplace and the different models for technology solutions given that they can dictate the direction of the selection.
In the marketplace for technology solutions, there are three prevailing models that define the available solutions. Each model differs in its approach to a few key areas: ownership, cost, flexibility and implementation. The three models are Build, Buy, and Lease. These models are not unique to technology in that the same fundamental choices can be found in shopping for a car (Buy or Lease), finding a home (Buy, Rent or Build) or hiring (Temp or Hire). The similarity of the technology models to other options is helpful in providing overall context for advantages and disadvantages of each model.
Build – custom build / integration
There are two defining characteristics of these solutions. One is the fact that the solution is Built or assembled from scratch or other parts to form a solution. The other factor is that the solution is custom, meaning that it is created based on the custom specifications of each client. Generally solutions that meet either of these two criteria fall into this category.
Buy – third-party products
Solutions that fit this model are well-defined products supported by a company and fill a specific niche in the marketplace. Products are characterized by having a roadmap that describes the schedule for upcoming releases and future product enhancements. The existence of a roadmap provides visibility into the value of the product in the future as well as assurances that the current product is not going to become outdated.
The other main model for technology solutions is the Application Service Provider. In this model the clients pay a single monthly fee that covers use of the software, support, and hosting costs. There are no license costs, there are no extra hosting charges, but there may be some setup charges. However, these are typically relatively small compared to the initial costs of the other models.
There are several other types of solutions that these three options account for. The other solutions are types of custom builds that mitigate some of most serious cons of this type of solution.
Open source software
These solutions are based on software programming languages or software frameworks built on tools that are open and not proprietary. Essentially, open source software is available under a broad license that allows for the code to be used, modified and shared with few restrictions, including cost. Most open source software is available free or at reduced cost.
There are also companies that offer quasi-custom development solutions based on code that they reuse from previous engagements, which is then customized each time for a new client's needs. Some vendors claim that type of solution is a product and sell it as a combination of software and services. In our view this solution is not a product because it does not have a defined set of features, does not typically have a roadmap, is not sold under a software license, and does not come with upgrades or enhancements. In the end, this solution suffers from many of the downsides of custom development and few of the upsides of being a product.
Now that we have reviewed the viable product models, our example client was faced with evaluating solutions in the Lease and Custom (open source) models.
Given the decision criteria outlined above for this client, the best-fit choice was pretty clear. Recall that the criteria were:
The Lease option was much more consistent with their top three criteria than the Open Source option despite the lack of licensing costs in that option. With the Lease option the organization would take a smaller risk by spending less money and getting more functionality more quickly. The other three considerations were more neutral in the decision. In the Lease option, the risk of disappointment in features and data integration was worth taking when compared to the budget and timeline risk offered by the open source alternative. In the end, what looked to be a complex and unwieldy choice became very clear
Now that we have defined all the considerations, options and challenges that impact the selection of technology, what are the common processes used for making a selection? How can I ensure that I identify, compare and analyze the most important considerations?
Each of the common selection processes is described in some detail in this section. In addition, note the process that we use for technology selection to provide an alternative method.
Request for Information
A Request for Information or RFI involves an organization asking vendors to respond with a statement of their capabilities, qualifications and general skills in an area identified by the organization.
The purpose of the RFI is for the organization to learn about different vendors, to gauge their interest in working with the organization and to gather ideas for ways to solve their problems. In the RFI, vendors describe their portfolio of skills, experiences and capabilities and are not asked to provide as much information about a specific solution to a client problem. Vendors are essentially providing information and selling themselves without much of an idea of the organization's problem. In the RFI, vendors may provide general cost ranges but the RFI is not intended to solicit pricing.
The RFI process yields a list of interested and preliminarily qualified vendors that the organization can continue to work with. The next step is usually a Request for Proposal.
Request for Proposal
The Request for Proposal or RFP is the most commonly used means of engaging vendors. The RFP is intended to solicit more specific information from vendors about the solution to the problem they identify. Organizations usually include the goals of the project, a list of requirements for vendors to use in crafting a response.
Compared to the RFI, this document focuses more on the organization's problem and possible solution than on the vendor's qualifications. The vendor provides creative thinking, suggestions and proposed solutions as a way to demonstrate its capabilities. The RFP is also used to obtain general pricing estimates from vendors for their proposed solutions.
Request for Quote
Request for Quote or RFQ is the most detailed of the three documents. As the name implies, an RFQ is very focused on soliciting proposals that include firm pricing. In order to ensure solid pricing, the RFQ contains very specific requirements for the vendor to use in the response, whereas in the RFP the organization typically provides more general information about their goals and requirement.
In organizations with strict vendor selection guidelines, like the government, it is not unusual for there to be several rounds of the selection process, and each of the documents above play a role. For most organizations, simply using one of these documents in the selection process will be far better than informal methods of selection. Our experience has been that organizations write a RFP but expect to receive the level of detail and certainty found in an RFQ. Often times the problem is that organizations do not provide enough information for vendors to provide solid and accurate pricing. At the root of the problem for the organization is that they don't know their requirements or don't provide a sense of the priorities they have when articulating an outline of a solution they would like to see priced.
All three of the methods rely heavily on formal written responses, with differing degrees of latitude offered to the vendors to shape their response. The RFQ is the most specific and therefore is the one that is the most likely to produce a strong comparison between vendors/products. The RFP can be valuable to the organization as a way to receive free advice or ideas that might be incorporated in the final solution regardless of the vendor selected. However, this value comes with a tradeoff; namely that it may be more difficult to compare each response directly to the other in an 'apples to apples' way.
One of the main challenges organizations face is ensuring an apples to apples comparison of each vendor/technology. As noted above, the RFQ and RFP offer some ability to compare responses, but the RFQ is the better way.
Beaconfire has created a process called a Software Evaluation to compare vendors and technologies in an objective fashion. It is based on comparing Functional, Business and Technical considerations.
The Software Evaluation process begins with the creation of a detailed features and functions list according to the client's requirements. Then we and the client prioritize the features and functions to determine the must-have functions. At this same time, the other business and technical considerations are reviewed to identify the priority considerations which then form the key decision criteria.
The next step involves creating a features matrix. Based on the features, initial vendors who appear to meet the major considerations including budget, features, and technology preferences are selected. The ability to select a small number of initial vendors is very much dependent on the limitations outlined in decision criteria. Then, we send the features matrix, without priorities identified, to the initial vendors and ask that they complete the matrix by identifying the features that they support and which are included in the product. The vendor is also asked to provide competitive pricing for its solution based on the features matrix. The vendor answers for each function in the matrix can be one of the following:
Based on these results, the matrix is scored with the various answers assigned a value. Then we combine the priorities of the features with the score to achieve a weighted point value. Depending on the complexity of the evaluation, we have also weighted categories of features to value the relative importance of some sets of functionality over others. Finally, based on this scoring, the vendor list is narrowed to a final 2-3 vendors.
Then the final vendors are invited to present demonstrations of the product to the client and to address remaining questions. The vendors also have an opportunity to present their companies in addition to their product; however, the presentations are limited generally to key questions about capabilities and outstanding issues.
We take all of the apples to apples information from the matrix, the demos, the questions that were answered and the quote. The end result of this software evaluation is a recommendation on a software product and vendor that best meets the requirements laid out by the organization. Beaconfire arrives at the recommendation by factoring the features matrix scoring (apples to apples comparison), the key decision criteria, and the totality of the Business, Functional and Technical considerations.
This paper suggests that organizations can fail to select the best technology product due to challenges that may be out of the control of the selection team. We also believe that there are many considerations impacting whether a decision is a best fit for a particular organization, including business, functional and technical considerations. We recognize that with all of the technology options to look at, this may lead to confusing choices and difficult comparisons that are not ‘apples to apples'. Finally we discuss the many processes that project teams have used to conduct a competitive selection process.
So in the end, which of these processes is the right way to select a technology product? Unfortunately, there is no perfect or right way to conduct a selection. Each of the processes has its place and is valuable when applied to the appropriate situation.
In the end, the best way to ensure best-fit technology selection is to use a formal process for comparing vendors/products on business, functional and technical considerations. If your decision is based on an honest review of the key decision criteria and the considerations outlined in this paper, then it is very likely that your decision will result in a best-fit selection.
I would like to acknowledge the work of the entire Beaconfire Consulting team in contributing to the thinking contained within this paper. This work is the culmination of our staff's efforts to assist our clients in serving their constituents and meeting their missions through the use of technology. I would specifically like to thank my business partners, Lynn Labieniec, Michael Cervino and Dottie Hodges, for their support, and our lead solutions architect, Usha Venkatachallam, for her contributions to this conceptual framework.