Organizations constantly struggle to acheive lasting success with a knowledge management system (KMS). With over 20 years of experience in KM implementation consulting and sales, I’ve found that many challenges begin early with a flawed system evaluation approach. Choosing an unsuitable vendor often sets the stage for persistent issues through the implementation and beyond that are costly to remedy. Recent buzz and uncertainty surrounding generative AI only appears to be amplifying the problem. This post explores how to leverage the most effective evaluation team and assessment techniques to discover new innovations while confidently selecting a KMS that is easy to implement, intelligently integrates with emerging technologies, and delivers long-term value across the enterprise.
Assemble an Evaluation Team With a Mixture of Strategic, Operational, and Technical Staff From Throughout the Enterprise
While one particular group may be experiencing a pressing need for a KMS, its leaders must ensure their goals align with the company’s overall mission and objectives. Any shortcomings in the current system should be clearly linked to challenges in supporting company-wide priorities like cost savings, revenue generation, customer loyalty, and employee efficiency or proficiency gains. The proposed new system must help directly enhance these critical outcomes.
Astute leaders not only align with the company’s overall knowledge strategy but also actively contribute to shaping it. This approach helps with short-term tactics, such as budgetary funding, while also fostering essential, ongoing collaboration with other teams to ensure long-term success.
Executive sponsorship for a KMS should go all the way up to the C-suite. This oftentimes includes multiple individuals, such as the Chief Customer Officer and Chief Information Officer. Individual program ownership should be represented by IT and the business. Many KM programs experience failure without these critical alignments.
A KMS usually integrates with a wide variety of technology, including user engagement platforms, content repositories, analytics, and core IT infrastructure while also impacting key operational processes. Many KMS evaluations underestimate this complexity and encounter late-stage obstacles that could have been avoided through proper representation from a diversity of all stakeholders. Commonly overlooked but critically important stakeholders include information security, data privacy, and change management.
Your team must determine how the system evaluation progress and long-term success will be measured and allocate appropriate resources to do so. Success metrics must align with corporate-wide goals and be reported back to all stakeholders, especially executive sponsors. Successful communication shares critical highlights without overwhelming or confusing stakeholders with unimportant or unrecognized terms.
Figure 1 shows a recommended team structure for a KMS evaluation project for customer service:
Ensure a Comprehensive and In-Depth Understanding of Knowledge Management System Capabilities
A KMS is composed of two primary capability categories: knowledge delivery and knowledge lifecycle management.
Knowledge delivery should be more than a reactive rendering of search results in response to a manual user query. Curation and contextualization must work together to deliver personalized and proactive knowledge that is displayed directly within a critical use case or process flow. Common examples include self-service case deflection or helping a call center agent better solve an issue by automatically displaying knowledge relevant to the case being worked. More innovative approaches include things like intelligent swarming, conversational user interfaces, and in-product experiences – all of which require extensive collaboration for design, implementation, and measurement of success.
Knowledge lifecycle management needs can vary wildly between teams. Some wish to author all content within a single system while others need a KMS that can index content from a wide variety of other repositories, a feature that not every KM vendor excels at. Some teams require advanced authoring capabilities that certain vendors do not offer or only support through customization while others will experience negative side effects caused by unnecessary complexity. These considerations are especially important for groups who author within another workflow, such as the Knowledge-Centered Support (KCS) methodology. In all cases, KMS analytics must go beyond the common problem of displaying “true but useless facts” and instead identify trends or knowledge gaps that can be easily actioned upon for demand-based knowledge creation and curation.
Focus on the “Know How” and “Know Who” of Knowledge
Knowledge management is often focused exclusively on the “know how” of an organization: documented knowledge. And yet a KMS will often possess an extensive inventory of how individuals have interacted with knowledge, including actions like:
- Who authored a high-value document that regularly deflects cases about a specific issue?
- Which users are collaborating on a uniquely challenging case?
- What are all of the documents that an individual has consumed?
This type of “know who” information is incredibly valuable to implicitly identify expertise, especially since explicit mechanisms are rarely comprehensive or up to date.
Processes like a knowledge risk assessment can identify areas of limited expertise that triggers a request to create formal documentation for wider distribution or capture potential lost knowledge before an employee departs the organization. Collaboration systems can leverage this expertise to perform actions like determining who to invite into an intelligent swarming process.
Most KMS vendors and client deployments significantly underestimate the value of this expertise data. Your organization should not make that same mistake. Even if not formally productized by the vendor, this data can typically be retrieved easily, oftentimes without any licensing costs.
Skillfully Leverage Innovations in Generative AI While Mitigating its Inherent Risk
Recent innovations in Generative AI (gen AI) have created unique opportunities for KM leaders, with most corporate boards directly recommending increased enterprise adoption of this new technology. KM leaders must educate others within the company about the critical importance of core knowledge management capabilities as they relate to gen AI.
Despite the widespread enthusiasm, there are real concerns with gen AI, including hallucinations, sharing sensitive data to a third-party provider that could use your information to train their models, and prompt injections that might allow an attacker to bypass security controls and extract confidential information or instruct the AI to produce malicious outputs. Organizations must leverage the same controls used for other enterprise tools to maximize the benefits of Gen AI while appropriately mitigating risks. This caution is especially warranted in regulatory environments like banking or healthcare.
Many KMS vendors already offer a variety of gen AI features, including capabilities such as:
- Generative answering, where responses are systematically and automatically generated in a human-like format thanks to the use of large language models (LLMs) and vector-based search, oftentimes referencing the exact document sources to increase the credibility of a generated answer.
- Automatic knowledge creation that can use the information contained in resolved cases and partial answers from other knowledge sources to aid in the creation of the first draft of new content.
- Rewriting content for improved findability or easier consumption, including simpler use cases like language translation to more advanced scenarios like converting a highly technical document into simpler terms for novice reading or summarizing a complicated policy document into a set of easier-to-follow bullet points.
Some vendors are more mature than others in the actual deployment and client value realization. Some features incur additional costs, while others may be included at no additional charge. All functionality should be analyzed for the value it delivers to your organization and your ability to understand critical aspects, such as how the model is trained and how the solution can be audited.
Don’t let the complexity and potential risks deter your organization from taking advantage of the incredible power that this new technology offers. Focus on high-value, ready-to-implement functionality with acceptable risk levels for ensured success. Various tips are offered in the next section to properly evaluate the authenticity of vendor claims and avoid future cost surprises to ensure that a selected KMS will help your organization’s knowledge management and generative AI strategy thrive.
Employ More Creative Techniques for Assessing Vendors and Discovering Innovation
Many technology vendor evaluations, KMS or otherwise, start with a Request for Information or Proposal (RFI or RFP). This approach typically contains a lengthy spreadsheet of functional requirements that are based on an evaluation team’s historical understanding rather than current market trends or future possibilities. Negative consequences can easily follow, even if the evaluation team doesn’t realize it, due to a controlled and mundane process that lacks opportunities for innovation and discovery.
A more fruitful evaluation starts by communicating your business strategy and prioritized, high-level objectives to KMS vendors to solicit their most creative ideas on how to best meet those needs. Vendors should also be encouraged to propose important topics even if they don’t map to the stated objectives. Collecting this information prior to creating a detailed list of requirements, especially when gathering input from multiple vendors, will reveal innovative approaches, offer valuable insight into how well each vendor aligns with industry trends, and help future-proof your organization’s KM strategy.
Each vendor demo should map all proposed functionality to your business objectives and strategy. Demonstrating on live client sites that your team can later access is preferable to sales sandbox environments. Always seek verifiable references regarding the value metrics clients have achieved. Vendors should even show how they utilize functionality within their own customer support and success organization.
It is important to assess the availability, maturity, and ease of implementation for critical functionality. Support documentation will help prove that a feature is fully productized and available out-of-the-box (OOTB). It also confirms that the implementation and ongoing administration approach aligns well with your organization’s expectations.
If a proof of concept (POC) is necessary, importance should be placed on truly new concepts that need to be proven by your team because they have not yet been validated by existing client deployments. Common KM system risk points that may warrant a POC include:
- Third-party user engagement platforms that have never been integrated with.
- Content repositories lacking OOTB connectors.
- Unique process flows or regulatory risk points.
- Unusual contractual obligations, such as publishing or response times.
A POC can be quite extensive, or it may be limited to a quick, one-time collaboration session between your team and the vendor’s technical staff. If hosting multiple POCs, a process should exist to apply learnings from one vendor’s POC to another.
Free resources like LinkedIn Groups and G2 Reviews are a great supplement to reference calls with vendor customers. Your organization may also have access to briefings with analysts at research companies like TSIA, Gartner, and Forrester.
Contractual negotiations must not overlook the unique aspects of a KMS, including issues like:
- Will any features, especially gen AI, incur an unexpected cost in the future? This cost could come from the KMS vendor or a third party, such as charges for API calls sent to an integrated system.
- Can any features, especially integrations with third-party systems like content repositories and user engagement platforms, be severely degraded or even deprecated due to changes made by that third-party vendor?
Conclusion
A knowledge management system is an essential technology for organizations that can be challenging to select and implement. This is especially true when needing to integrate with a diversity of other technologies. Achieving long-term business value often proves to be the biggest problem. The recommendations provided are proven to help facilitate a positive KMS implementation that intelligently harnesses the power of new technologies like generative AI for significant and sustained success throughout the enterprise.
Need Help With Your KM System Selection?
Feel free to contact me if I can be of any assistance in selecting the best knowledge management system for your organization…or perhaps remedying a selection process that didn’t follow the above recommendations 🙂
Leave a Reply