Meaningful Metrics to Quantify ROI for KM and Enterprise 2.0 Deployments

By Joseph Raczynski

Clark R. Cordner – Orrick, Herrington & Sutcliffe LLP
Charlotte Herring – The US Judge Advocate General’s Corps


Recently some select firms have forgone the necessity to prove return on investment (ROI), for KM, portal and Enterprise 2.0.  However for the vast majority, ROI justification remains a constant.  In this session at ILTA, the panel examined the basics of metrics, how to measure productivity rather than busy-ness, how to measure engagement, and concrete ways to measure portal and Enterprise 2.0 applications.

As a baseline understanding for this discussion, metrics are numbers to gage progress, i.e. a quantifiable means to measure if there is a move from one point to another.  Firms engage in this activity to evaluate success and decide what to fund.  Simply stated they create these objectives to see if they are materially advancing the goals of the organization.

Metrics as a lens, key points to ponder:

  • Focus on the factors that have the most positive correlation for success
  • Ask yourself, “If success is… X, then I need to measure… Y.”
  • What are the business pain points?
  • Knowledge management is not always just technological issue, often it is education
  • What does success look like? Is it the bottom line for a firm?
  • KM tools are a critical component to success, choose wisely

Testing those metrics is a major component to your KM success.  Start with an educated guess based on your project goals and gather some basic data.  Also key to this empirical testing is to consult with focus groups and friendly stakeholders.  This not only verifies that you are working in the right direction; thus validating your work, it also tests your view verses others, establishing a complete balanced vision for the firm.  Lastly based on any feedback from these groups in testing these components, a firm should revise their methodology if necessary, adjust and repeat this process.

Portal metrics:

  • How many hits does the portal get? This will give the firm a glimpse into the level of interest and usefulness of the portal.
  • What kind of frequency do users return to the portal and for how long do they stay? These numbers will also assist in understanding interest, usefulness or possibly… forgetfulness.
  • Your firm should gather data on the identity of users. From these metrics you can figure out patterns of adoption; that is what age group, which practice group and what geographic area is using the portal?
  • Pull data on how current is the content is and to whom it is available. You can gather information on if people are using the information via a “like” button or if they are sharing it.
  • View the impact on other systems or tools. Has traffic to the other applications been reduced?  Do people refer to this tool or others?

Enterprise 2.0, aka Activity Streams or Social Media metrics:

  • Listen to the number of requests for access to this new tool. If you see something like Yammer jump in requests, you will be able to consider that as a major driver for implementation.
  • Once the tool is in place, view the usage over time. Often these new media applications are classified as a “five day wonder”, i.e. is this tool sticky, and will the usage change over time?
  • Another metric to consider is the pace of conversation for the tool. Is the tool a “go to” resource or peripheral to the organization?
  • Other areas to consider viewing metrics on are the numbers of links that people add to their work product, “retweets” and the time of day for usage
  • Lastly it is important to understand the impact on other tools. Is this new application used in preference of other tools, like email?

ROI is an impactful and important aspect of any business plan at law firms.  Examining the factors that have the most positive correlation for success is essential.  Once you have established a set of metrics which have been vetted by all major stakeholders, continue to test and understand that the mythology may need to be adjusted over time.  Those quantitative metrics will help guide the firm to success.

Enterprise Content Management: Integrating KM, RM and DM

By Joseph Raczynski

Massive collections of information are inundating law firms.  With multiple points of integration, interfaces, and varying disciplines this further muddies the waters.  This session at ILTA answered questions relating the issues of organizing and managing content across all firm repositories to reduce silos, improve data quality, facilitate KM and ensure full and accurate firm records.

The goal of most firms now is to devise an ECM (Enterprise Content Management) system which includes connecting all of the following silos of information: email, DMS, KM system, records management system and others.

This is a cumbersome process because each data set is unique and live in differing systems.  Another issue, rarely understood, is that storage of data is not cheap.  With the necessities of infrastructure, backing up and maintaining data and a facility is rather expensive.

Another area of consideration is risk management.  A firm must be able to craft policies about enterprise information management.  That is, how does the firm handle electronic discovery and where is it stored?  Who has rights to the data?  How should users get access to it and how long should it be stored?

Additionally law firms must consider the client-side challenges.  Typically if not planned out, users at a firm can experience time consuming search for documents.  They also can learn that it is difficult to “find” documents located in various locations.

In essence, firms must build “tridges”, multi-pointed bridges which connect all groups and information together. To do so a firm needs to:

  • Understand there will be some hard cultural changes at the firm
  • Find top level champions to be brought on board for support of the ECM
  • The organization must highlight (WIIFM) What Is In it For Me… so that all aspects of the organization can see how they fit in
  • Address user concerns by showing them what will be brought into via search on the ECM. By doing this you create a vested interest and they typically like seeing their work product pulled up in the searches

Ultimately when conjoining all of the various disparate pieces of information together, there are several point to consider:

  • The process has to be defined and paramount to this is communication. Among all of the parties which will be involved in the ECM process, making sure everyone is on the same page is critical
  • Define objectives: every aspect of what the firm wishes to have as a net result must be detailed
  • Redefine the process. Understand that many new processes will evolve which will then be disseminated across the organization
  • Understand that technologies can get in the way of each other and be prepared for these temporary setbacks.
  • Lastly accept and accommodate the delta; i.e. know that all of these things can get you so far but you still have a system that will need to be adjusted over time