Evaluation Instruments

Level One Evaluation

At the completion of the each of the modules, participants will be asked to complete a level one evaluation.  All level one evaluations will be computer based.

Likert Scale Questions

Please indicate the extent to which you agree/disagree with the following:
[scale from 1-5 provided; 1 = strongly agree, 3 = neutral, 5 = strongly disagree]

1.       The purpose of the training module was clearly stated.

2.       The content was presented clearly.

3.       The training module was engaging.

4.       The content of the module was applicable to me.

5.       The module was easy to navigate.

6.       The module was delivered without any technical issues.

Open-Ended Questions

7.       What was most useful in this training?

8.       What was least useful, and why?

9.       Additional comments.

Level Two Evaluation

At the conclusion of each module, participants will be required to complete a level two evaluation which will be administered through the company LMS.  The quiz for each module most be successfully completed (scoring at least an 85%) to display mastery and receive credit for completing the course.

Navigation & Functions

Participants will be evaluated on their answers to the following questions:

1.        List the primary functions used in Lotus Notes.

Answer: The primary uses of Lotus Notes at Company XYZ is for email, calendar, information storage in the form of databases, and electronic forms also within databases.

2.        Describe the structure of the database system.

Answer: Databases are grouped according to the functional group who is the “owner” of said database (e.g., Travel and Expense database is grouped with all other Accounts Payable databases).

Searching in Databases

Participants will be given the following scenario and a simulation screen in which they must successfully complete the steps based on the given scenario:

You recently went on a business trip for Company XYZ and need to submit an expense report to apply for reimbursement.  This is completed through the Travel & Expense Database of which you have never accessed or used.  Using the simulation Lotus Notes site on the screen, demonstrate how you would go about searching for and accessing this database in order to successfully submit your travel expenses.  Clicks are tracked, so be cautious to clicking erroneously as that may result in a failing score.

Level Three Evaluation

After 60 days of completing the Lotus Notes modules, the participants will be provided with an electronic survey, similar to the one provided in the performance gap analysis, to measure if the training met their needs, the transfer of knowledge to their job tasks, and if the learning objectives were met.

In addition to this survey, a report will be gathered to analyze the IT Help Desk requests submitted for Lotus Notes assistance.  Cross checking will take place to see if associates who have completed the course still required IT assistance in the material covered through the training.  If it is found that these individuals did still need help, an interview will be held with them to determine what they needed help with, and then determine if any changes need to be made to the material and/or delivery of the content.

Level Four Evaluation

The performance gap analysis conducted for this project indicated the following current and desired states:

Current State - Of the 576 respondents, 13% reported understanding how to search for databases in Lotus Notes.

Desired State - 100% of associates should be able to easily and quickly search for required databases in order to efficiently perform their job duties.

Delta - Based on the sample of respondents, the percentage increase in database searching capabilities should be 87%.

In the first year of this training being implemented, data based on the level three evaluation will be collected quarterly to determine if Company XYZ is approaching its desired state of performance.  At the end of the first year, the data will be analyzed to determine if further action needs to be taken to close the performance gap or if the established path meets the business needs.

 

Design Plan

Rationale
The Lotus Notes Database training is designed to offer as-needed training to new and existing Company XYZ associates to better prepare them for the heavy employment of this system in their role.  The Lotus Notes system is used by all associates for email and calendar functions, as well as highly relying on the database function as a means to provide associates with information and track and submit information (e.g., travel and expense reimbursements, tuition reimbursements, IT Help Desk requests, etc.).  Many associates join with familiarity to systems like Outlook and Gmail, so there is a significant learning curve to overcome as they come onboard. 

Target Population

Newly hired associates will be the primary focus of this training, as a majority of incoming associates are completely unfamiliar with the functionality of Lotus Notes and the databases it contains.  At the same time, this training will be available for existing associates as a means to refresh their knowledge or for just-in-time training.  With this in mind, the population will range from those 20-70 years of age with varying levels of technological knowledge.

Course Description

The Lotus Notes Database training consists of two-15 minute online, asynchronous modules provided through Company XYZ’s learning management system:

Modules 1: Navigating Lotus Notes (15 minutes)

Participants will understand the structure and layout of the Lotus Notes system and the functionalities that exist.

Module 2: Searching in Lotus Notes Databases (15 minutes)

Participants will understand and demonstrate how to search for information and databases within the Lotus Notes system.

Objectives

Module 1

Terminal:
Given the Navigating Lotus Notes module and a properly functioning Lotus Notes system, the Company XYZ associate should be able to correctly identify and explain the basic functionalities (e.g., email, calendar, databases) and structure of the system within three attempts.

Enabling:
Given the Navigating Lotus Notes module and a properly functioning Lotus Notes system, the Company XYZ associate should be to correctly navigate to the basic functionalities within three attempts.

Given the Navigating Lotus Notes module and a properly functioning Lotus Notes system, the Company XYZ associate should be to correctly explain the purpose to the basic functionalities within three attempts.

Given the Navigating Lotus Notes module and a properly functioning Lotus Notes system, the Company XYZ associate should be to correctly explain the structure of the system within three attempts.

Module 2

Terminal:
Given the Searching in Lotus Notes Databases module and a properly functioning Lotus Notes system, the Company XYZ associate should be able to correctly search for specified information and databases in the Lotus Notes system within three attempts.

Enabling:

Given the Searching in Lotus Notes Databases module and a properly functioning Lotus Notes system, the Company XYZ associate should be able to correctly explain how to search for specified databases in the Lotus Notes system within three attempts.

Given the Searching in Lotus Notes Databases module and a properly functioning Lotus Notes system, the Company XYZ associate should be able to correctly demonstrate how to search for specified databases in the Lotus Notes system within three attempts.

Given the Searching in Lotus Notes Databases module and a properly functioning Lotus Notes system, the Company XYZ associate should be able to correctly explain how to search for specified information in the Lotus Notes system within three attempts.

Given the Searching in Lotus Notes Databases module and a properly functioning Lotus Notes system, the Company XYZ associate should be able to correctly demonstrate how to search for specified information in the Lotus Notes system within three attempts.

Prerequisites
WBT Narrator must:

·         either have consulted a SME or be a SME in Lotus Notes functionality and usage.

Participant should:
·         be unfamiliar with or unsure of using Lotus Notes functions
·         must be English speaking
·         must be familiar with the use of email and electronic calendars.

Evaluation Strategy

Level 1 Evaluation
At the conclusion of each module, the participants will be presented with an electronic survey created on a Likert scale (1 = Strongly Disagree, 5 = Strongly Agree) in which they will be asked to assess the course’s narration, interactivity, content presentation, content usefulness, and overall satisfaction.

Level 2 Evaluation
Each module includes a level 2 evaluation:

Module	Evaluation
Navigating & Functions	Open-ended questions
Searching in Databases	Scenario Simulation

Level 3 Evaluation
60 days following the completion of the course an electronic survey will be administered to the participants in which they will rate their level of understanding and use of the concepts covered in these training modules. 

The names of participants who completed the course, along with their scores on the course, will be compared with the IT Help Desk tickets submitted for Lotus Notes assistance to determine if the transfer of knowledge did occur or if the training needs to be re-examined.

Level 4 Evaluation
Quarterly data will be collected in the level 3 evaluation to determine if participants who complete this training meet the desired state of performance.  Based on that, it will be determined if the training needs to be reevaluated at that time or if it is appropriate until future auditing.

Deliverables

·         Narrator script
·         Screen captures of the Lotus Notes system for the web-based course
·         Navigating Lotus Notes quick reference guide available at the end of the module and in the LMS
·         Searching in Lotus Notes quick reference guide available at the end of the module and in the LMS

  

Training Needs Analysis

Problem Statement: Associates are unable to find the applicable Lotus Notes databases, resulting in delayed processes and information retrieval.

Conclusions from Front End Analysis: The front end analysis uncovered several factors including out of date software, out of date databases and information, and no logical organization to the database structure.  The recommendations for these root causes have been presented to the client and are being acted on accordingly with no further involvement from the consultant.

The front end analysis also presented items that may require a training intervention: associates do not understand how to search for databases and information in Lotus Notes that they do not already have access to. 

A training needs analysis was conducted to identify the training needs for associates to be successful in this task.  It sought to identify:

• What databases and information associates typically need
• What knowledge and skills associates typically struggle with or lack

Input	Purpose	Method
2014 IT Help Desk Requests	Determine what information and databases are typically requested and how frequently	Data Analysis
Interview with  associates	Determine the skills and knowledge associates struggle with or lack	Survey

 Instruments

2014 IT Help Desk Requests

Data of year-to-date IT Help Desk requests was pulled for all items relating to Lotus Notes.  The purpose of this was to see who submitted these requests (new associates vs tenured, certain job functions, etc.), what the most common requests were for Lotus Notes, and how frequently these requests occurred.  This data would provide more insight as to what information and assistance associates were needing and if this was something that could be resolved through training. 

Associate Survey

Surveys were conducted via Survey Monkey with a spectrum of associates representing each functional group, job level, level of tenure with the company, and familiarity with Lotus Notes.  The goal behind this instrument was to get a deeper perspective of what associates use Lotus Notes for, what they do when they need to find information or a database, their general attitude toward the system, etc.

Questions and discussion points included:

1.       List the primary functions you use in Lotus Notes.

2.       List the databases you rely on the most.

3.       Of those most regularly used databases, how frequently do you use them?

4.       Of those databases you use most, how did you receive access to them?

a.       Searched for myself

b.       Another associate (including IT) sent the database link

c.       Other

d.       A combination of all of the above

5.       If you cannot find information or a database you need, how do you handle it?

a.       Ask another associate to send the database link

b.       Submit an IT Help Desk request

c.       Continue trying to search on your own

d.       Give up

e.       Other __________________________________

6.       Do you know of any resources (e.g., videos, quick reference guides) to assist with using Lotus Notes?

a.       Yes

b.       No

7.       Using the scale below, rate the ease of finding information and databases in Lotus Notes

1 – Extremely Difficult

2 – Difficult

3 – Neutral

4 – Easy

5 – Extremely Easy

8.       If you could change something about Lotus Notes, what would it be?

9.       Any other comments you would like to provide about Lotus Notes?

Results

2014 IT Help Desk Requests

Upon reviewing the data gathered regarding Lotus Notes linked requests, the following was found:

·         Year-to-date 17% of all help desk requests have been Lotus Notes related.

·         There was no significant difference in the number of help desk requests submitted across functional groups.

·         At least 4 Lotus Notes related requests were submitted per business day.

·         Of these requests, 32% were related to assistance in searching for a database or requesting access to a database.

Associate Survey

·         The primary uses of Lotus Notes ranked as follows:

1.       E-mail & Calendar

2.       Databases

3.       Instant Message

·         The databases used most ranked as follows:

1.       Travel & Expense

2.       Community Information

3.       Procurement

4.       Asset Management

5.       HR Office Suite

·         41% of respondents said they use databases daily, 33% reported using databases weekly, and 26% reported using them irregularly

·         73% said they received access to a database from another associate (including IT), 21% said they accessed a database from searching on their own, 6% reported other

·         87% reported that if they cannot find information or a database on their own that they either ask another associate or submit an IT Help Desk request.

·         90% reported not knowing of any resources to guide them through using Lotus Notes.

·         89% of respondents reported the ease of finding databases and information as difficult or extremely difficult

·         93% of respondents reported wanting to change something related to usability (e.g., search function for databases and e-mails, user interface, etc.).

Findings

Currently there is no publicly dispersed resource, other than the IT Help Desk, that associates can reach for to assist them in navigating Lotus Notes.  A majority of associates reported having difficulty using Lotus Notes.  Typically if associates are looking for information or a database and cannot find what they need, then they reach out to someone else for help, which may result in higher help desk volume for IT and distractions for other associates.    There clearly is a need for some guidance as to how to navigate and successfully use Lotus Notes, especially when it comes to finding information.

Recommendations

Based on the data collection and findings, the following are recommendations to minimize this performance gap:

·         Create short web-based videos that associates can access through the LMS and watch as needed to see how to perform a function in Lotus Notes.

·         Offer job-aids for associates to use to walk themselves through the steps to complete a task

·         Schedule monthly “Lunch and Learns” where in an informal setting associates can see Lotus Notes demonstrations and ask questions

·         Have IT evaluate if Lotus Notes can be customized for a more intuitive user interface for increased usability.

 

Front End Analysis

Client: Company XYZ is a real estate investment trust in the multi-family housing industry with offices and communities residing all along the East and West coasts.

Background Information: Company XYZ uses IBM Lotus Notes (LN) for company email and also heavily relies on the database capabilities to house policies, procedures, and electronic forms for all different purposes.  In the 10+ years that XYZ has used LN, databases have grown and policies/procedures have changed, resulting in out-of-date information, a bogged down system, and confusion of what information takes precedence. Associate dissatisfaction with the functionality, maintenance, and navigation of LN came to light in Q1 of 2014 through regional town halls.

Methods: To obtain further insight regarding the specifics of the frustration with LN, data was gathered utilizing an electronic survey administered to all regional and corporate office associates (approximately 1,200 associates) with a response rate of 48%.  This survey consisted of 15 Likert scale questions and 3 open-ended responses.  The respondents expressed their frustration with finding databases they need for processes and finding databases that have information they are looking for.  They also expressed concern with information in the applicable databases not being up to date or matching information in other systems.

Problem Statement: Associates are unable to find the applicable Lotus Notes databases, resulting in delayed processes and information retrieval.

Performance Gap Analysis: Currently 13% of respondents reported understanding how to search for databases and information in Lotus Notes.  In the future, 100% of associates will understand how to search for databases in LN.

Factors Influencing Performance: The following elements have been identified as potential influences in this performance gap:

Resources:
·         Technology is not always consistent between associate computers (e.g., not all associates have the same/most recent version of LN)

Information:

·         Some departments send new associates an automate email with links to the basic databases (e.g., HR/Benefits, Travel Policy), but not all.
·         There is nothing  to aid associates in how to search for databases.

Motivation

·         After spending a lot of time attempting to search for information or databases, many associates get frustrated and give up trying to look when it is not something needed for a job duty.

Structure and Process

·         Some reported approving the structure of the HR Office Suite database in that it was easy to navigate to find either the information they needed or who to contact for further assistance.  
·         No logical organization to the databases in the search function, simply appear alphabetically.

Knowledge: 
·         Associates do not understand how to search for databases and information in LN that they do not already have access to.

Findings & Recommendations

Factors Findings Recommendations Resources Technology is not always consistent between associate computers (e.g., not all associates have the same/most recent version of LN) Have IT run diagnostics of which computers are out of date, and do a mass push of updates to all computers so they're all alike. Information Some departments send new associates an automate email with links to the basic databases (e.g., HR/Benefits, Travel Policy), but not all. As part of onboarding, have a mass list of all positions & applicable/necessary databases & systems access, so this information is automated & sent to the new associate on their first day of work. There is nothing  to aid associates in how to search for databases. Create a short tutorial or job aid demonstrating the different ways associates can search for databases and information in LN. Motivation After spending a lot of time attempting to search for information or databases, many associates get frustrated and give up trying to look when it is not something needed for a job duty. Inform associates that the company is aware of the frustations with LN and is addressing the problem with potential solutions from this list. Structure and Process Some reported approving the structure of the HR Office Suite database in that it was easy to navigate to find either the information they needed or who to contact for further assistance.   Determine what it is about the structure of the HR Office Suite database that associates prefer, and see if there's a way to copy this structure into other databases. No logical organization to the databases in the search function, simply appear alphabetically. Create a file-type organization to the databases where associates can click on the overall topic, say Financial Services and drill down to the subtopics and databases within that. Knowledge Associates do not understand how to search for databases and information in LN that they do not already have access to. Create a short tutorial or job aid demonstrating the different ways associates can search for databases and information in LN.
	.

Week 5 - Root Cause Analysis

Problem Statement
A significant number of associates at company XYZ are unable to find the applicable Lotus Notes databases, resulting in delayed processes and information retrieval.

Methods

An electronic survey was administered for associates to offer feedback regarding the process of searching for Lotus Notes databases required for job duties as well as for general organizational information.  After analyzing the reported data for trends relating to the categories below, 10 associates were selected - 5 of whom expressed a mostly positive/indifferent review and 5 of whom expressed a negative review- to then provide further information as to their reasoning for their report.  These conversations were conducted either through phone calls or email and asked the associate to elaborate, with examples if possible, as to why they reported what they did. 

Factors Influencing Performance

Resources:
·         Technology is not always consistent between associate computers (e.g., not all associates have the same/most recent version of LN)

Information:

·         Some departments send new associates an automate email with links to the basic databases (e.g., HR/Benefits, Travel Policy), but not all.
·         There is nothing  to aid associates in how to search for databases.

Motivation

·         After spending a lot of time attempting to search for information or databases, many associates get frustrated and give up trying to look when it is not something needed for a job duty.

Structure and Process

·         Some reported approving the structure of the HR Office Suite database in that it was easy to navigate to find either the information they needed or who to contact for further assistance. 
·         No logical organization to the databases in the search function, simply appear alphabetically.

Knowledge:
·         Associates do not understand how to search for databases and information in LN that they do not already have access to.