Evaluation Studies: A methodological overview

Evaluation studies are delivered via a plethora of user research methods. In this article I showcase the methodological process from initial analysis and instrumentation to reporting activities.

Evaluation studies are typically undertaken in accordance with a Methodology Handbook (MH) that is devised during the project initiation phase. The following table presents an overview of the methods that were deployed in a phased approach during a five (5) month period, from July to November 2017 for the evaluation of three key website services.

The table presents the complete list of methods used to deliver three evaluation studies.

The project started with an initial content audit that took place to allow the Service Team members to familiarise with the website's content, the CMS and the objectives of the study. Following this onboarding phase, the evaluation methods were rolled out gradually to web audiences. The majority of the outcomes of the methods were available towards the end of October 2017, when the Service Team undertook the synthesis of the recommendations and the compilation of design recommendations.

For each of the methods used in the project an initial analysis took place in collaboration with the Client in order to define the instrumentation of the study, the planning of its roll-out and key promotion activities to recruit participants. Following the initial analysis, the data collection phase aimed to deliver the studies (collect data, perform user sessions, etc.). Finally the reporting phase compiled the results of each method in a stand-alone report presenting the study’s detailed findings.

Methodological overview

In the context of the given assignment a set of fit-for-purpose behavioural and attitudinal research methods were used to assess and improve the usability and user experience of an existing website.

Expert Review

An Expert review pertained the heuristic evaluation of the user interface for compliance to well-defined criteria (heuristics) by at least two usability experts. Findings were later used to improve the usability of the given website.

SEO and Web Analytics Audit

A structured inspection and assessment of the website in question aimed to describe the following business elements:

Online User Surveys

User Surveys were utilized to gather data concerning the actual usage of the website and the available services and features. The main objectives were the following:

User Interviews and Focus Groups

In the chosen methodological framework, user interviews provided direct contact with external users and agency stakeholders and staff, aiming to describe who they are and what are the main tasks they try to accomplish as well as their concerns and needs. User interviews were semi-structured and were carried out remotely and through planned site visits.

Focus groups were deployed (a) in order to engage multiple users in thought-shower sessions especially for new/emerging needs and (b) when complex requirements needed to be discussed or requirement conflicts between project stakeholders needed to be resolved.

Personas and User Journeys

Personas are realistic representations (profiles) of the key web audience segments of the website and are used to describe and communicate the users' expectations of the website.

Example of a persona.

Personas were built through a review of web analytic data, user survey responses and primarily through direct user interviews. Personas were used as a validation tool in the design stages of the project (supporting the evaluation of new requirements, layout, targeted campaigns, labelling and informing the website's content/communication strategy). Depending on the complexity of the assignment the personas can be extended through the definition/mapping of the most important actual user journeys ( User Journey Mapping ) to uncover possibly problematic user journeys and define the conversion funnels. User Journey Mapping was deployed to assess the effectiveness of complex hierarchically-structured websites and multi-step conversion funnels.

Example of User Journey mapping.

Prototyping

Low and High Fidelity Prototyping was employed as a technique to quickly visualise new ideas, alternative concepts and review the usability and stylistic approach to addressing identified usability issues and elicited user experience insights.

Example of wireframes produced in Figma.

Tree Testing

The effectiveness of the website's actual sitemap was validated with IA Validation software in order to answer structure and labeling-related questions and address findability and discoverability issues. The results lead to the creation of an improved sitemap in the form of a flow diagram that identified all pages and showed the pathways linking each page.

Usability Testing

Usability Testing involved the design and implementation of formal usability experiments that aimed to collect suitable qualitative and quantitative data in order to uncover important issues relating to the usability and accessibility of given user scenarios. Usability test sessions were organised with a fixed number of users (approximately 10) who were asked to perform a number of typical test scenarios or tasks. The Usability Testing approach that is typically employeed can be moderated or un-moderated, performed in a usability test lab or remotely through the participant's computer.

The objective of usability testing was to derive actionable recommendations related to the usability criteria of learnability, efficiency, memorability, errors and subjective user satisfaction.

Acknowledgements

Many thanks to Eworx for giving me the opportunity to be a part of a team of unique UX practitioners and most of all amazing individuals.

Watch your habits; they become character. Watch your character; it becomes your destiny.