Project detail
Data Generation Tool
Usability Test
CITIC Technology IP design
IP Design, Product Design
2023
CITIC Technology
About Synthetic Data Forge
Synthetic Data Forge is an online fake (Synthetic) data generation website targeting Data Scientists and Human Resource Officers. This Usability Test aimed to improve their beta version of the website.
Research Goal
Are users able to complete the tasks efficiently and without errors or confusion?
Which elements in the interface create resistance that mostly affects users unexperienced in using the tool?
Methods
This usability test includes 5 different methods, as listed below.
Jump to each section by clicking the names below.
User Interview
Goal
Test out How intuitive is the system for new users to navigate and complete essential tasks
Recrutement
Recruit 5 professionals in Data field
Participants handling sensitive governmental and industrial datasets
Participants lacked access to Reveal Global's production environment due to confidentiality constraints.
Method
used direct outreach and snowball sampling to recruit participants
semi-structured interview guide for data collection
Questions consists of open-ended questions focusing on the challenges in handling datasets and opinions on synthetic data and its applications.
Interview Script
Coded Interview Quotes
Key Findings
The data tools they commonly use have fixed formats that cannot be freely adjusted according to project needs. This rigid format makes it difficult for them to generate data that truly fits their work requirements
Manual data cleaning was one of the most time-consuming jobs for a data professional.
The tools lack the ability to annotate data fields or data logic, making it difficult for business and development teams to communicate the meaning of the data
The inability to capture real-world scenarios due to the lack of “unique datasets” available for those types of scenarios.
User Survey
Goal
Understand adoption, uses, issues, and trust of synthetic data.
Gather insights from HR professionals, government data scientists, and engineers through surveys.
Method
Created a draft of the survey questionnaire with over 20 questions
Seeking participants for feedback on the survey
Convenience sampling to gather participants with relevant knowledge and experience
Invited five individuals to participate in a pilot survey and provide feedback.
Survey Questions
Heuristic Evaluation
Goal
Identify usability issues that may affect user interaction and provide actionable recommendations for improvement.
Method
Team discussion to prioritize evaluation focus areas.
Identified key screens and flows based on their importance to user experience.
Used Nielsen’s 10 Usability Heuristics as evaluation criteria.
Severity Assessment Based on impact on user experience and task completion
Analyzation Script
Nielsen’s set of heuristics (Nielsen, 1994) that we used for our heuristic evaluations
Team members' first note-taking
Most Risky Issues
Key Findings & Recommendations
Finding: System status lack of color contrast, causing low readability.
Suggest: Modifying the font color of the header or the contrast. Develop Structured layouts, icons, and visual cues
Finding: Inconsistent UI elements, such as "logout" button appears in different locations in different pages.
Suggest: Bring Log-out button to the top right. Develop standardization UI components.
Finding: Absence of error prevention and recovery notification when user types in formula variable & invalid input numbers.
Suggest: Add contextual help features, like question mark icons or an in-line help sidebar, Improving error recovery through explicit error messages and inline notifications
Finding: No contextual assistance or intuitive documentation, causing difficulty in understand features.
Suggest: Including a preview function with example real-world data examples
Comparative Evaluation
Goal
Identify how Synthetic Data Forge compares with competitor products for it's target users and RGC in the following respects:
Desirability (customer needs and market demand)
Feasibility (technical and operational capability)
Visability (business sustainability and economic value)
Method
Categorizing competitors based on their relevance to our product, considering:
Direct Competitors
Products providing the same primary function and targeting the same audience.
Inirect Competitors
Products offering similar functions but aimed at different user groups or purposes.
Partial Competitors
Products sharing some core features but differing in scope.
Parallel Competitors
Products in different domains but with comparable interaction styles, workflows, or user experience patterns.
Analogous Competitors
Products or services in entirely different domains, but similar in operational style or interaction approach.
Criteria
What core features and capabilities does the product offer?
Who is the primary user or customer of the product?
How do users typically engage or interact with the product?
Can the product handle growth and high data volumes?
How is the product priced and what monetization strategies are used?
What level of product adaptation is available?
How closely does the product generate synthetic data mimicking real-world data?
Comparative Evaluation Charts & Findings
Usability Test
Goal
Identify how Synthetic Data Forge compares with competitor products for it's target users and RGC in the following respects:
Desirability (customer needs and market demand)
Feasibility (technical and operational capability)
Visability (business sustainability and economic value)
Method
Conducted 1 usability pilot test and 5 usability tests
Participants had experience working in data field, but never used Synthetic Data Forge before.
Each test included a pre-test survey, 6 tasks that focused on the search, and wrapped up with a debrief interview.
Participants
Work in data field (Data scientist / Data analyzer / Data student)
Never used Synthetic Data Forge before
Pre-test Survey
Consists of 8 questions
Asking about demographic information of the participants.
Tasks
Focusing on the commonly used features of the Synthetic Data Forge website.
Record think-out loud thinking process
Calculate time spent on each task & subtasks
Debrief Interview
Consists of 8 questions
About participants' thoughts on usability, visual design, experience and general understanding of the site.
Coded as documentation for analyzation
User Testing Recording
Scripts & guides during the user test (including Tasks, Task Instruction File, and User Content Form
Data Log-in Form (Note system to capture, measure, and analyze users' every small moves)
Performance Data Form
Unclear navigation and layout
Finding: Participants found the navigation confusing, often struggling to locate key pages like instructions or download, and had difficulty moving smoothly through the workflow.
Suggestion: Restructure the navigation to reflect the linear data generation workflow, add breadcrumbs or a stepper component to visually indicate user progress
Lack of Visual Feedback and Confirmation
Finding: Users lacked clear feedback after saving or uploading actions, causing confusion and repeated manual input.
Suggestion: Introduce clear, in-line success messages or toast notifications. Provide visual affordances such as progress bars or icons that confirm a completed step.
Difficulties in Correcting or Editing Inputs
Finding: Users found it difficult to correct or edit inputs, as the interface lacked clear options for going back or safely adjusting previous entries without losing progress.
Suggestion: Add a persistent “Edit” or “Back” button in each step, add auto-save functionality or clearer “save” states
Minimal Instructional Support for New Users
Finding: New users lacked orientation upon login and had difficulty mapping tasks to system features.
Suggestion: Create a welcome dashboard that provides a brief overview and quick links to key sections. Offer optional onboarding, Add hover-based tooltips to clarify icon meanings and terminology
Visual Design is Functional but Flat
Finding: Users appreciated the platform’s simplicity but found it visually bland, making it harder to focus on key areas or actions.
Suggestion: Incorporate a more vibrant and accessible color palette， emphasize primary actions
