Cloud Workspace Research
Overview
BMO is developing a Data & Analytics Marketplace to help data consumers easily find and request access to data assets such as schema, tables, columns, etc. Based on the strategic direction, I was tasked to understand the current process of setting up a workspace, find out what users think of that experience, and bring forward any recommendations to improve the process.
My Contribution
-
Conducted User interviews
-
Analyzed Data​
-
Stakeholder Presentation​
Team
-
Product Owner
-
Technical Product Manager
-
UX Lead (that was me)
Tools
-
Figma
-
FigJam
-
Confluence
Context
The workspace phase in the DnA Marketplace consists of setting up and managing workspaces. We were assigned to focus on the "Setup Workspace" process.
Mary's Journey​
-
She is a Data Scientist at BMO
-
She is working on a model to detect credit fraud
-
She needs data and heads to DnA Marketplace to begin her journey
Discovery
Access
-
Full view of access and controls
-
Request Data Assets
-
Track Access Requests
Workspace
-
Setup Workspace on demand
-
Load required data and tools
-
Monitor Cloud billing/usage
Governance and Knowledge Base
-
Quick Access to the Analytics and other Governance Directives
-
Knowledge articles published by BMO users
Publish to Production
-
Automated process to publish developed work to production
-
When applicable, publish developed work into Marketplace Analytics catalogue for re-use
-
Keyword Search
-
Browse Assets
-
Find the right data
-
Find data fast
-
Full view of Metadata
This case study is about the "Workspace" phase of Mary's Journey
Approach
Understand current process
-
Partner with the Technical Product Manager who leads the current solution.
-
Partner with the Product Owner of DnA Marketplace
Talk to users
-
Run User Interviews
-
Run Stakeholder Meetings
Recommend next steps
-
Present recommendations to the Managing Directors
Understand the current process
A manual form was required to be filled out by the users to request a new workspace. Based on the initial conversations with the group that was managing this form, my product owner and I found out that:
​
-
The users had to fill out an MS Word document to request a workspace.
-
The users had to answer 40+ questions in the document.
-
The document had no clear instructions around mandatory and optional questions.​
-
The questions in the document were based on the team's knowledge and were not tested with users yet.
List of all questions with their purpose and notes from the team
Current Form (MS Word)
We decided to
-
Eliminate a few questions after consulting with the concerned team and the stakeholders.
-
Create a prototype branded with BMO standards and test that prototype with users.​
User Interviews
Goal
To gather feedback from data scientists on the digitized version of the workspace creation form.
Research Questions
-
Do users understand the questions asked in the form?
-
What do users think is missing from the form?
-
What do users think of the look and feel of the form?
-
How easy or difficult was it for users to learn and use the form?
Research Details
6
Participants
User Interview
Method
60 mins
Interview Length
User Interview #1
User Interview #2
Key Findings
-
Data scientists need to experiment: The form assumes a data scientist has selected a model. The participants pointed out that when a data scientist is in an experimental phase, they are unable to answer many of the questions in the form. For example, questions about RAID approval, model inference, or the “type of model” they are building. 

-
Duplication in the intake form: The participants noted that project details (like Business Justification) are already shared with Model Risk and PMO, leading to work duplication. It was also noted that project details, such as the ones asked, were typically handled by a Project Manager, not a Data Scientist. 

-
Structured vs Unstructured data usage: The participants increasingly saw themselves working with unstructured data, such as documents or images. The form’s focus is on structured data.

-
Lack of utility in some questions: The participants felt certain questions lacked utility or had an obvious answer. 

-
Sensitive data deletion is a concern: The participants wanted further information about the PCI/PII disclaimer. They wanted to know how MLOPs would identify sensitive data. They also wanted to know what the scale of the deletion would be (all the data or just the sensitive elements). They expressed a concern that they would have to reload their data every 24 hours. 

-
Clarity code requirement raises billing and administrative concerns: The participants didn’t understand why two clarity codes were needed. Other concerns included: a junior data scientist should not be able to link a clarity code to a project, someone could enter the wrong clarity code, clarity codes can change during the lifetime of a project, and a “pay as you go” formula could make people very uncomfortable. 

-
New to AWS: Some participants felt they were new to AWS and would need more information to fully understand all of the questions.

-
SFTP to S3 is inefficient and potentially costly: Some participants noted that it would be cheaper and more efficient to work on-prem and not have to move data using this method.

-
Liked the look and feel: The participants liked the look and feel of the form and overall found it easy to use. Certain UI design updates like using checkboxes for multi-select and info icons for more context/information are needed. The participants were looking for an easy way to know the privileges of each persona.
Notes on FigJam
Recommendations
Problem
Recommendations
Lead
Support
Duplication of questions
Remove questions that are more relevant to Project Managers and/or Model Risk team. This means working with both the groups to identify these questions
MLOps
Marketplace
Lack of coverage for model experimentation
Create an alternate set of questions for model experimentation
MLOps
Marketplace
Concern about sensitive data deletion
Add information text describing the MLOps policy for handling PCI/PII data. Update the text in the form to reflect this policy
MLOps
Marketplace
Concern about unexpected billing
Add informative text to reassure users that the workspace billing for setup and usage is subject to senior leadership approval
Marketplace
MLOps
Lack of coverage for unstructured data
Add a new question about unstructured data, allowing data scientists to describe the data, source, and senstivity
Marketplace
MLOps
Concern about SFTp-S3 pattern
Add informative text to explain that more efficient and secure alternatives to SFTP-S3 pattern are being worked on
Marketplace
MLOps
Insufficient persona clarity
Clarify personas in the form so that permissions are more obvious. Add a leadership persona for oversight
Marketplace
MLOps
Marketplace
MLOps
UI issues
Make the necessary UI design issues
Marketplace
MLOps
Failure to understand the significance of certain questions
Remove unnecessary questions. Provide additional context for the questions that are difficult to understand
MLOps
Marketplace
Presentation
I led the presentations of these recommendations to the Managing Directors of the Cloud and Development team. Overall, the recommendations were received well, and it was decided to set up a new POD (a small cross-functional team) to take this initiative further.
Learnings
-
Presenting to Managing Directors (MDs) can make one anxious. However, I learned it's all about being prepared, calm, and energetic.
-
The data science field is evolving so much. For example, the need for unstructured data has increased due to GenAI, whereas this form was designed primarily for structured data. I learned it's important to stay up-to-date with the trends related to your users and business.