top of page

Evaluation & recommendations

Evaluation plan | Evaluation report |

Conclusion and recommendations

We decide to test our prototype as a portion of the entire orientation experience with some ECT  first-year students for the next round of iteration. Ideally, we would propose for a soft launch and pilot with the incoming new students coming Fall 2021. Here is a summary of our evaluation plan and report. 
Evaluation
Overview
Our primary evaluation method is usability testing on the integration of the new ECT website, Slack bot Donut, Slack channel interactions, the new ECT students checklist, student mixer event, live orientation sessions. 
ECT Website with bite-sized videos

Purpose:

  • How did users experience the website design? Where can we improve on with: instructions we provide, amount of content displayed, use of video and additional links/resources?

  • Where can students find quick tips?

  • How can we make videos more engaging using learning design principles?

  • How can ECT admins update content quickly?

Justification: 

  • This stage of usability testing will help us understand what students find valuable with our design and user flow. 

  • Faculty will be able to create and maintain updated videos that helpful for the students

Methods:​ User Testing and Survey results

​

Slack channel interactions (including Slack bot Donut)

Purpose:

  • How can we introduce Slack to graduate students as the main virtual communication tool?

  • How can students navigate through channels for different purposes?

  • How can we leverage Slack integrations to help students and admins engage?

  • Do students find the Slackbot helpful and engaging?

  • Where can we improve on:

    • creating slack channels

    • the frequency of Donut reminders and tasks

    • arrangement of virtual meet-up and connection 

Justification: 

  • Students expressed an interest in knowing more about their peer cohort both in person and virtually. 

    • Feedbacks on new Slack channels for new students help the design team to determine the best ways to interact and engage with the students that help students the most according to our goals

  • Slackbot: 

    • This will tell us whether or not students are actually completing the tasks as shown on the website or not by sending out reminders (students will need to click complete)

    • Evaluations on Slack bots help the design team to determine the frequency and content the bots messages should be designed to assist the new students' orientation experience

Methods:​ 

  • Collect the clicks/completion rate 

  • Count the number of questions asked in the new channel

  • Count the number of intro responses to the prompt, and categorize the questions by topics

 

Student mixer events

Purpose:

  • Did new students attend mixers?

  • Did students find mixers engaging? 

  • Were students able to get to know and connect with their peers in a more informal setting?

  • Where can we improve on:

    • frequency and format of new students/students mixer activities

    • ways to connect students with each other through fun activities, casual networking, etc., for different purposes and goals

Justification: 

  • New students are currently invited to open sessions to meet faculty, however, there is no dedicated forum for students to meet and engage with each other. 

  • ECT programs geared towards students may increase engagement and facilitate conversations amongst students. 

  • Psychological safety may also be strengthened, because new students can begin to build trust and friendships amongst each other

Methods:​ Keep track of registration signups and feedback results for events

 

Live orientation session 

Purpose:

  • Did new students attend live orientation? 

  • Did ECT students know the agenda and expectations of the orientation?

  • Did students find the ECT website helpful prior to joining the live orientation session?

  • Where can we improve on: contents to be covered during live orientation, types of activities should be included 

Justification: 

  • This will need to be determined with the help of faculty as they redesign the live session. 

  • We can use a strong experimental research design if possible.

Methods:​ 

  • Observe the level of live engagement 

  • Follow Up Survey 

 

New ECT student checklist

Purpose:

  • Did students download and make a copy of the checklist for themselves?

  • Did students use the checklist to help with preparing for the new semester? 

  • Is everything on the checklist covered by the new student orientation website? 

  • How can faculty edit checklists and websites simultaneously?

Justification: 

  • New students are either a part of a full-time or part-time program with different career goals. 

  • Checklist templates will enable students to make a copy of documents and decide which activities will support their graduate program in the best way. 

Methods:​ 

  • Survey Results

  • Observe the Slack channel dedicated to new student questions to collect data on student pain points

Summative evaluation
After rolling out the first pilot group of our design solution with the new cohort, we would like to ask the students to fill out a survey regarding their experiences with the orientation program, as a summative evaluation for future revisions for the new cohort.  
Target audience

ECT master students who just finished the first semester

 

When to conduct

At the end of each semester

 

Purpose

We are trying to understand new students’ needs up to date in order to:

  • Iterate design ideation and revision

  • Provide better user experience 

​

Methodology 

  • Short survey via Google Form. We would interested in asking questions such as: 

    • Awareness and accessibility of the upgraded website

    • If new students participated in the orientation and students mixer activities

    • The engagement of the orientation and students mixer activities

    • How useful of the provided resources (the bit size video, links on website, and new student checklist)

    • Anything we need to improve or revise?

    • What else supports did they wish they had as a new ECT student? 

  • Observation of interactions on Slack channels:

    • The new student Slack channel dedicated to asking questions (collect data on most asked questions, etc)

    • New peer cohort connection building 

 
User testing protocol
This part showcases a detailed plan of our evaluation plan for our prototypes as a portion of the new ECT orientation experience: 
  • ECT new students website
  • Slack interaction with Slack bot Donut 
  • ECT new students checklist  
Users

New ECT graduate students (or at least first year ECT graduate students)
 

Evaluation purpose

Help us understand what students find valuable with our design and user flow. 

  • Evaluation issues/description of the instruction being evaluated:

  • How will new students discover instructions?

  • What resources listed in the checklist and website are most valuable to new students?

  • What is the realistic timeline for students to complete tasks?

  • How did users navigate the website design? 

  • Where can we improve on with: instructions we provide, amount of content displayed, use of video, and additional links/resources?

  • Are Slackbot notifications and student groups convenient for new students?

 

Methodology 

A usability testing implemented with a think-aloud protocol and interview session on:

  • Web page prototype

  • Slack bot Donut

  • New Students Checklist

(Due to technical difficulties, we have to show and demo the interviewees through Slack bot Donut and how it works)

​

  • Interviewee Tasks:​

    • Navigate and think aloud while browsing the newly-revamped ECT new student site

    • Browse through the sample videos and its introduction text

    • Look for the customizable new student checklist

    • Be shown a brief demo of Donut

  • Interviewee recruiting 

    • Invite 3 students who are not in our Design Process for Learning class (who have never seen our design before)

    • They will receive the link to the new student site prototype and explained Donut

    • Record reaction and mini-interview

      • Website prototype: 

        • Interview: 

        • Would you find these short-form videos helpful? 

        • What resources do you wish were here?

        • Is the format of the page easy to navigate?

        • What do you think about having a slack channel just for new students?

        • What else would you recommend as for the orientation process?

      • Slackbot Donut​

        • Would they find the Slack bot (Donut) helpful?

        • Have they used a Slack bot before?

        • Were the instructions easy to follow from the Slackbot (Donut)?

        • What do you think about having Donut assisting you as an orientation buddy?

    • User testing scripts:

Hi [participant name]. Thank you so much for joining us today. My name is Samhita, and I will be facilitating today’s study. Before we begin, I want to spend some time reviewing how this session will be structured. We were asked to re-design the ECT new student experience. As you might recall, ECT currently provides a step-by-step guide on the website, as well as, an in-person orientation session which is held closer to the start of the semester. After conducting research to better understand how people interact with these elements, we spent time redesigning the whole orientation experience including 4 key elements: ECT new student website with short videos on how to complete essential items for first-year students, Slack bot that send reminders on new student essentials, student mixers, and an interactive new student checklist. 

 

For today’s session, we’ll be asking you to review two of these 4 elements. The new student website page and interaction with the slack bot. I want to emphasize that we are testing this site, not how you are using it. You can’t be wrong at any point in the study. Throughout the session, I will ask you to move to the next phase to ensure we are respecting your time. We are hoping to understand what you are doing or trying to do. We would appreciate your honest reactions. At the end of the observation, we will be asking you a few follow-up questions. Overall, the session should take 30-45 minutes.

 

If you have any questions throughout, I will do my best to answer them. This session is really about how you are exploring the application and understanding it. If there is anything you need, like a break, please do let me know. We want to make this as comfortable of an experience for you.

 

If you are open to it, we would like to record this session. We will be using the recording to identify key areas where the app needs to be improved and ensure we didn’t miss any details from my notes.  It will be viewed by our internal team. Would you be okay with us recording?

 

Before we begin, did you have any questions?

First, you’ll be taking a look at the ECT new student website prototype. Like I mentioned before, as you are navigating through, let us know your thoughts around what you are seeing. 

​

Results analysis

Steps to Analyze: 

  • We would record the testing of the slack notification and hi-fi prototype 

  • Write down key reactions,  roadblocks and what the students were expecting to see.

  • Decoding interviewees’ interview answers and feedback 

  • Analyze participation/completion rate from Donut

  • Make a plan to interview 5 more people after the course is done to be able to create an affinity map 

  • Identify themes

  • Visualize results 

  • Determine areas of improvement for the experience

 
Conclusions and recommendations 

This is a viable product that can be piloted this Fall semester with the help of ECT faculty. In order to do this successfully we recommend:

  • Meeting with Stakeholder over the summer to review design and feasibility 

  • Identify resources that can be converted into short videos for the website 

  • Confirm whether or not ECT can leverage Donut or another Slackbot 

  • Identify student and faculty volunteers to provide feedback on the new asynchronous format for the fall semester

Evaluation report
After analyzing what we got from the three 30 -45 minutes usability testings with the tenured students of ECT program, as well as class peer feedback, we have gained a comprehensive overview of the strength of our redesign and the revisions we have to make during the iteration process. 

We tested the redesign of the ECT New student website with three first-year students. Our testing is usability testing embedded with a think-loud process and interviews to help our team determine: 

  • If our redesigned website is effortless and intuitive to navigate 

  • If the features of our redesign are most helpful to the students. 

  • If we are missing any information that would have helped students to smooth their journey to the ECT program. 

 

The data that we collected from the think-loud showed that students favored the redesign and the checklist that could be found on the site. There are mixed reviews about Donut (Slackbot) due to the uncertainty of the amount of notifications students will receive during the orientation process. Here is the affinity map with our usability testing results, as well as class feedback, clustered by themes: 

Design iteration implications 

A list of the strength that our redesign based on feedback, which we would like to keep the design:

  • Website design: Good visuals and organized content make students feel welcomed. 

  • “Get connected” has a good Slack tutorial that is helpful for new students to walk through and sign up for Slack.

  • Short videos are attractive and efficient for information grabbing.

  • The checklist helps new students make sure what should be completed and having the student feel supported.

  • Donut is an excellent application as a reminder and a progress checker.

​

Revisions that need to be made (has been made) to our redesign based on feedback include: 

  • ECT Student website:​

    • Adding a “go back to home” button/link on each page

    • Making the checklist more visible with a change of color

    • Adding end dates to the checklist

    • Highlighting and distinguishing mandatory videos versus additional resources content 

    • Creating a video carousel to help make mandatory videos more visible and easy to scroll

  • Donut

    • Creating an additional note that the Slackbot will only be sending notifications in the first 2 months of the orientation experience

    • Creating recommendations to Admin on frequency caps for Slackbot notifications

Recommendations for next steps 

Our recommendation for the next step is to proceed with plans to soft launch elements of the new orientation process in the fall. In order to do so the next iteration phase would include:

  • Reconnecting with stakeholders on Fall expectations on: 

    • The number of new students coming in (in-person either virtually or ), as well as their statuses 

    • If the orientation for the Fall semester is an in-person session or online synchronous session via Zoom 

  • Finding out if Donut (Slackbot) is a viable financial option for ECT

  • Finding out if the implementation of creating a cohort channel for the incoming class and how and who to implement and manage the channel 

  • Create a timeline on when the actually ECT website and checklist can be update

  • Create a list of tasks and activities that would connect the new students together personally on Slack even before the in-person orientation session starts 

  • Co-design an implementation plan with ECT faculty in charge of orientation 

© 2021. Proudly created with Wix.com

bottom of page