Dr Stuart Grey
Home Publications Public Engagement All Posts

stu@stugrey.com

Video and Audio Feedback at Scale for HE

Sep 18, 2021

During a recent chat with HE colleagues (Hi Jon 👋) I was reminded that I never did get round to writing up my approach to, and lessons learned from, attempting to create personalised video feedback for all (350) of my 1st year Engineering Mechanics Students, so here goes...

Background

The piece of work itself is a solution to a mechanics problem that involves interleaved sketches, equations, descriptive text and a particular type of engineering diagram called a Free-Body Diagram.

Students submit their hand-drawn/hand-written solutions and we give them written feedback by hand-drawing/hand-writing comments on the submitted PDF.

Issues

There were a few issues that I was trying to tackle:

  1. How can we give richer feedback in the short time we have to work on each student's work?
  2. How can we combine feedback on the gestalt with feedback on the detail?
  3. How can we provide general long form feedback with hand-drawn annotations?

This led to a number of constraints:

  1. We still want students to submit hand-drawn/hand-written solutions, a structured submission (maths here, prose there, diagram over here...) might enable some of the above but we would lose what is so valuable about communicating a complete engineering solution by combining, all of these parts.
  2. Had to be more time efficient than just writing an order of magnitude more written and visual feedback onto the PDF which would technically satisfy the above constraints, but would not be practical given the numbers involved.

Over the years we moved over to entirely electronic submission of the solutions and marked up the PDFs using tablets which did help improve efficiency and thus allow for more expansive feedback but it wasn't enough.

The Idea

It then occurred to me that we have a basis for a model that really works, what we do in our face-to-face tutorials.

In our tutorials we'll sit down next to a student and chat away while annotating their solution. Not much text is written as we are talking at the same time so the majority of the annotations are either diagrammatic or 1-2 words.

Actually speaking to the student is incredibly valuable and complements perfectly the annotations we make.

  • Speech - Gestalt, thought processes, approaches
  • Annotation - Detail, precision, specific correction

So how can we combine both in asynchronous feedback using current tools?

The Solution - Screen Recording of the PDF Annotation

We were using iPads to annotate the PDFs and they were working really well, the idea I had was to record an audio track to go along with the annotated PDF. But why stop there?

It turns out that it's actually easier to just do a full on screen recording (including audio) of the whole PDF annotation process.

In this way we provide a video to the student of their PDF being annotated with a continuous audio commentary.

There are a few obvious 1st order benefits:

  1. We can talk and annotate at the same time! This makes it much more time efficient for everyone.
  2. Students can immediately see why, and in what order things were highlighted (good feedback is surprisingly non-linear).
  3. Longer form feedback is spoken rather than written avoiding large blocks of hard to read handwritten commentary.

And few unexpected but really powerful 2nd order benefits (as highlighted by students):

  1. The simultaneous speech and annotation gives a real insight into the thought process of the teacher, allowing us to demonstrate and encourage reflection by the students.
  2. It's really easy to personalise. As I went through I quite naturally felt more comfortable if the process was more like talking to the student directly rather than at the iPad so I just opened each screen recording video with a "Hi Firstname, this is Dr Grey, so lets look at your work...."
  3. Producing transcripts is now just an accepted part of my practice wherever video is involved and these screen recording videos were no different, the nice thing was though, that the text of the transcript gave the student yet another avenue to consume the feedback.

Results

One benefit of having such a large class and two courseworks of this type per year is that we could quite rapidly A/B test the new "screen recording of the annotation process" approach against the "annotated PDF" approach.

We split the cohort randomly into 2 and gave half the screen recording video and half just the marked-up PDF. We then asked all the students two questions and recorded the answers on a Likert (Strongly Agree - Strongly Disagree) scale:

  1. The individual feedback on my coursework helped me understand the course content.

  2. The individual feedback on my coursework helped me revise for my exam.

Written Feedback

55% of the Students who got the written feedback agreed, or strongly agreed with the Statement 1 and 54% agreed or strongly agreed with Statement 2.

Screen Recording Video Feedback

85% of the Students who got the screen recording feedback agreed, or strongly agreed with the Statement 1 and 85% agreed or strongly agreed with Statement 2.

This is a clear indication that we are onto something here!

Please do get in touch (stugrey@stugrey.com) if you would like any more details or have any comments at all.