2.1.1 Digital audio workstations

For more than fifty years, audio was recorded on magnetic tape. Combining sound sources required the use of a large mixing console which was used to control the sound with faders, knobs and buttons that had to be triggered at the right time. Editing was performed by cutting the magnetic tape with a razor blade and sticking it back together again (Barbour 2004).

The development of fast processors and high quality audio interfaces has since allowed audio to be stored and manipulated digitally using computer software. The primary tool for editing digital audio is the digital audio workstation, or DAW. A DAW is software that provides recording, mixing and editing capabilities for digital audio. DAWs were first introduced in the 1980s (Ingebretsen and Stockham 1982), and have since evolved into powerful tools that are accessible to anybody with a computer. Examples of popular commercial DAWs include Pro Tools by Avid, Logic Pro by Apple and Cubase by Steinberg (Ask Audio 2015; Producer Spot 2015).

DAWs provide a feature-rich toolset for manipulating audio signals. They can be used to navigate and edit audio with very fine control over timing, even down to individual samples. Automation means that changes made to the audio are remembered and repeated each time the audio is played. Automatic cross-fading between clips can be used to create inaudible edits.

The introduction of DAWs has transformed radio broadcasting by allowing fast random access, high storage densities, improved portability, and greater cost-effectiveness than analogue systems (Pizzi 1989). The powerful features of a DAW can replace most of the activities that would traditionally have to be performed using a radio studio. The accessibility of digital audio production has allowed audio editing to be performed by producers without requiring specialist knowledge of sound engineering (Peus 2011). McLeish and Link (2015, p. 44) suggested that the improved usability of DAWs has created a “high level of personal job satisfaction” (McLeish and Link 2015, p. 44). However, the deskilling of audio editing has also caused a reduction in the number of people required to produce radio programmes (Dunaway 2000).

As the audio is being stored and manipulated digitally, DAWs can be used to edit audio without any loss in sound quality. However, when the edited audio is saved, there are two approaches that can be taken — destructive and non-destructive (McLeish and Link 2015, p. 45). Destructive editing occurs when a change is made that alters the structure of the sound file. This prevents the edits from being easily undone. Non-destructive editing occurs when the original audio components are retained and can be re-used to make a change to the edit. DAWs can perform non-destructive editing by saving an edit decision list, or EDL, which records the positions of the edits, but does change any audio files. With EDLs, audio edits can be moved or undone retrospectively. Only when the final edit is ready does the audio get destructively “rendered” or “bounced” to an audio file.

2.1.2 Visual representation

Digital audio editing is performed using a visual representation on a computer screen (Derry 2003; Hausman et al. 2012). Barbour (2004) found through observation and interviews with radio producers that “visualization of audio on a screen has become the dominant focus in a radio production studio”, and that visual representations are used to assemble, modify and balance the audio for radio programmes.

Using visual means to interact with audio has a number of benefits. It allows users to manipulate the audio using a mouse and screen, which are commonly used in computing. Mapping audio to an image allows the temporal information of the sound to be displayed spatially, which means it can be searched and skimmed quickly and randomly. However, visualizing audio is difficult, and there are limitations to what audio visualizations can tell us about the audio.

Bouamrane and Luz (2007) argue that “visually representing audio in a meaningful manner is a particularly difficult task as there is no obvious or intuitive way of doing so”. Currently visual representations cannot fully represent the sound, so producers must listen to comprehend the audio. McLeish and Link (2015, p. 45) argue that “while it is tempting to edit visually using the waveform on the screen, it is essential to listen carefully to the sound, [such as to] distinguish between an end-of-sentence breath and a mid-sentence breath”. Visual representations may also serve as a distraction to the producer. Barbour (2004) found that to concentrate on listening, radio producers disengaged their visual senses by shutting or de-focusing their eyes, or looking away.

Although we could not find any studies that surveyed the use of visualizations in DAWs, we looked at the five most popular DAWs (Ask Audio 2015) and found that all of them visualized the audio using a “waveform”.

2.1.2.1 Waveforms

An audio waveform is a common graphical representation of an audio signal that is produced by plotting the amplitude of an audio signal over time. Audio signals are periodic, as sound is produced through compression and rarefaction. This can be seen from the repeating curved lines of the waveform. Lines that are closer together represent higher pitch sounds and lines that are farther apart represent a lower pitch. The height of a waveform corresponds to the amplitude, or “volume”, of the audio.

Waveforms have been used to visually represent audio content since the first digital audio workstations started to appear (Ingebretsen and Stockham 1982). Today, they are the default audio visualization used in the DAWs we surveyed. The simplicity of the waveform makes it conceptually easy for users to understand and interpret the audio. Waveforms are relatively compact, so can be arranged vertically on top of each other to view multiple audio tracks simultaneously. They are also computationally efficient to generate, as they are plotted in the time domain.

Figure 2.1.2.1.1: Example audio waveforms of speech, demonstrating the effect of zoom on the visibility of frequency information.

Despite their widespread use, waveforms display relatively little information about the audio. Figure 2.1.2.1.1 (a) shows a waveform that has been zoomed-in. At this scale, we can see the individual cycles of the audio signal, and the mix of frequencies that make up the sound. However, when we zoom out, these curves are compressed to the point where they are no longer visible. Figure 2.1.2.1.1 (b) shows a waveform at a zoom level typical in audio production. At this scale, it is impossible to determine which frequencies are present. What remains is an “amplitude envelope” that indicates the volume of the sound over time.

Without frequency content, there is a limit to the amount of information waveforms can convey. The amplitude envelope can be used to identify silences, peaks and the relative volume of different parts of the audio. With experience, it is possible to use the amplitude envelope to distinguish different types of sounds. For example, the frequent short periods of silence in Figure 2.1.2.1.1 (b) indicate that this may be speech, because unlike music, speech is broken up into words.

In order to be able to infer this information, users must learn what the amplitude envelope of different sounds look like. This would be a problem for novice producers, but not for professionals who work with audio on a daily basis. However, the level of information that can be inferred is limited (Hausman et al. 2012, p. 114). For example, it is very difficult to use a waveform to distinguish editorially relevant features, such as individual people’s voices, or different styles of music.

We are interested in learning how audio waveforms affect the performance of audio editing tasks. However, despite the widespread use of waveforms to visualize audio, we could not find any studies that have attempted to evaluate their performance as a method of interacting with audio.

2.1.2.2 Spectrograms

A spectrogram is a plot of the intensity of the Short-Time Fourier Transform (Smith 2007), which visually represents of the spectrum of frequencies in an audio signal over time. Higher frequencies are displayed at the top of a spectrogram, and the intensity of the signal is mapped to the brightness (or sometimes colour) of the image. Figure 2.1.2.2.1 shows an example spectrogram of a speech recording.

Spectrograms clearly display the frequencies that make up the sound, and in what proportions. With spectrograms, time and frequency can be scaled independently. Unlike waveforms, when a spectrogram is viewed at different zoom levels, the frequency information is still visible. Spectrograms are based on frequency analysis, so they are more computationally expensive to generate than waveforms, but this is rarely an issue with modern processors.

Like waveforms, spectrograms are general-purpose, so can be used for a variety of tasks and applications. Spectrograms display a much higher density of information than waveforms, which can be used to infer more information. For example, Zue and Cole (1979; Zue and Lamel 1986) found that expert users were able to use spectrograms to read individual phonemes of speech, but inexperienced users were unable to achieve this. Although spectrograms present the data clearly, users must still learn how to read the information.

Reading spectrograms requires users to have a theoretical understand of audio frequencies and how they behave, such as how a single pitch can be composed of many harmonics. Although spectrograms display the intensity of the signal in each frequency band, it is not apparent what the overall volume of the audio is at a given time. Additionally, spectrograms have a wide range of parameters that control how they are displayed, including FFT window size and shape, linear/non-linear frequency and intensity scaling, min/max values and colour mapping. This creates inconsistencies between different spectrograms, which can make it difficult for users to move between software. Waveforms don’t have as many parameters, so are much more consistent.

In Section 2.3, we will show how waveforms and spectrograms can be enhanced using semantic audio features, but first we will introduce the relevant methods and applications of semantic audio analysis.

2.2.1 Semantic audio features

Semantic audio analysis is conducted by processing the audio using an algorithm to extract one or more semantically relevant “features”. This process known as feature extraction. Audio features are numerical representations of certain properties of the audio, which are often categorised into low-level and high-level features (Fazekas 2012, p. 31). Low-level features include physical and perceptual properties, such as the energy and spectral content of the sound. High-level audio features correspond to more meaningful concepts, such as words and people, or structural segments, such as programmes or topics. Many semantic audio algorithms use classification or machine learning to map low-level features into high-level features. For example, in speech recognition, a language model is used to map individual phonemes of speech into words and sentences (Junqua and Haton 1995).

There are many different types of audio features that can be extracted. With music, rhythmic features are used to extract the beats and tempo, and harmonic features are used to determine the notes and chords. Speech is in some ways a more complex signal to analyse, so more generic features are often used. In this section, we will describe selected audio features that are touched upon later in this thesis, to help illuminate the reader’s understanding of their origin. Below we have outlined three types: energy, temporal and spectral features.

2.2.2 Applications

Semantic audio analysis allows us to gain insights into the content of audio recordings without having to listen to them. The semantic audio features we described have already been used to tackle a variety of problems (Foote 1999). In this section, we outline the aim, methods and performance of three applications of semantic audio analysis that are used later in this thesis: speech/music discrimination, speaker diarization and automatic speech recognition.

2.2.2.2 Speaker diarization

Speaker diarization is the task of segmenting an audio recording into labelled segments that identify “who spoke when” (Anguera Miro et al. 2012). With this task, the location of any speech content and number of speakers is usually unknown. Speaker diarization has clear applications to the production of radio, where there are often multiple people speaking in a single recording, and it is desirable to know where they are speaking without having to listen.

Review papers from Tranter and Reynolds (2006) and Anguera Miro et al. (2012) show that the vast majority of speaker diarization systems are based on clustering of MFCCs, and that current research is focused on the improvement of clustering algorithms and pre-processing stages, rather than audio features. They also show that most of the recent research has focused on recordings of meetings, rather than broadcast content.

Anguera Miro et al. (2012) found that the average error rate for speaker diarization systems was 11.6% and 17.7% for two standard data sets (National Institute of Standards and Technology (NIST) 2016). However, these data sets are based on microphone recordings of meetings, rather than broadcast content. Bell et al. (2015) conducted an evaluation of speaker diarization systems on television recordings of multiple genres. These results showed that the error rate was 47.5%, which is considerably higher. However, rather than just trying to match speakers within individual recordings, their evaluation was conducted across multiple recordings, which made matching speakers between them all more difficult. A breakdown of the results showed that most of the errors were misidentification of speakers, and that misidentification of speech accounted for less than 8% of the error rate.

Speaker diarization systems assign a unique identity to each speaker, but they do not attempt to identify who the speaker is. Speaker recognition is the task of identifying a person based on the sound of their voice (Doddington 1985; Lee, Soong, and Paliwal 1999). Extracting metadata such as participant names and genders from radio content could be used to enable automated information searching and indexing (Kinnunen and Li 2010). Speaker recognition relies on access to a database of trained speaker models, which represent people’s voices. In radio, many of the contributors are from a small pool of presenters, so it may be feasible to use speaker recognition techniques to detect their voices with sufficient accuracy.

Raimond et al. (2014) introduced the BBC World Service Archive prototype, which was an interface that used automatic keyword tagging and crowd-sourcing to support the search and discovery of a large radio archive. The interface used speaker diarization and speaker recognition to help users navigate within individual radio programmes. Figure 2.2.2.2.1 shows an example of a radio programme that has been segmented into five named speakers.

[Speaker 1] the manchurian candidate both seems to play up these fears and to be in a way in she comes to sudden he can have a critique of the idea of the moral panic around brainwashing i wondered where pavlov fits into that story and how seriously are his ideas taken in the literature of the nineteen fifties around brainwashing
 
[Speaker 2] we'll have a viz is everywhere in in the discussion of the american p.o.w.s they're sometimes referred to in magazine articles and in popular commentary at the time as as prisoners of pavlov so there was a larger of of of our popular discussion about pavlov often not very well informed but only rouge to his experiments with the conditioned reflex and his famous salivating dogs and ringing bow and so on that was was everywhere so certainly many americans would have at associated some kind of pavlovian conditioning with what had been done to the p.o.w.s but but it wasn't generally carried very far into in terms of actually trying to him better understand how pavlovian principles or psychology might might actually have been at work in the p.o.w. camps

2.3.1 Crossmodality

To be able to represent audio visually, we must map auditory properties to visual properties. When attempting to link sound and vision, it is desirable to create a mapping that is coherent and makes sense to the user. By creating an audio visualisation that “looks likes it sounds”, it might be possible for users to comprehend the sound without having to listen to it.

Crossmodal perception is a term used to describe interaction between the different senses (Spence 2011). Previous work has shown that there are perceptual mappings between auditory and visual stimuli that are experienced by most of the population. These could be exploited to aid the navigation and editing of audio recordings.

The “bouba/kiki effect” is a demonstration of crossmodal mapping between speech sounds and the visual shape of objects, originally discovered in an experiment by Köhler (1929). Participants were shown two abstract shapes, shown in Figure 2.3.1.1, and asked which shape was called “bouba” and which was called “kiki”Köhler used the words “baluma” and “takete” in the original experiment, but the result was the same.

. Ramachandran and Hubbard (2001) found that 95–98% of the population gave the same answerThe vast majority of participants chose to name the curvy, rounded shape on the left “bouba”, and the sharp, pointy shape on the right “kiki”.

. This is an example of just one audio-visual mapping that is common amongst the population.

Spence (2011) presented a review of psychology experiments that attempted to find crossmodal links in the human brain, including audio-visual mapping. He found that there was strong evidence for five audio-visual mappings, shown in Table 2.3.1.1. These findings were supported by Tsiros (2014), who attempted to generate images to match different sounds, and measured their success through a user study. In addition to confirming the strong links between loudness/size and pitch/elevation, he found weaker links for pitch/colour, dissonance/granularity, and dissonance/colour complexity.

Current audio visualizations exploit some of these crossmodal mappings. For example, waveforms map loudness to size, and spectrograms map loudness to brightness, and pitch to elevation. However, this previous work shows that there are many more links between sound and vision that could be further exploited by audio visualizations.

2.3.2 Waveforms

As we discussed in Section 2.1.2.1, the audio waveform is commonly used by DAWs as a visualization of an audio signal (Derry 2003). As such, many users are familiar with navigating audio content using waveforms, and have learned how to read the shapes of the waveform. Enhancing a waveform, either by processing it or adding additional information to it, could allow users to navigate and edit audio content more efficiently whilst retaining this familiarity, and using the skills they have developed. Our survey of the literature found that two main approaches have been used to enhance waveforms — scaling and colour.

2.3.2.1 Scaling

When an audio waveform is zoomed out, the curves of the waveform are compressed which can make it difficult to read. This affects both horizontal zoom (on the time axis) and vertical zoom. One very simple technique for improving waveform readability is to automatically scale the vertical zoom to match what is visible on the horizontal timeline. However, if the scale of the waveform constantly shifts, there is no reference level by which to compare the amplitude of the audio. The solution proposed by Goudeseune (2012, p. 39) was to overlaying a dimmed version of the scaled waveform on top of the normal waveform. This allowed users to simultaneously judge the overall amplitude whilst being able to see the detail of the amplitude envelope.

Frequency information is useful for understanding the timbre of an audio signal. When viewed at the right scale, this information is visible in a waveform, but at typical zoom levels, this information is lost. J. Loviscach (2011b) proposed a novel solution to this problem called the quintessence waveform. This approach used extreme pitch shifting so that the individual cycles of the audio waveform are visible, even at different scales. This works well for repeating monoaural sounds — for example, a sine wave would be identifiable as a sine wave at every zoom level. However, typical real-life applications use complex polyphonic audio, which would not benefit from quintessence waveforms as there is no repeating signal to display.

Gohlke et al. (2010) proposed five novel ideas on how to improve multi-track DAW displays, including techniques for saving screen space by overlaying and stacking waveforms. One of these proposals was for a lens-like view, shown in Figure 2.3.2.1.1, which magnified the area of the waveform around the current playhead position. This allowed users to simultaneously view the waveform at two different scales — an overview of the audio waveform and a detailed local view. This technique has the potential to display frequency information in regions of interest, and help make more precise audio edits without having to adjust the overall zoom level.

2.3.2.2 Colour

The use of colour is a simple and effective way of adding additional information to a waveform. However, many DAWs only use waveform colour to allow users to label audio clips, and most others have monochromatic waveforms. Previous research has experimented with mapping semantic audio features to colour, using either pseudocolour or false colour.

Pseudocolour is a method of mapping a scalar value to a colour gradient (Moreland 2009), an example of which can be seen on thermal imaging cameras. Colour gradients are composed of at least two colours (e.g. blue to red) or a spectrum of colours (e.g. a rainbow). Pseudocolour allows values to be mapped to colours that might be perceptually relevant (e.g. green/red for good/bad). It can emphasise small variations between values by using a full spectrum, pick out high/low values using non-linear gradients, or categorise values using stepped gradients. However, as pseudocolour can only represent one dimension, it does not make full use of the available colour space.

False colour exploits the tristimulus theory of vision to map three values to the dimensions of a colour space (Moreland 2009). Commonly, values are mapped to red/green/blue (RGB) colour space. Other colour spaces can be used, such as hue, saturation, value (HSV), which better matches human perception of colour (Smith 1978). Hue can be described as “the colour on a rainbow”, saturation represents lack of greyness, and value means brightness. The advantage of false colour is that it can make full use of the available colours. On the other hand, it can be challenging to select three values and map them to colour in a way that is perceptually relevant and understandable.

Rice (2005) presented Comparisonics — a patented (Rice and Patten 2001) method of using pseudocolour to map the frequency content of an audio signal to a colour spectrum. Comparisonics was designed for identifying timbrally distinct sounds and he claims that, with training, it can be used to identify certain sound effects. His technique maps frequency to colour using an unpublished algorithm, where low frequencies are blue and high frequencies are red. Comparisonics has since been integrated into the Scratch LIVE DJ software from Serato Audio Research, where it is used to distinguish between different drum noises, such as bass kicks, snares and high-hats. However, the author could not find any formal evaluation of Comparisonics.

Akkermans et al. (2011) implemented a similar system in the audio clip sharing website Freesound to help users quickly find and compare sound effects and music clips. They used pseudocolour to map the spectral centroid of the audio (see Section 2.2.1.3) to a rainbow colour gradient. This colours lower frequency sounds blue and higher frequency sounds red, matching the effect seen in Rice (2005). An example of this approach is shown in Figure 2.3.2.2.1. J. Loviscach (2011a) used pseudocolour to enhance the navigation of speech in a video editor by distinguishing different phonemes of speech. This was achieved by mapping the zero-crossing rate of the audio (see Section 2.2.1.2) to a rainbow colour spectrum. The author could not find any studies that attempted to evaluate these approaches.

Tzanetakis and Cook (2000) used false colour to design a visualisation technique known as Timbregrams. Their aim was to “use colour perception and the pattern recognition capabilities of the human visual system to depict timbral and temporal information”. Their implementation extracted a large vector of common audio features, then used principal component analysis to reduce the size of the vector. They mapped the first three principal components, which contained 80% of the variance in their data, to RGB or HSV colour space. They found that the RGB colour space was more uniform and aesthetically pleasing, but that the HSV colour space had better contrast at segmentation boundaries. When using RGB, speech, classical music and rock could easily be distinguished as they appeared as light green, dark blue and dark green, respectively. Tibregrams were later used to colour a waveform in a basic audio editor (Tzanetakis and Cook 2001, p. 253), but the author could not find any formal evaluation of Timbregrams.

Mason, Evans, and Sheikh (2007) used false colour to assist radio listeners in navigating recently-broadcast material. They mapped three empirically-chosen audio features to RGB colour space. The authors reported that the system was successful at indicating the location of music within speech content, and highlighting low-bandwidth material such as phone calls. However, this was not formally evaluated. The authors proposed that the system could be also be applied to other applications such as segmentation of radio programmes for re-editing into podcasts. Figure 2.3.2.2.2 shows an example of this approach for a BBC radio programme that contains five segments. Although the segments are not visible in the waveform, the false colour visualization displays the voice of the female presenter in a lighter colour, which makes the segments visible.

2.3.3 Spectrograms

As we discussed in Section 2.1.2.2, spectrograms are an information-rich representation of the spectrum of frequencies in an audio signal over time, but they can be difficult for novice users to read. Lin et al. (2012) introduced a method of filtering spectrograms to visually emphasise non-speech events in long audio recordings. The filtering was done using an “image saliency algorithm” that detected differences in the intensity and orientation of the spectrogram. This saliency-maximised spectrogram was integrated into an audio navigation interface called Timeliner (Goudeseune 2012), which displayed the spectrogram alongside a waveform. Lin et al. (2013) describes an evaluation in which 12 novice participants used Timeliner to find sound effects hidden in meeting room recordings using both saliency-maximised and normal spectrograms. The results show that saliency-maximised spectrograms significantly outperformed normal spectrograms. Filtering spectrograms shows promise as a way of detecting unusual events, however it is unclear how useful this sort of application would be in the context of radio production.

2.4.1 Transcript generation

Transcripts can be written manually, either using pen and paper or a word processor, but this is a slow and tedious process. Transcription can be completed faster by only transcribing the most salient words, but this makes the transcript much less readable, particularly to others who haven’t heard the original recording. Alternatively, a third-party can be used to transcribe the speech, but this slow and expensive. For example, transcribing speech using rev.com currently costs US$1 per minute and takes 12 hourshttps://www.rev.com/transcription, accessed 11/12/2017

.

As we saw in Section 2.2.2.3, ASR can be used to convert speech to text automatically. ASR is quicker and cheaper than manual transcription. ASR also produces accurate timestamps for each word, which can be used to precisely navigate and edit the audio, but word-level timestamps can also be added to manually-written transcripts using speech alignment (Griggs 2007; Boháč and Blavka 2013).

Erroneous transcripts reduce listener comprehension (Stark, Whittaker, and Hirschberg 2000; Vemuri et al. 2004) and increase the time it takes to search audio content (Ranjan, Balakrishnan, and Chignell 2006) and correct errors (Burke, Amento, and Isenhour 2006). However, despite the errors in ASR transcripts, they provide a highly effective tool for browsing audio content as users can visually scan the text to focus on regions of interest, known as “strategic fixation” (Whittaker and Hirschberg 2007).

2.4.2 Transcript navigation

Transcripts have previously been used by several systems as an interface for improving the navigation of speech-based content, such as news reports and voicemail messages. One of the first such systems was NewsTime from Horner (1993), which used transcripts to aid the navigation of audio news stories. For television news, subtitles were aligned the audio to provide an accurate transcript with word timings. NewsTime included several additional features including searching by keyword, segmenting the transcript by story and speaker, jumping to the next or previous speaker/story, and categorising stories into one of seven common topics. There were no reported user studies of NewsTime.

SCAN (Whittaker et al. 1999) was an interface designed to support retrieval from speech archives. It used ASR transcripts to allow users to search for keywords and visually search the recording by reading the transcript. In a user study of 12 participants, the transcript was found to support navigation by reducing the listening time needed to complete information retrieval tasks. Participants rated the tasks as being easier, and the browser as being more useful, with the transcript than without. SCAN was further developed into SCANMail (Whittaker et al. 2002), an interface designed for interacting with voicemail messages. It added a number of features including paragraph segmentation, and the ability to seek to a point in the audio recording by clicking on a word in the transcript. Whittaker et al. (2002) evaluated SCANMail through a study of eight experienced users, which found that the transcript display enabled them to visually scan the content of recordings to quickly extract information, and to judge which parts were relevant, without having to play the audio.

2.4.3 Semantic speech editing

In addition to supporting the navigation of speech recordings, transcripts have also been used as a method of editing speech content, known as semantic speech editing. The first of these was the “Large Interactive Display System Wave Speech Editor”, catchily shortened as LIDSWSEdit, from Apperley et al. (2002), which used ASR transcripts to allow users to navigate and edit lecture recordings. Any edits made to the transcript were correspondingly applied to the underlying audio recording. Users could re-arrange sentences and words by selecting the text, and using a drag-and-drop action. Alternatively, speech could be removed by selecting text then clicking a button to either delete the selected text, or everything except the selected text. LIDSWSEdit was further developed into the “TRanscription-based Audio EDitor”, or TRAED (Masoodian et al. 2006). TRAED used the same editing actions as LIDSWSEdit, but rather than displaying the text and audio waveform separately, it displayed the waveform in-line with the text. Individual words were delineated by drawing boxes around the waveform/word pair. The boundary between each pair could be adjusted by dragging the boundary edge. The author could not find any user studies of LIDSWSEdit or TRAED.

Whittaker and Amento (2004) created an interface for editing voicemail messages using ASR transcripts. Users could cut-and-paste parts that they wanted, or delete parts they didn’t. They evaluated their system in a formal study of 16 voicemail users, which found that semantic editing was faster and as accurate as editing with a waveform. Crucially, they found that this was true even though the transcripts had an average word error rate of 28%. This suggests that semantic editing is beneficial even when using erroneous transcripts.

Rubin et al. (2013; Rubin 2015) presented a novel interface for creating “audio stories” that combine speech and music, which is similar to radio production. The interface, shown in Figure 2.4.3.1, used an editable transcript with two columns, one for each of a pair of speakers. It allowed the user to cut, copy, paste and delete the audio using the text, and highlighted repeated words and similar sentences. The transcripts were generated using an online service that produced 100% accurate, or “verbatim”, transcripts. As they were manually-generated, the transcripts also included “umm”s, breaths and pauses, which were displayed and labelled in the interface. However, the manual transcripts did not include timestamps, so speech alignment software was used to recover the timestamps for each word. The system also included additional functionality for finding and adding music tracks, and for varying the length of music using automatic looping. The system was evaluated through a short informal study of four participants where the editing capabilities received positive feedback. The author could not find any follow-up studies.

Sivaraman, Yoon, and Mitros (2016) created a semantic editing system for asynchronous voice-based discussions, where users could quickly edit their speech recording before sending it to the recipient. Their system used near-live ASR and detected pauses in the speech. Their interface allowed users to delete selected words/pauses, insert additional pauses and fix incorrect words. In a formal qualitative study of their system with nine users, they found that text-based editing was considered good enough to replace waveform editing, and to be more accessible. They observed that most users only used the system to make fine-grained edits, instead of editing large chunks. Users said that the transcript also allowed them to quickly review all the points that were made, and that the errors in the transcript weren’t a heavy distraction.

Yoon et al. (2014) created a collaborative tablet-based document annotation system called RichReview, which offered users three modalities in which to annotate documents — free-form inking, voice recording and deictic gestures (i.e. pointing to areas of interest). The voice recordings were displayed using a waveform, overlaid with an ASR transcript of the speech. Users could trim or tidy the voice recordings by drawing a line through words or pauses to remove them. The system was evaluated using a qualitative study of 12 students which found that the editing features were considered easy to use and efficient for removing “umm”s and long pauses. However many participants reported that the transcripts were not accurate enough to use without having to listen to the audio. Yoon et al. (2016) describes two deployment studies that used a similar system called RichReview⁺⁺, but they did not report there being any semantic editing functionality.

2.4.4 Video editing

Semantic speech editing has also been used to support video editing. SILVER (Casares et al. 2002; Long et al. 2003) was a video editor that aligned words from subtitles to the video, and displayed them in a transcript window. Gaps, errors and edits were displayed in the transcript using special characters, such as “||’‘for clip boundaries, “—’’ for gaps, and”*’’ for noise or recognition errors. The video could be edited by deleting text in the transcript. SILVER was evaluated in an informal study with seven students, but the study did not report any results about the transcript-based editing feature.

Hyperaudio Pad is an open-source audio and video editor, first proposed by Boas (2011), and now available online as a free service (Hyperaudio Inc. 2016). This web-based interface, shown in Figure 2.4.4.1, allows users to navigate and edit online media using transcripts, which are generated from subtitles. Editing is performed by selecting a part of the transcript and dragging it into a window on the right to create a “clip”. Clips can be re-ordered, split using a “trim” tool, and fade effects can be added between clips. Clips from different recordings can be mixed together, and the final edited version can be played and shared with others. No user studies of this system could be found.

When editing a video interview, it is desirable to avoid making a cut while the person speaking is in shot, because it causes the image to jump unnaturally. Berthouzoz, Li, and Agrawala (2012) used image processing algorithms to create a video editor that can help the user hide these edit points. The system had an editable transcript window that displayed suitable edit points and allowed the user to edit the video by selecting and deleting text. The transcripts were generated manually using an online crowd-sourcing service, and word timings were added using speech alignment software. The system also allowed users to easily remove “umm”s or repeated words as they were explicitly marked in the manual transcript. No user study was reported, however the system received positive feedback from nine professionals who were given a demonstration.

2.4.5 Pre-written scripts

The systems so far have only considered transcripts that have been generated from the speech itself. Sometimes speech is recorded based on a pre-written script, or from notes. Avid Technology released a feature for their Media Composer video editing software in 2007 called ScriptSync (Avid Technology Inc. 2011). This feature aligns a user-supplied transcript to a video recording by placing a marker in the video for each line of the transcript (Griggs 2007). This allows users to jump to a particular line, or see which line in the transcript corresponds to the current point in the video. A second version of ScriptSync was launched in February 2017 (Avid Technology Inc. 2017) which added script correction and collaborative note-taking.

Shin, Li, and Durand (2016) created a system called Voice Script that supports an integrated workflow for writing scripts, and recording/editing audio. An informal study with four amateur participants found that it could support various workflows including multiple iterations. It included a “master script” layout to bring together different recordings, which was found to work well. A second study of four amateur participants directly compared the system to that of Rubin et al. (2013), which found that participants were able to complete an audio production task 25% faster using the Voice Script system. This study demonstrates that for workflows that involve pre-written scripts, there is potential to improve the audio editing by using an integrated writing and editing system.

QuickCut from Truong et al. (2016) was an interface designed to help producers edit a narrated video from a pre-written script, voiceover audio and raw video footage. Producers could label their video footage using their voice, which was manually transcribed using a crowd-sourced online service in combination with speech alignment. Selecting text in the script also selected the corresponding segment in the voiceover audio, and displayed video clips labelled with similar words. After selecting an appropriate clip, it could be associated with a position in the script by using drag-and-drop to add it to the timeline. The completed timeline could then be exported as an EDL for use in professional video editing software. QuickCut was evaluated by the researchers themselves and one professional filmmaker, who were able to use the system to produce a minute of video in 8–31 minutes, rather than the 2–5 hours professional filmmakers suggest they require. Voice-based logging makes sense for logging video footage as it is easy to watch and talk at the same time. However, for speech content it would be difficult to talk and listen simultaneously. The ability to export edits to professional software allows for a smooth continuation of the production workflow.

2.4.6 Transcript correction

Whittaker and Amento (2004) found that users of their semantic speech editing system “wanted to be able to correct errors they found in the transcript”. ASR errors reduce listener comprehension (Stark, Whittaker, and Hirschberg 2000; Vemuri et al. 2004) and increase the time it takes to search audio content (Ranjan, Balakrishnan, and Chignell 2006) and correct errors (Burke, Amento, and Isenhour 2006).

Four of the ASR transcript interfaces mentioned above included correction functionality, with each using a different method to edit the text. SILVER (Casares et al. 2002) required the user to both type the replacement word and select the start and end time of the word in the video timeline. TRAED (Masoodian et al. 2006) allowed users to correct a word by selecting it and typing the replacement. Typing a space created a new word by dividing the time of the original word in half. Sivaraman, Yoon, and Mitros (2016) initially planned to have two editing modes — one for audio editing and the other for text editing. However, in pilot testing they found that having two modes confused users, so they developed a pop-up box that indicated to the user when they are editing the text, rather than the audio. SCANMail did not initially include transcript correction, but this was later added and evaluated by Burke, Amento, and Isenhour (2006). These changes allowed users to either replace an individual word by selecting a replacement from a drop-down menu, or replace multiple words by selecting them and typing the replacement. A user study of 16 participants who corrected voicemail messages found that compared to typing, selecting the replacement word required the user to listen to less of the audio.

The correction process can be made more efficient by correlating the user’s input with contextual information (Suhm, Myers, and Waibel 2001). In particular, the words immediately before and after the incorrect word can be used to reduce the number of candidate words, or even estimate the replacement. Liang, Iwano, and Shinoda (2014) showed that once an incorrect word is identified, in 30% of cases the correct word can automatically be inferred using \(n\)-grams and acoustic features.

Correction is normally a process that happens after ASR transcription, but as Wald et al. (2007) demonstrated, it is possible to correct ASR transcripts in real-time as the audio is captured. However, during recordings, radio producers are normally pre-occupied with operating the equipment, asking questions or listening to the answers. As this does not leave enough space for performing real-time correction, an extra producer would be required, which is costly.

The above systems used a keyboard and mouse interface to correct transcripts. Suhm, Myers, and Waibel (2001) tested alternatives methods that used vocal correction and a pen interface. They found that for skilled typists, keyboard and mouse input was faster than the alternatives, but that voice and pen input would be attractive for use by poor typists, or for devices that don’t allow fast keyboard input.

ASR transcripts contain errors in the text, but sometimes there are errors in the speech itself that producers may want to correct. TypeTalker from Arawjo, Yoon, and Guimbretière (2017) was an interface for editing synthesised speech using ASR transcripts. The speech was not synthesised from text, but from a recording of the user speaking. This was done to reduce the self-consciousness that results from hearing one’s own voice. As well as being able to remove unwanted speech, the use of speech synthesis meant that new speech could be synthesised and words could be changed. Adobe Systems Inc. (2016) demonstrated an unreleased prototype system called VoCo that enabled users to change a word in a speech recording, whilst retaining the natural characteristics of the original speaker’s voice. Such technology could be used to create seamless repairs to errors in speech recordings. However, the use of such technology has ethical and legal implications, particularly in a broadcasting context (Bendel 2017)There’s an interesting episode of Radiolab that discusses this: http://www.radiolab.org/story/breaking-news/

.

2.4.7 Confidence shading

In addition to producing a transcript, many ASR systems return a confidence score for each transcribed word, indicating how sure the system is that the word is correct. Confidence shading is a technique for displaying this score by colouring words with a low confidence score in a lighter shade (Suhm, Myers, and Waibel 2001). Confidence shading has been used to try to make mistakes easier to locate, and transcripts easier to read. However, confidence scores may themselves be incorrect by indicating that a correct word is probably incorrect, or that an incorrect word is probably correct. The trade-off between these two types of errors is controlled using the threshold value (Feng and Sears 2004).

Suhm, Myers, and Waibel (2001) conducted a user study of 15 participants who corrected an ASR transcript with and without confidence shading (in this case, highlighting). The results showed that correction with confidence shading took slightly longer than without, although this was not statistically significant. Conversely, Burke, Amento, and Isenhour (2006) reported that in their user study of 16 participants, most agreed that confidence shading was helpful for identifying mistakes in the transcripts. One notable difference between these studies is that Suhm, Myers, and Waibel (2001) optimised their confidence threshold to minimise the overall accuracy of the confidence shading, whilst Burke, Amento, and Isenhour (2006) increased the threshold to treat false negatives more seriously.

Vemuri et al. (2004) studied whether confidence shading improved comprehension of the transcript. They conducted a user study of 34 participants and measured the comprehension of short audio clips when using ASR transcripts with and without confidence shading. Although the results indicated better comprehension with confidence shading, there was no statistically significant difference.

2.5.1 Time compression

Listening to long audio recordings of speech can be time-consuming. A simple way to reduce the listening time is to increase the rate of playback. However, this increase in speed causes an upward shift in the pitch of the sound, which is sometimes described as sounding “like chipmunks” (Vemuri et al. 2004; Ranjan, Balakrishnan, and Chignell 2006). The increased speed with which the content is presented also makes it difficult for listeners to process the information fast enough.

In this field, intelligibility is defined as the ability to identify words, and can be measured by the accuracy with which a specific word is recalled. Comprehension is defined as the ability to understand the content of the material, measured by the number of correctly answered questions about the subject matter (Foulke and Sticht 1969). The change in pitch caused by speeding-up audio negatively affects both the intelligibility and comprehension of speech (Arons 1997).

There are several approaches for reducing the time required to listen to a recording while being able to extract critical information, which can be divided into two categories. Speed-up techniques aim to increase the speed of playback without affecting the pitch of the speech, and excision techniques aim to remove parts of the speech in a way that minimises the reduction in comprehension (Arons 1997).

Tucker and Whittaker (2006) performed a user study that compared two different excision techniques and a speed-up technique, using both 5-min and 30-min audio recordings. Participants ranked a list of utterances to match what they heard, which was compared to a reference response to produce a score for comprehension. This score was normalised by the listening time to measure “comprehension efficiency”. The results showed that for short recordings, excision outperformed speed-up, but that they performed similarly for long recordings. However, when using excision, participants were less likely to switch to normal-speed playback, and they reported that they preferred excision to speed-up.

The simplest excision technique is to remove frames of audio at regular intervals, known as “isochronous sampling” (Arons 1997). However, this approach does not discriminate between valuable and redundant information. It also fails to take into account speech boundaries, so may cut the audio mid-way through a word. Shortening or removing pauses between words is a simple and effective approach that reduces the length of the audio whilst retaining all of the information and respecting speech boundaries. However, once all of the pauses have been removed, other techniques must be used to further compress the speech.

Many excision algorithms operate by segmenting the audio at points of increased saliency, then playing only the beginning of each segment before moving onto the next. The saliency can be determined by measuring pause length, pitch, speaker turns and using transcripts. Long pauses in speech often signal a new sentence, thought or topic, which can be an indication of importance. The pitch of the voice tends to increase in range when introducing a new topic (Hirschberg and Grosz 1992), which can be used as a measure of emphasis. Speaker diarization techniques (Anguera Miro et al. 2012) can be used to detect changes in speaker, which can be a cue for changes in topic. Transcripts of the speech have also been used with summarisation techniques to determine the most salient parts of the speech, using both ASR transcripts (Hori and Furui 2003) or manually-written transcripts (Tucker and Whittaker 2006).

SpeechSkimmer by Arons (1997) combined three excision techniques into a single time compression interface by switching between them for different rates of playback. He used pause shortening and removal for modest speed increases, followed by pause-based segmentation for faster playback. For the fastest playback rate, he used segmentation resulting from a pitch-based emphasis detection algorithm. He evaluated the system through a qualitative study of 12 participants, which compared two systems that used different algorithms for the fastest playback rate — one using pitch-based emphasis segmentation and the other using isochronous sampling. The participants reported that pitch-based emphasis was effective at extracting interesting points, and performed better than excision using isochronous sampling.

There are limits to how far time compression can be used to increase playback speed. For example, speed-up techniques are only intelligible up to a maximum of around \(2\times\) to \(2.6\times\) real-time (Vemuri et al. 2004; Tucker and Whittaker 2006; Ranjan, Balakrishnan, and Chignell 2006; Arons 1997). However, transcripts can be used in combination with time compression to increased this maximum rate. Vemuri et al. (2004) conducted a user study of 34 participants and measured their comprehension of short audio clips at different rates of playback using speed-up. The mean self-reported maximum playback rate was \(2.6\times\) real-time for listening only. The addition of an ASR transcript increased this to \(2.8\times\), and a verbatim transcript increased this further to \(3.0\times\). Whittaker et al. (2002) exploited this by including time-compressed playback in the SCANMail semantic speech interface.

2.5.2 Simultaneous playback

The cocktail party effect is “the ability to focus one’s listening attention on a single talker among a cacophony of conversations and background noise” (Arons 1992). This effect can be exploited to help listeners find a particular piece of audio in a recording by playing different parts of that recording simultaneously. To help listeners separate the sounds, previous work has experimented with using headphones to play different sounds in each ear, or using binaural audio to spatially separate the sounds.

AudioStreamer from Schmandt and Mullins (1995) used binaural spatialization techniques to play three simultaneous audio streams of broadcast news around a listener’s head. The system tracked the movement of the listener’s head to boost the level of the stream they were facing as they turned. In addition, they used pause-based segmentation and speaker diarization to alert the listener to new stories using a short bleep sound. No user studies of AudioStreamer were conducted.

Dynamic Soundscape from Kobayashi and Schmandt (1997) also used spatialization to help users navigate audio files by mapping the sound to fixed positions a virtual soundscape. The system was designed to take advantage of human abilities for simultaneous listening and memorising location. Users would start by listening to a virtual “speaker” that played the audio while slowly orbiting their head in a clockwise direction. Audio could be replayed by pointing their hand at the location where it was originally heard, which would create a second speaker that played from that position. Similarly, users could skip ahead by pointing to a position ahead of the original source. Speakers could be grabbed and moved, and an audible “cursor” allowed users to hear where they were pointing. Through informal feedback, users suggested that they could use their spatial memory to navigate the audio. Based on their observations, the authors suggested that the system could also help with transfer to long-term memory.

Ranjan, Balakrishnan, and Chignell (2006) attempted to reduce the time needed to search an audio recording by using dichotic presentation, where different sounds are played into each ear. In their system, the left ear played from the beginning of the recording while the right ear played from the half-way point. Through a user study of 13 participants, they tested the effectiveness of this approach for a search task. The results showed that dichotic presentation reduces the overall search time compared to normal audio playback, particularly when the answer is in the second half of the recording. The overall time reduction was around 20%. Dichotic presentation can be combined with time compression, but this creates high cognitive load and 8 of the 13 participants reported it to be “very demanding”.

3.1.1 Data collection

We wanted to exploit our position of working within the BBC by collecting our data through a workplace study at BBC Radio. In their book on workplace studies, Luff, Hindmarsh, and Heath (2000) argue that “it is important to observe and analyse work as it occurs” and criticise the use of interviews and questionnaires. They point out that a researcher cannot know in advance what the right questions are to ask participants in an interview, and often there is a difference between what participants believe they do and what they actually do. As Luff, Hindmarsh, and Heath (2000, p. 245) note, “many activities are performed repeatedly and become tacit in nature; they are seen but not noticed”.

We chose to use direct observation to collect our data, where we witnessed radio production first-hand without taking part. Producers are very busy and direct observation allowed us unobtrusively to collect the data without adding to the producer’s workload. Additionally we could observe the real-world process, as opposed to a theoretical or reported one, and take into account the context of the working environment.

Some studies have successfully used video recording to capture and analyse interactional organisation in the workplace, which can provide insights into communication and collaboration (Luff, Hindmarsh, and Heath 2000, p. 16). We were not able to take this approach for our study as the observation took place at the BBC, whose policies prevented us from using video recordings in the workplace. This policy exists to protect the staff’s privacy and any sensitive information, which is often handled by journalists. Rather than disrupting the workflow and environment of the production team to use video recording, we recorded the observational data by writing field notes. In addition to observation, we used free time between activities to conduct ad hoc, in situ interviews to clarify the participant’s workflow and decision-making process.

Due to the scale and variety of the radio operations at the BBC, it would be impossible to cover all production genres and techniques. To limit the scope of our work, we followed a “maximum variation sampling” strategy (Patton 1990, p. 172) to choose a small number of heterogeneous case studies. We selected programmes of different genres to cover a variety of cultures and work practices.

The time needed to produce programmes can vary significantly, with some being produced over many weeks or months. To reduce our observation time, we worked with each production team to create a schedule of observations that sampled every stage of the process and every role in the team. This allowed us to capture the entire workflow without having to be present throughout.

3.1.2 Recruitment

We recruited participants using an invitation email sent to BBC R&D’s contact list of Studio Managers working in BBC Radio. Through this process, we recruited three production teams who create the following programmes from the departments listed:

• Hourly news bulletin (Radio Summaries, BBC News)

• “15 Minute Drama” radio drama (London Radio Drama, BBC Radio)

• “The Report” documentary (Radio Current Affairs, BBC News)

These three programmes (news report, drama, documentary) fulfilled our criteria for a heterogeneous sample of programmes from different genres.

3.1.3 Procedure

We used a single researcher to collect the data for our study through direct observation. The researcher observed the production of each programme from the beginning of audio recording/collection, to the point where the audio had been finalised. The observation did not focus on a single member of the team, but covered the entire team that contributed to the audio production. This allowed us to study the different roles involved and how they interact. The production teams were observed in their normal place of work. This allowed us to take into account the context of the environment in which the teams work, both in terms of the physical location and layout, and of the tools and software they use to perform their tasks.

We worked with each production team to design a schedule of observation that would cover the entire production process. Each programme required a different amount of time. Observing the news bulletin took half a day, the drama took two days, and the documentary took four days. When the team members were not busy, the researcher used ad hoc, in situ interviews to ensure they understood the reasons and motivation behind the producer’s actions.

The researcher used a laptop computer to type field notes throughout the observation. The notes specifically included the following:

• Roles — Who are the team members? What are their responsibilities? Which other teams are involved?

• Environment — What is the location? How is the physical space laid out?

• Tools — Which tools are used? For what purpose? How do the users operate them?

• Tasks — What tasks are involved? Who does what? In what sequence do they take place?

• Challenges — Were there any problems or frustrations? Which activities were demanding or mundane?

The researcher also took photographs of any relevant locations, tools or other items, with the permission of the participant.

3.1.4 Analysis

We used the observation notes to populate a list of roles involved in the production and wrote a description of each of their responsibilities. We wrote a description of the working environment in which the production took place, including the location, the tools that were used, and how they were used. We also drew a map of the physical environment, including the spatial layout of the team members.

We used hierarchical task analysis (Kirwan and Ainsworth 1992; Annett and Stanton 2000) to deconstruct the production process into a sequence of individual tasks. We assigned each task to the role that was responsible and the location in which it took place. We then used a partitioned operational sequence diagram (Kirwan and Ainsworth 1992) to graph the sequence of tasks in chronological order, arranged into columns to indicate the role and location.

Finally, we identified any challenges that were noted by the researcher, and wrote a description of the current approach and any suggested improvements that could be made.

3.2.1 News bulletin

The Radio Summaries team at BBC News (known as “Summaries”) write most of the hourly news bulletins for most of the national radio networksThe 6pm and 10pm bulletins, and all of the bulletins for Radio 1, 1Xtra, Asian Network and Radio 5 are not written by Summaries.

. The bulletins written by the Summaries team are read live on-air by a Newsreader at the start of every hour. The researcher observed the team for five hours during a morning weekday shift, from 7am to midday. The pace of work in the team was so fast that there was little time to talk to the participants to ask any questions, so the results are mostly based on direct observation.

3.2.1.2 Environment and tools

The Summaries team sit together at a single desk in the BBC newsroom at New Broadcasting House in London. When it opened in 1932, Broadcasting House was the first purpose-built broadcast centre in the UK (Hines 2008). Following a major expansion between 2003 and 2013, Broadcasting House now contains 35 radio studios from which 30 domestic and World Service radio stations are broadcast, and is the workplace for 6,000 staff (BBC News 2013). Figure 3.2.1.2.1 shows the newsroom and Figure 3.2.1.2.2 shows the location of the team within the space. The BBC newsroom is the largest in the world and is supported by some 3,000 journalists (McLeish and Link 2015, p. 80). The newsroom houses teams from around the BBC News division and is spatially arranged to facilitate the fast flow of communication. The teams for each platform (e.g. TV, radio, online) sit together at desks that fan out from a central area. Decisions on which stories to cover are made in this central area, and are communicated outwards.

Figure 3.2.1.2.3 shows an example of the desks used by members of the Radio Summaries team. The desk includes an intercom that can be used to communicate with other teams using labelled “push-to-talk” buttons. A desktop TV monitor is used to keep an eye on the 24-hour BBC News television channel, to track which stories are currently being reported. Most communication is within the team, which is helped by sitting close together at the same desk.

The Broadcast Journalists and Assistant Editor write the summaries on a desktop PC using a software package called “Electronic News Production System” (ENPS). Figure 3.2.1.2.4 shows the ENPS interface. ENPS is used for writing scripts, compiling and approving news bulletins, and monitoring newswire servicesNewswire services are global news agencies that provide news reports to subscribing news organisations, such as Reuters and Associated Press (AP).

. The system is networked so that any changes made in ENPS are instantaneously updated for everybody. Audio clips are browsed, imported and edited using a plugin called Media Object Server (MOS), which integrates with the dira! radio production system.

When Intake upload a clip to dira!, it appears on a large screen in the news floor called the “arrivals board” (see Figure 3.2.1.2.5). The display includes a reference number which can be used to find the clip, as well as the upload time and a description. The description often contains the subject, reporter’s name and their location, but the descriptions are written quickly and there is no formal syntax.

3.2.2 Drama

Radio 4’s “15 Minute Drama” is a series of original drama and book dramatisations, broadcast twice-daily. Drama production is very different from most other genres in radio, as it is based on a pre-written script of a radio play. Each episode takes around two days to produce, but production of the series spans several weeks. The researcher observed the production over two full days — one for the recording of the drama and the other for the editing process. The observation did not include the writing of the script.

3.2.2.2 Environment and tools

The drama we observed was recorded in Studio 60A, which is a purpose-built flexible performance space at BBC New Broadcasting House in London. The studio contains various spaces with different acoustic properties, including a staircase with wood, concrete and carpet steps, an upstairs bedroom, a fully working kitchen and a foam-lined spiral corridor, used to simulate distance. There are many fixtures and a range of props for re-creating spot effects of common environmental sounds such as freestanding doors/windows with a variety of knockers and letter boxes. The studio is a large space, so there are four CCTV cameras which are used to help the producers in the cubicle to see parts of the studio that are out of sight.

The studio is connected by a large acoustically-isolated window to the control room, known as the “cubicle”, where the production team sit. Figure 3.2.2.2.1 shows the layout of the two spaces and Figure 3.2.2.2.2 shows the view of the studio from the cubicle. The mixing desk is operated by the Panel SM and is positioned directly in front of the window. The Broadcast Assistant sits to the right of the Panel SM and the Director sits directly behind them. The Grams SM sits at the back of the room, while the Spot SM spends most of their time in the studio. The Director, Broadcast Assistant and Panel SM each have an intercom, which they can use to communicate with people in the studio with a “push-to-talk” button.

The Panel SM sets the sound levels and balances the audio using the mixing desk, and makes a backup recording of the performances using a CD recorder located in a rack to the left of the mixing desk. Under the mixing desk, there is a foot pedal which controls a light in the studio, known as a “cue light”, to silently indicate the start of a performance. The Panel SM also controls warning lights displayed above the door to the studio to stop people from walking in during recording. The Grams SM uses software called “SpotOn” (Cridford 2005) to select and play pre-recorded sound effects. They use the ProTools DAW to record audio clips of each performance, and label them with the episode, scene and take numbers.

Figure 3.2.2.2.3 shows the editing suite where the Panel SM and Director perform the post-production work. The editing suite is a compact acoustically-treated room which houses a PC running ProTools, stereo speakers, a small mixing desk and level meters. The left monitor is used to find pre-recorded sound effects and the right monitor is used to arrange and edit the audio clips. The Panel SM uses an annotated copy of the script to guide this process.

3.2.2.3 Task analysis

Figures 3.2.2.3.1 and 3.2.2.3.3 show the operational sequence diagrams of the radio drama recording and editing processes, respectively. Descriptions of each workflow with labelled tasks are presented individually below.

Recording The recording process involves the entire production team and cast. The Broadcast Assistant co-ordinates the team members by organising the recording date, and booking the studio and cast (1). Between one and two episodes are recorded in a day. Prior to recording, the cast will assemble in the “green room” with the Director and rehearse the play (2). During the rehearsal, the Broadcast Assistant measures how long the performances take (3), to ensure they are not too long or too short. The Director provides feedback to the cast on their performances (4) before they move to the studio (5). The Grams SM sets up the studio by arranging acoustic panelling, preparing props and arranging microphones (6). In the cubicle, the Panel SM sets the sounds levels of the microphones (7), and the Grams SM selects and loads the sound effects that will be played during the recording (8).

The drama is recorded as a series of “takes”. Each take is a short segment of a scene, often only 20–30 seconds long. Multiple takes of the same material are recorded so that the Director can give feedback to the cast between takes. The Director starts the recording process by announcing the episode, scene and take number (9). The Panel SM starts the backup recording (10), and the Grams SM starts the DAW recording (11). When everything is ready, the Panel SM flashes the cue light, which silently signals the cast to start performing (12). During each take, everybody listens to the performance (13). The Spot SM performs live spot effects in the studio (14), while the Panel SM balances the audio levels (15), and the Grams SM triggers pre-recorded sound effects (16).

The Broadcast Assistant annotates a printed script with information about each take (17). Figure 3.2.2.3.2 shows an example of an annotated drama script. The start and end of each take is marked with a vertical line on the side of the page. The take and backup CD numbers are written at the top of each line, and the duration of the take is written at the bottom. If the take starts again during a recording, the line continues back to the top. The best take for each scene is marked by highlighting the line for that take. Words that are spoken incorrectly are underlined, and repeated words or phrases are marked with square brackets. Different coloured pens are used to distinguish the marks for each take.

At the end of a take, the Panel SM stops the backup CD recording (18) and the Grams SM stops the DAW recording (19). The Grams SM uses the DAW to label the clip with the episode, scene and take number — for example, “e2s3t1” (20). The Broadcast Assistant marks the printed script and spreadsheet with the length of the take and the backup CD number (21). If the Director wants to record another take (22), they discuss the performance with the production team, then provide the cast with feedback, either by walking into the studio, or using the intercom (23). If the Director is satisfied that they have recorded what they need, then they select the best take and move onto the next scene (24). The Broadcast Assistant marks the best take on the script and spreadsheet, which calculates the current overall duration of the recording (25).

At the end of the episode (26), the team either move on to the next episode, or the Director sends everybody home (27). The Grams SM copies the audio clips from the DAW onto a portable hard drive (28). Portable hard drives are used as they can handle large file sizes better than than the local computer network. The hard drive and annotated drama script are given to the Panel SM to use for editing.

Editing The recordings for the drama are edited into a final programme by the Panel SM and Director. The Panel SM starts by creating a rough edit of the programme by themself, using the annotated script as a guide. As the files are stored on a portable hard drive, this process can be done either in an edit suite or on a laptop at home. The first step is to import the audio files (1) and create a sequence of the best takes from the recordings, as marked in the annotated script, by dragging them onto a time-line from the list of labelled clips (2). The Panel SM uses the script to identify and remove errors in the takes, such as re-takes or repeated words (3). They adjust the sound level to be consistent throughout by using the mouse to draw level curves, or by recording automation using a fader on the mixing desk (4).

The Panel SM adds any additional sound effects (5) using an effects library on their PC, which contains roughly 900 hours of sound effects. The effects are found either by using text to search their metadata, or by browsing to specific collections of effects. Before the Director joins them, the Panel SM listens through the rough edit (6) to identify any mistakes or errors, such as repeated words and phrases, or noise caused by actors handling their paper scripts. Double-speed playback is used to save time during this process. Any errors that are identified are fixed before moving on to the next stage (7).

Once the rough edit is complete, the Director joins the Panel SM in the edit suite. They listen to the rough edit to judge the quality of the performances they previously selected as the best takes (8). Often, the Director will ask the Panel SM to play other takes so that they can compare them (9). To do this, the SM finds the correct recording in the clip list, drags it onto the time-line and finds the correct position in the clip. The Director may ask the Panel SM to swap a take, or combine the start and end of different takes to use the best performances (10). The same process happens for the sound effects (11), which may be swapped or mixed together (12).

Music is not specified in the script, so the Director has the freedom to choose what they want (13). Popular consumer music services such as iTunes are used to find commercial tracks, but often Directors will choose “production music”, which is designed for TV/radio and is easier to license. These can be searched using descriptive keywords on one of a number of online music libraries such as Audio Networkhttps://www.audionetwork.com/

or Desktop Jukeboxhttps://desktopjukebox.broadchart.com/

. The Director provides the music to the Panel SM on a USB storage device for them to add using the DAW (14).

The final programme must have an exact duration to fit its assigned broadcast slot. The programme is often edited to be slightly too long, so some lines can be removed to reduce the programme length. Removing lines has a strong editorial impact, so the Director decides which lines to remove (15) and the Panel SM edits them out using the DAW (16). Once finished, the final edit is rendered to an audio file by playing the programme in real-time through a digital loop-back (17). Although this can be done faster, this playback forces the Director and Panel SM to listen to the programme from beginning to end in one go. If they are happy with the edit (18), the Director sends the file to their Editor for sign-off and broadcast (19).

3.2.3 Documentary

“The Report” is a weekly investigative documentary that covers topical news stories. It is produced over a three week period by the Radio Current Affairs department in BBC News and is broadcast at 8pm every Thursday on Radio 4. The researcher observed the team for a total of four days — one day during their research phase, one day during their interviewing phase and two days during their editing phase.

3.2.3.3 Task analysis

Figure 3.2.3.3.1 shows the operational sequence diagram of the documentary production. A description of the workflow with labelled tasks is presented below.

Production starts with researching the chosen topic of the documentary (1). The purpose of the research stage is to form the story-line for the documentary, and to identify people to interview. Often the topic will be in the news that week and the Producer will be looking for an interesting angle which can be explored in greater depth. The Producer and Researcher listen to news reports and documentaries, read newspaper articles and encyclopedias, and talk to contacts who know about the subject. The Producer makes rough notes for themself in Microsoft Word and prepares a draft outline of the programme (2).

Once the team have identified who they might want to interview, the Producer or Researcher will approach them to see if they are interested (3). If the interviewee has the time available, the Producer or Researcher will do a “pre-interview” over the phone, which simulates a real interview but is not recorded (4). This is done to see what the person will say and whether it suits the story-line of the documentary. Most interviews are recorded face-to-face, either on-location or in a studio, depending on the situation (5). During interviews, the Presenter asks the questions while the Producer records the audio and monitors the levels. For on-location interviews, the Presenter holds the microphone and the Producer operates the portable recorder (6).

All of the interview recordings are transcribed (7). Some recordings are sent to a third-party transcription service in Australia that transcribes the audio overnight. However, often the programme’s budget can only cover transcription of less than half of the interviews. The rest must be transcribed by the Producer or Researcher. For this, they use Microsoft Word to manually type the words. To save time, they will only do a rough transcription by skipping most words, leaving only enough to get a good idea of what was said. With both the third-party and manual transcriptions, timestamps are written every few minutes to help the reader identify the location of the audio.

The interview transcripts are printed onto paper, which the team use to help them collaborate face-to-face. The team go through the interview transcripts and mark with a highlighter pen lines that they want to use (8). Notes and labels are informally written on the page. After the team have marked-up the interview transcripts, the Producer uses them as a guide to find and create audio clips of the content they highlighted (9), and piece it together into a rough edit using a DAW (10). The timestamps written in the transcript are used to help them navigate the audio.

While the Producer creates a rough edit, the Presenter writes the programme’s “links” — the narrative elements that sit between the interview clips (11). When the first rough edit is complete, the whole team sits down with the Editor for a “run-through” (12). The programme is performed out loud from beginning to end, with the Presenter reading the links and the Producer playing the clips from the DAW. This allows the Editor to hear the programme and give early feedback. It also allows the Producer to determine the current length of the programme. This run-through process typically happens two or three times for each programme (13).

Once the Editor is happy with the rough edit, a Studio Manager (SM) is brought it to help turn it into the final programme. This often happens on the day of the broadcast. The Presenter records the links (14) with the help of the SM and the Producer who gives them feedback. The SM adds the links to the DAW in the correct positions (15). The Producer chooses any music that they want to add to the programme (16), often production music from an online library. The SM adds this music to the programme, along with the programme’s theme music (17). The SM cleans up the interview clips by removing redundant noises (e.g. “umm”) and phrases (e.g. “you know”), and ensures a good pace by removing long pauses (18). However, some redundant noises or pauses are left in because they are difficult to remove or editorially relevant. The SM also balances the sound levels by recording automation using a fader on the mixing desk or by dragging in/out fades.

Once all of the elements have come together, the Producer and SM work together to cut the programme down to a specific duration, in this case 27 minutes and 45 seconds (19). This is done by removing sections of speech, usually from the beginning and end of interview clips. The finished programme is played to the Editor for their approval (20), and any final changes are made by the Producer (21). Once the programme is signed-off, the documentary is rendered to an audio file and imported into the dira! radio production system. The Producer must then listen to the entire programme in dira! to ensure the audio that will be broadcast contains no errors (22).

3.3.1 Waveform navigation

In all of the case studies we observed, the audio was navigated using a waveform representation. In the news bulletin, the journalist often had to search for a quote in a 40-minute clip using only the audio waveform and a rough idea of the location of the quote. In the drama production, the SM used a waveform to navigate and edit the audio within each clip to match the annotations made on the script. In the documentary production, the Producer assembled their final programme by using a waveform to find, cut and arrange each audio clip marked on the transcript.

As we discussed in Section 2.1.2.1, audio waveforms are limited in the information they can display, which makes it difficult to identify words, sentences and speakers. This can make it difficult for the producers to navigate and edit the audio content, as the location of their desired content in not apparent using the waveform. In Section 2.3.2, we saw that previous work has used scaling and colour to enhance audio waveforms. These approaches may be able to provide producers with a richer visualization that would make it easier to identify the desired region of audio.

3.3.2 Textual representation

We observed that in all three case studies, the producers interacted with audio using a textual representation. A script was used to refer to and mark-up the audio recorded in the drama. The documentary Producer transcribed all of their interview recordings, and used the transcripts for most of the production. The news bulletin producers also used text to label and insert “out-words” of the audio clips in their scripts.

Sound is a medium that must be experienced over a period of time, so listening to large quantities of audio can be slow. Representing audio using text allows users to quickly glance through and search audio content without having to invest time in listening. Text can be used to display word-for-word what was said, but it cannot display how it was said. The nuances of the spoken word can contain deeper meaning, which may not be visible in a written transcript. Despite this, the documentary Producer we observed chose to work with text rather than directly with the audio, which demonstrated that they found value in using a textual representation.

Generating transcripts of speech can be an expensive process. We saw that the documentary Producer could only afford to pay for transcription of less than half of their interviews, so had to transcribe the rest themselves, which took a long time and produced poor quality results. As we saw in Section 2.2.2.3, ASR could make this process faster and more affordable. We also saw in Section 2.4 that semantic speech editing interfaces can be used to edit audio content using a transcript-based interface. However, ASR transcriptions contain errors and it is not known how these would affect the usability of such transcripts in a radio production environment.

3.3.3 Use of paper

We saw that in both the drama and documentary productions, the teams used printed scripts or transcripts as part of their workflow. The two reasons we could identify for using paper rather than a screen interface were to make handwritten annotations, and to facilitate face-to-face collaboration. The drama production team used scripts to help them coordinate what they were recording, and to make annotations that were later used as a guide to edit the programme. The documentary Producer printed the transcripts of their recordings to help them collaborate face-to-face in deciding which parts to use, then marking their selections using a highlighter pen.

Paper is a flexible medium that can be handed around, pointed at and annotated freely, amongst other things. However, with both the documentary and drama, edit decisions that were written on paper had to be manually translated into audio edits using a DAW. This is a tedious and slow process. If these annotations could be captured in a structured digital format, they could then be automatically translated into audio edit commands. This may save producers time and effort without deviating much from their existing workflow.

3.5.1 Audio waveforms

In all of the case studies, producers used audio waveforms to navigate their audio content. As we saw in Section 2.1.2.1, waveforms display limited information which can make them difficult to use for navigation. However, despite their widespread use, the author could not find any studies that attempted to measure the performance of audio waveforms. Section 2.3.2 described several promising methods for enhancing audio waveforms by using colour to add semantic information. However, the author could not find any formal evaluations of these methods.

To answer our research question on the role and efficacy of audio visualization (see Section 2.6), we will perform a user study in Chapter 4 to evaluate the performance of audio waveforms and colour-enhanced visualizations. Our interest is in using audio visualization techniques to make the production of radio programmes more efficient. Therefore, we will design an audio visualization that uses colour enhancement to support a radio production task. We will evaluate our visualization using audio content that is representative of radio broadcasting, and measure the efficiency of users in completing the task using our visualization. Due to the lack of formal studies of both methods, we will not only compare our colour-enhanced visualization to normal waveforms, but also measure the performance of normal audio waveforms themselves.

3.5.2 Semantic speech editing

The producers in all three of our case studies interacted with audio content using textual representations. We also saw that the documentary producer wrote transcripts of all their interviews. In Section 2.2.2.3, we described how ASR can be used to automatically transcribe speech, and in Section 2.4 we saw that transcripts have been successfully used to allow for semantic navigation and editing of speech content.

As we discussed in Section 2.6, Rubin et al. (2013) demonstrated a system for the production of “audio stories”, which has many similarities to radio production. However, this system was not formally evaluated, so it is still unclear what effect semantic editing has on the radio production process. Other semantic editing systems have been formally evaluated (Whittaker and Amento 2004; Yoon et al. 2014; Sivaraman, Yoon, and Mitros 2016), but they were designed for navigating and editing voice messages and comments, which use a different style of audio content and have different requirements from radio production.

To answer our research question on how transcripts can be applied to radio production (see Section 2.6), we will investigate in Chapter 5 whether semantic speech editing can be used for radio production by designing and developing a semantic speech editor for radio production, and evaluating it through a user study. To ensure that the results relate to real-life requirements, we will use our access to BBC Radio to recruit professional radio producers, and evaluate semantic speech editing for use in production of genuine radio programmes. To measure its performance, we will compare semantic speech editing to the current production workflow.

3.5.3 Paper interface

In two of our three case studies, producers used paper scripts and transcripts as part of their production workflow. This allowed them to collaborate face-to-face and to describe their audio edits using annotations. The remainder of their production workflow was digital, and transferring between the paper and digital representations was a manual process. It may be possible to use technology to “bridge the gap” between paper and digital, which may have benefits to the radio production workflow.

This finding about the use of paper within radio production was unexpected, but provided an interesting new avenue for our research. This gave rise to an additional research question that we wanted to address in this thesis:

Question 4: What is the potential role of augmented paper in radio production?

In Chapter 6, we will investigate how augmented paper can best be applied to radio production, and develop and evaluate a system for producing radio content using a paper interface. To ensure that our system fulfils the real-life requirements of radio production, we will evaluate it for use in the production of genuine programmes by professional producers. We will compare this approach to a fully digital workflow by measuring our system against a similar screen interface. To understand the benefits of paper-digital integration, we will measure the performance of the system with and without the integration between the two.

4.1.1 Recruitment

The radio production community is relatively small. Producers are very busy and not used to participating in academic studies. We wanted to recruit enough participants for our results to be able to show statistical significance. Based on estimates from informal testing, we needed at least 40 participants. To get enough respondents, we chose to recruit technology researchers with experience in media technology and production, of which there is a much larger population. To attract enough participants, we designed our study so that it could be completed online, and in 15 mins or less. We used email distribution lists to advertise our study to everyone working at BBC Research and Development, and the Electronics and Computer Science department at Queen Mary University of London.

4.1.2 Task

In radio production, music has to be removed when turning a broadcast programme recording into a podcast. This is due to music licensing issues, which are different for downloadable content than they are for radio broadcasts. The music is removed by editing the audio in a DAW that uses a waveform to visualize the audio. Although the waveform can be used to allow users to distinguish between music and speech, at typical zoom levels it is not always visible. This means that removing the music can be a slow and tedious process. Therefore, we chose the task to be segmentation of music and speech.

4.1.3 Test interface

To conduct the experiment, we developed a web-based test interface, shown in Figure 4.1.3.1. The interface displayed the overall visualization as well as a zoomed-in view above it. The participant could navigate the audio by clicking on either view, which would seek to that position in the audio. Buttons below the visualization controlled play/pause, zoom level and setting the in and out points of the selection. The selection was displayed by highlighting both the visualization itself and a slider below it. The selection could be adjusted by dragging either end of the slider. Training was provided using a “pop-up tour”, which guided the user through the interface’s features and operation using a series of pop-up text boxes. The interface also captured the participant’s questionnaire responses and preferences.

We generated the visualizations of the audio clips using a plugin framework we developed called “Vampeyer”, which mapped the results of audio analysis plugins to a bitmap image. We then integrated those images into our test interface by developing an interactive web-based audio visualization library called “BeatMap”. We describe Vampeyer and Beatmap in greater detail in Appendices A.2 and A.3.

4.1.4 Conditions

As we saw in Section 2.1.2, radio producers edit audio with DAWs that use audio waveforms as the medium for interaction. In Section 2.3.2, we also saw that by adding colour to audio waveforms, additional semantic information about the audio could be displayed, while retaining the familiar audio waveform visualization. We were interested in measuring the performance of both the normal audio waveform and a semantically enhanced waveform, and also comparing them directly. Therefore, we chose to use the following conditions for the audio visualizations. An example of each condition is shown in Figure 4.1.3.2.

1. None: No visualization, audio only.

2. Waveform: Audio waveform in a single colour.

3. Enhanced: Audio waveform with colour mapped to low energy ratio.

We included a condition in which there was no visualization (C1) to use it as a baseline to measure the performance of the normal waveform. For this condition, the participant must rely purely on listening to the audio. For the other two conditions, they are able to both listen and use the visualization.

For the enhanced visualization, we extracted an audio feature that was relevant to the task and mapped it to the colour of the waveform. Speech-music discrimination (SMD) is a research topic that aims to automatically segment speech and music. This research often targets recordings of radio broadcasts (Goodwin and Laroche 2004; Wieser, Husinsky, and Seidl 2014; Saunders 1996; Pikrakis, Giannakopoulos, and Theodoridis 2008; A. Pikrakis, Giannakopoulos, and Theodoridis 2006).

We wanted to select an audio feature that would assist the participant in completing their task. However, we also wanted there to be an element of human judgement involved in the task. If we chose an audio feature that was very accurate, the algorithm would be doing all of the work, and the human would only be processing the results. To avoid this situation, we restricted the performance of the audio feature by only allowing a one-dimensional (scalar) feature.

Low energy ratio (LER, also known as ‘silent interval frequency’, ‘energy contour dip’ and ‘low short-term energy ratio’) is the frequency in which the energy of a signal falls below a threshold. This is a simple but effective feature which exploits the fact that speech has frequent silent gaps between words, whereas music does not. Panagiotakis and Tziritas (2005) found that on its own, LER can classify 50% of music segments with 100% precision.

We calculated the low energy ratio by extracting the RMS of the signal (20ms frames, no overlap) and counting the proportion of frames which fell below a threshold (see Equation 4.1.4.1). The threshold can be set as a fixed value (Liang et al. 2005; Panagiotakis and Tziritas 2005), a function of the moving average (Ericsson 2009), or a function of the moving peak value (Saunders 1996). After empirically testing a variety of radio programme recordings, we chose to use a function of the moving average, which we configured as 3% of the mean RMS in a one second sliding window.

\[\begin{aligned} x_{rms}(n) &= \sqrt{\frac{1}{F} \displaystyle\sum\limits_{i=Fn}^{F(n+1)} x_i^2}\\ X &= \{x_{rms}(n) \mid 0 \leq n < \frac{f_s}{F}\}\\ X_{low} &= \{x\ | \ x \in X \wedge x < 0.03 \overline{X} \}\\ \text{LER} &= 100 \times \frac{|X_{low}|}{|X|} \end{aligned}\qquad(4.1.4.1)\]

where \(x_i\) are the audio samples, \(f_s\) is the audio sample rate, and \(F\) is the number of samples in each frame.

We coloured the waveform by mapping the low energy ratio to a linear gradient between two easily distinguished colours. We used blue for representing speech to match the waveform colour of the most commonly used DAW in BBC Radio. To represent music, we inverted the RGB values of the waveform colour to produce pink.

4.1.5 Audio clips

We used radio programme recordings for the audio clips, by choosing a representative selection of programme formats, musical genres and radio stations, shown in Table 4.1.5.1. We sourced the audio content from recordings of BBC Radio broadcasts. We selected ten 5-minute clips that contained only one section of music, with speech before and after, where the music had clear start and end points. We checked the performance of the LER feature to ensure it had only modest performance and was consistent between clips. We cut the clips so that the music was in a different position in each clip. We chose to use nine 5-minute clips so that the segmentation tasks could be completed in around 15 mins, and one clip for training.

4.1.6 Metrics

We wanted to measure the user’s performance in completing the speech/music segmentation task when using different audio waveforms. We were interested not only in measuring whether there was an actual difference in the task performance, but also measuring whether the user perceived any difference in their performance. To do this, we used both quantitative and qualitative metrics.

4.1.7 Hypotheses

We expected that for all of the metrics, the enhanced waveform would perform better than the normal waveform, and the normal waveform would perform better than no visualization. Specifically, we defined the following hypotheses:

1. Effort: Audio visualization affects the actual and perceived effort required to segment music from speech, where C1 requires more effort than C2, and C2 requires more effort than C3.

2. Time: Audio visualization affects the actual and perceived time taken to segment music from speech, where C1 requires more time than C2, and C2 requires more time than C3.

3. Accuracy: Audio visualization affects the actual and perceived accuracy of segmenting music from speech, where C1 is less accurate than C2, and C2 is less accurate than C3.

4.1.8 Procedure

Before the study began, we asked the participant to read an information sheet and agree to a consent form. There were four stages to the study:

Stage 1: Demographics We asked the participant about their gender, age and the following questions, to gauge their familiarity with DAWs and professional experience:

• Do you understand what an audio waveform is? [Yes/No]

• Have you previously used any consumer audio editing software? ( e.g. Audacity, GarageBand) [Yes/No]

• Have you previously used any professional audio editing software? (e.g. ProTools, Logic, Cubase/Nuendo, SADiE, Startrack) [Yes/No]

• How many years (if any) have you worked with audio in a professional capacity? [number]

Stage 2: Training We used a ‘pop-up tour’ to overlay a sequence of text boxes on the interface (see Figure 4.1.3.1). These explained the features and operation of the test interface, then prompted the user to complete and submit a training task using the enhanced waveform (C3). We measured the error of the training task and did not allow the participant to continue until they had completed the task successfully (as defined in Section 4.1.9).

Stage 3: Segmentation task The participant used the test interface to mark the position of music in a speech recording, then submit their result. We logged and timestamped the participant’s actions, including seek, play/pause, zoom and mark. This exercise was repeated three times for each condition, for a total of nine tasks. We wanted to gather feedback directly after each condition to capture the participant’s reaction, and to avoid possible confusion due to switching too often. To achieve this, we grouped the presentation of the conditions (e.g. AAABBBCCC) rather than interleaving them (e.g. ACACBABCB).

Each audio clip can only be used once per participant, otherwise they would be able to remember the location of the the music. To define a balanced sequence for the audio clips, we used a Williams design Latin square (Williams 1949), generated using the crossdes package in R (Sailer 2013). We used Latin squares to block out the effect of the order of presentation, and a Williams design, which is balanced for first-order carryover effects. As the sequence length is an odd number, the Williams design uses two Latin squares to produce an \(18\times9\) matrix.

To generate the order of visualizations, we needed to produce a balanced \(18\times3\) sequence. We did this by taking three columns from our \(18\times9\) Latin square and mapping the values 1–3, 4–6 and 7–9 to 1, 2 and 3, respectively. By calculating the carryover effect of each column of the \(18\times9\) matrix, we found that the middle three columns had the minimum carryover effect, so we used them for our visualization sequence.

After completing the three tasks for each condition, the participant rated the workload of those tasks using the NASA-TLX metrics. We captured the responses using sliders with the labels for each question on either end.

Stage 4: User preference After all the tasks were completed, we asked the participant to select which condition they thought was the easiest, and which was the most frustrating. We used thumbnail images from Figure 4.1.3.2 to remind them of what each condition looked like.

4.1.9 Analysis

As the experiment was unsupervised, we wanted to ensure that all participants completed the tasks correctly. We did this by measuring the error of the task and rejecting participants that submitted a response with an error of 5 seconds or greater. We chose this threshold after running informal tests which showed that under supervision, all responses had an error of up to 5 seconds. We calculated the error as the sum of the absolute error of the in-point and out-point.

We tested for statistically significant differences in the TLX ratings by using SPSS to conduct a repeated measures analysis of variance (rANOVA) (Shalabh 2009, p. 409). We tested the underlying assumptions of normality and sphericity by plotting the distribution for each condition and using Mauchly’s Test of Sphericity (Shalabh 2009, p. 415). Mauchly’s Test indicated that the assumption of sphericity had been violated for the effort metric [\(\chi^2(2) = 9.657, p=.008\)] and temporal demand metric [\(\chi^2(2) = 17.918, p<.001\)]. Therefore, we corrected the degrees of freedom using the conservative Greenhouse-Geisser Correction (Shalabh 2009, p. 416). We then used Tukey’s honest significant difference (HSD) post-hoc test (Shalabh 2009, p. 139) to make pairwise comparisons between the mean values of the metrics.

Our study contained one fixed effect (visualization) and two random effects (audio clip and participant). We could not re-use audio clips for the tasks as participants would remember the position of the music. Therefore, as we did not have results from every participant for every combination of audio clip and visualization, our dataset was incomplete. As we are using a repeated measures design, we would normally analyse the results using a repeated measures ANOVA. However this requires a complete dataset, so we were unable to use this analysis.

We used a linear mixed model to analyse the results of the metrics because it can account for an incomplete dataset and a repeated measures design (Gueorguieva and Krystal 2004). We used SPSS to perform a linear mixed effects analysis (Shalabh 2009, p. 274) of the relationship between visualization and the three performance metrics (seek actions, task completion time and task error). We configured the visualizations as a fixed effect and the audio clips and participants as random effects. We tested the underlying assumptions of homoscedasticity and normality by plotting the residual errors and visually inspecting them, which did not reveal any obvious deviations. We then used the Least Significant Difference (LSD) post-hoc test (Shalabh 2009, p. 137) to make pairwise comparisons between the mean values of the metrics.

4.2.1 Quantitative metrics

We analysed the quantitative performance metrics using a linear mixed model (Shalabh 2009, p. 274). Figure 4.2.1.1 shows the mean values and confidence intervals of the performance metrics and Table 4.2.1.1 lists the statistical significance of the pairwise comparisons between the conditions.

4.2.2 Qualitative metrics

We analysed the TLX ratings using repeated measures ANOVA (Shalabh 2009, p. 409), which found that the audio visualization had a significant effect on the TLX ratings from participants [\(F(12,152)=3.552, p<.001\)]. Figure 4.2.2.1 shows the mean values and confidence intervals of the TLX metrics. Table 4.2.2.1 lists the statistical significance of the pairwise comparisons between the conditions.

The TLX ratings from participants show that compared to both the normal waveform (C2) and no visualization (C1), the enhanced waveform (C3) was perceived to be less mentally and physically demanding, better performing, less frustrating and requiring less effort (all \(p<.05\)). The participants also rated the enhanced waveform as less temporally demanding than the normal waveform (\(p<.05\)). Compared to no visualization (C1), participants rated the normal waveform (C2) as being less frustrating and requiring less effort (both \(p<.05\)).

The effort ratings confirm hypothesis H1 (effort); the temporal demand ratings confirm hypothesis H2 (time) for C2\(>\)C3; and the performance ratings confirm hypothesis H3 (accuracy) for C2\(<\)C3.

The participants rated the enhanced waveform (C3) as significantly less physically demanding than the normal waveform (C2). This was surprising, as all of the tasks were conducted using a screen and mouse, so did not require much physical exertion. We do not know how participants interpreted this metric, but it’s possible that some may have classified the movement of the mouse or number of mouse clicks as physical activity.

Participants were asked to select which audio visualization was the easiest to use, and which was the most frustrating. The results are shown in Figure 4.2.2.2. The enhanced waveform (C3) was rated as the easiest to use by three quarters of the participants. Having no visualization (C1) was rated as the most frustrating condition by two thirds of the participants. The normal waveform (C2) was not considered by many to be the easiest, nor the most frustrating.

5.1.1 Transcription

As we discovered in Chapter 3, radio programmes are assigned a slot in the broadcast schedule, so producers have a strict deadline for finishing their programme. Programmes are sometimes scheduled up to three weeks in advance, but sometimes as little as one week in advance. This means that producers have very little time to spare. If a programme’s budget allows, interview recordings can be sent to a transcription service where they are transcribed by a person overnight. However, many programmes do not have the budget for this, in which case the producer transcribes the recordings themselves.

Transcripts are used to help the producer make editorial decisions, but are usually not published. For this reason, the transcripts only have to be sufficiently accurate to use for editing. Both Whittaker and Amento (2004) and Sivaraman, Yoon, and Mitros (2016) found that the errors in the transcripts did not prevent users from being able to edit using them. However, both also found that users wanted to be able to fix incorrect words in the transcript.

Requirement: Our semantic editing system needs to be able to produce a transcript quickly and cheaply. It should be accurate enough to be useful for editing, and allow for correction where necessary.

5.1.2 Editing

There are already well-established systems and software packages in place for producing radio programmes. As we discussed in Section 2.1.1, producers use a digital audio workstation (DAW) to select the parts of each interview that they want to use in their programme, and to arrange them into a narrative. In Chapter 3, we learned that the BBC provides two different DAWs – dira! StarTrack (made by SCISYS) and SADiE (made by Prism Sound). Both of these use waveforms to visually represent audio to help the user navigate the recordings. The edits performed in a DAW are “non-destructive” because the original recordings remain untouched (see Section 2.1.1). This allows the producer the flexibility to adjust or undo their decisions at any point during the editing process.

For the documentary production we observed in Chapter 3, a specialist sound engineer, known in the BBC as a studio manager (SM), was brought in on the last day of production to ensure that the sound was well balanced, and to do any advanced editing that was required. This included removal of unwanted “umm”s or breaths in a process called “de-umming”. The SM for the observed documentary suggested that being able to de-umm speech in a way that is inaudible to the listener is a skilled task that requires precision, judgement and experience.

Music is often included in a programme, either as a theme tune, a short interlude or in the background. We observed that producers select the music either from their personal collection, or from one of a number of services for finding commercial or rights-free music, such as Audio Networkhttps://www.audionetwork.com/ (accessed 15.08.2016)

. The music is added and edited using the DAW.

At the end of the editing process, the editor listens to the programme with the producer to give their feedback and sign-off. As part of this process, they both listen out for repeated words or phrases. However, this is only usually a problem in drama production where multiple takes of the same lines are recorded.

Requirement: Our semantic editing system needs to be able to select and arrange parts of audio recordings. Given that there are well-established radio production systems for advanced editing tasks such as de-umming and addition of music, it also needs to be able to integrate with these so that it can be used in professional radio production.

5.2.1 Transcript

We considered three factors when choosing a transcription method – turnaround time, cost and accuracy. Manual transcriptions are nearly 100% accurate, however they are expensive (about $1 per minute) and slow (typically 24 hours). Automatic transcriptions are imperfect, but cheap (about $1 per hour) and fast (quicker than real-time listening). Our system requires quick and cheap transcripts that are sufficiently accurate, so we chose to use automatic transcripts generated by a state-of-the-art commercial web servicehttps://www.speechmatics.com/ (accessed 15.08.2016)

. Whittaker and Amento (2004) and Sivaraman, Yoon, and Mitros (2016) found that users want to be able to correct the transcript, so we designed our system so that users can fix incorrect words. Rubin et al. (2013) did not include this feature as they used verbatim transcripts.

5.2.2 Interface

Our semantic editing system needs to be able to select and arrange parts of audio recordings. To achieve this, we used the same drag-and-drop interface as Hyperaudio Inc. (2016) as it is a simple method for extracting and re-ordering clips. As shown in Figure 2.4.4.1, it also allows clips from different recordings to be added and re-arranged. Casares et al. (2002), Sivaraman, Yoon, and Mitros (2016) and Berthouzoz, Li, and Agrawala (2012) used a select/delete interface, where parts of an individual transcript could be chosen or removed, and Whittaker and Amento (2004) and Rubin et al. (2013) used a cut/paste/delete interface.

We designed our interface to be browser-based, as the BBC corporate policy meant that it was not possible to install new software on the producers’ computers. This came with the added benefit of allowing users to work from anywhere in the world on any operating system, but the downside is that they have to be connected to the Internet.

5.2.3 DAW integration

Our system needs to be able to integrate with the existing radio production tools. We designed Dialogger to be used as the first stage of the editing process, and to smoothly integrate with the DAWs that are used in BBC Radio. We achieved this by providing a novel feature to export edited content from our system, either as a WAVE audio file or as an edit decision list (EDL).

The first option exported a single WAVE audio file of the edit. This method is a destructive edit, in that it throws away the pieces of the recording which weren’t selected. The other option exported an EDL, which contains metadata about which parts of an audio or video recording make up an edit. These can be read by the two most common audio editors used at the BBC – SADiE and dira! StarTrack. This method is considered non-destructive as the full original recordings are retained and the edit points can be re-adjusted in the audio editor.

5.2.4 Excluded features

Rubin et al. (2013) included features for finding music tracks and creating loops within them. In Chapter 3, we found that specialist tools are already used for finding and choosing music, and that editing of music is already efficiently handled by the DAW. Therefore, we chose not to include features for adding or editing music.

Rubin et al. (2013) also included detection of repeated words and phrases. We chose not to include this, as we found in Chapter 3 that repeats are only an issue in drama production. As the production of drama involves a very different workflow of recording multiple takes of lines from a script, we chose to focus on production workflows for pre-recorded content in our system design or study.

5.2.5 System description

This section gives a brief overview of Dialogger, including its functionality and operation. A screenshot of the interface, and numbered list of its main features, are shown in Figure 5.2.5.1.

5.3.1 Approach

Gaining access to radio producers can be difficult as there are not many of them and they are normally very busy. For example, Kim, Oard, and Soergel (2003) attempted to recruit radio producers but was unsuccessful because the producers were “so highly tasked”. However, in this case we were able to recruit professional radio producers from the BBC Radio due to the access available to us from working within the BBC.

As we would not be able to recruit a large number of participants, we chose to take a qualitative approach to maximise the depth of the information gathering. To take advantage of the available access to the work environment, we chose to use contextual inquiry techniques to allow us to learn about the workflows, tools, and the social, technical and physical environments. This took the form of an initial interview, followed by a period of observation, then a final interview.

Radio producers find it difficult to step away from their day-to-day work for too long. To account for this, we designed our study so that the tasks overlapped with the production of the programme that the participant was working on at the time, and the audio content they needed to edit that day. We interviewed and observed participants in their normal working environment to ensure that the production workflow was not affected by an artificial setting.

5.3.2 Recruitment

We invited professional radio producers with at least five years of experience to take part by sending an email to departments in BBC Radio that create pre-produced factual programmes. Drama programmes were excluded from the study as we saw in Chapter 3 that their production workflow involves making multiple recordings of lines in a script and selecting the best ones. This is a sufficiently different process to other programme genres that it warrants a different interface.

Five participants (P1–P5) were recruited (4 male, 1 female) who each had between 6 and 20 years experience working as a radio producer. Although we had a small number of participants, the experience of the producers and the depth of the study means that their feedback should carry significant weight. Five participants is also considered sufficient for identifying most usability problems (Nielsen and Landauer 1993).

During the experiment, the participants worked on programmes of different lengths from a range of genres: P1 produced a single 27-min documentary, P2 produced a 27-min documentary as part of a ten-part series, P3 produced a single 45-min documentary, P4 produced a 14-min archive programme (based around material from the archive) as part of a ten-part series, and P5 produced a single 27-min magazine show (covering multiple stories on a single theme).

5.3.3 Procedure

We designed a five-stage experimental procedure that followed a typical contextual inquiry format of interview/observation/interview. In addition, we recorded some simple metrics such as task completion time and feature usage.

Stage 1: Background interview The participant was first given an overview of the project and the study, and asked to complete a consent form. This was immediately followed by a semi-structured interview to learn about the participant’s background, their existing production workflow and the tools they used. The investigator asked the participant to describe the radio production process in detail, and used follow-up questions to clarify their understanding. This information was recorded using written notes.

Stage 2: Dialogger training Each participant was trained on the functionality of the Dialogger interface using a pre-written “tool-tip tour”, in which the participant was presented with a sequence of instructional pop-up dialog boxes overlaid on the interface. This ensured consistency of training between participants. Each participant was then issued with a series of tasks that utilised all of the system functionality. The investigator observed this stage and wrote notes of any unexpected behaviour or stumbling blocks.

Stage 3: Task observation Each programme is composed of a number of interviews on a single topic, or set of topics. We observed the participant while they logged and rough-edited two different interviews for the same programme. They did this by editing an interview under each condition – one using their existing production workflow, and the other using Dialogger. The order of the conditions was counterbalanced.

The investigator sat beside the participant during the task and wrote notes on the workflow, tools, generated metadata, usability issues, task completion time, and unexpected reactions or usage. The actions of the participant on Dialogger were logged electronically. After they completed the task, they were asked to rate each condition using the NASA Task Load Index metrics (Hart and Staveland 1988).

Stage 4: Interview We conducted a semi-structured interview about the participant’s experience of each system and how they compared. The investigator questioned participants about the advantages and disadvantages of each workflow, then asked about any specific topics, issues or questions that arose during observation. The audio from this interview was recorded and transcribed to allow for further detailed analysis.

Stage 5: Longitudinal deployment Each participant was then given access to Dialogger for a further month, and was invited to continue to use it if they wished. Each week, they were asked via email whether they had been using the system, and if so, which features they valued most/least or were missing. During this time, their usage of Dialogger was also logged electronically.

5.3.4 Analysis

Our study produced observation notes, interview transcripts and metrics. We used thematic analysis with open, flat coding to interpret the textual data, and we used statistical analysis to process the numerical data, as described below.

5.4.1 Existing workflow

In this section, we consider the comments made by participants about their existing workflow. We have organised these by the categories from the thematic coding (see Table 5.4.1).

5.4.2 Semantic editing workflow

This section discusses the results and themes that emerged from the evaluation of the semantic editing interface. Participants were first introduced to Dialogger through a training stage (Stage 2). All of the participants completed the training without any major issues. However, this stage highlighted a requirement for keyboard shortcuts which was not previously identified. P2 and P3 kept trying to use the space bar to start and pause audio playback. This is a common shortcut in most DAWs which these participants naturally reached for. Reports on previous semantic editing systems have not mentioned keyboard shortcuts, however they could be used to assist the editing process.

In the rest of this section, we will discuss each of the themes that came out of the thematic coding (see Table 5.4.1).

5.4.2.3 Mark-up

During the study, P1 and P3 copied the transcript text from the interface into Microsoft Word. They reported that they did this because there was no annotation functionality available within Dialogger.

They inserted paragraph breaks, added notes after paragraphs, and highlighted desired parts of the transcript in bold. Once the transcript was annotated in Microsoft Word, they went back to Dialogger, found the parts of the transcript they wanted by scrolling though the text, then dragged and exported each clip individually as a WAVE file.

“it would be better to take raw lumps of transcripts and plonking them in Word because Word has higher functionality than this” (P3)

Producers are very familiar with the Microsoft Word interface so a later version of our system could seek to provide a similar interface. This would allow producers to make annotations in the same way they do already.

“With text editing, the reflexes are very much Microsoft Word” (P4)

The most basic feature that could be added is highlighting, which is often used to note parts of interest

“If you just put a little star or underline or something simple to mark things, that would be a big gain for a small change” (P3)

5.4.3 Metrics

5.4.3.1 Time

We recorded the time participants took to complete the observed tasks (see Figure 5.4.3.1.1). As various recordings of different lengths were used for the existing and new workflows, we divided the edit time by the audio duration to calculate the relative edit time. In all cases, the producers were able to run the ASR processing as a background task so this was not included in the calculation. P1, P2 and P3 used the semantic editor after their existing process, while P4 and P5 did the opposite. However, as different recordings were edited on each system, the presentation order is not expected to affect the results.

The mean average time for semantic editing was 0.79 minutes per minute of audio, versus 1.13 minutes for the existing method, which is a 44% improvement. However, a paired t-test revealed that there was no statistically significant difference (p = 0.24). This is due to the small sample size and the large variations in timings resulting from P4 and P5 not doing any logging, and P2 printing out and annotating their transcript before editing. Semantic speech editing may have the potential to reduce the time needed for logging and rough-editing material, but further investigation with a larger sample and consistent workflow is required to measure time performance.

5.4.4 Longitudinal deployment

After the interviews and observations were complete, the participants were given access to Dialogger for a further month (Stage 5). During this time their actions were logged electronically and they were emailed each week to ask which features they found useful, or were missing. P3 was unavailable immediately after the study, so could not take part in this stage.

Most of the comments received in the longitudinal deployment were already picked up by the first part of the study. In the remaining comments, all of the participants said they enjoyed being able to use Dialogger outside of the office and at home. Some reported that they had issues uploading content with their slow network connections, and P2 suggested that allowing multiple simultaneous uploads would allow them to leave it running overnight.

Participants were given access to the system for one month after the study. The logs from the interface were analysed to see how the participants used Dialogger during this stage of the study. All of the participants continued to use the semantic editor of their own accord as part of their work. The total time spent by the four remaining participants (P1, P2, P4, P5) using Dialogger in the month-long deployment period was 23 hours and 58 minutes. Over 14 hours of those were from P2, with P4 using it for 5 hours, P1 for 3 hours and P5 for 20 minutes. During this period, 86 recordings were uploaded and 58 audio edits were exported.

Users could navigate the content by either clicking on the waveform or by clicking on a word in the transcript. The interaction log showed that over 98% of navigation actions were executed by clicking on a word, which shows a clear preference for navigating by text compared to waveforms.

5.5.1 Listening

Logging is an important process that primarily involves labelling and organising content, however it is time consuming. Some participants found the logging process to be valuable because it gave them the opportunity to listen back through their recordings, and make connections between various bits of content. This cross-referencing could also be assisted by providing links between words within and between recordings. For example, selecting a word could display and replay other mentions of that word in other recordings.

Another important reason for listening is to ensure a high “sound quality”. Participants wanted to avoid low quality audio such as “umm”s, mumbling, coughing and excessive background noise, but they also wanted to ensure they didn’t miss any high quality audio moments that might not have been identified using the transcript. Faster playback is already used in radio production to reduce the time spent listening to material, however more sophisticated time compression algorithms such as those described by Arons (1997) could be used. Time compression has not been included in previous semantic editing systems, but should be considered in the future, especially as Vemuri et al. (2004) found that the maximum time compression factor is significantly higher when an automated transcript is present.

Removal of “umm”s and breaths through de-umming is either done by the producer themselves or with the help of a sound engineer, depending on the producer’s experience and time pressure. To maintain sound quality, the removal of umms/breaths must be audibly transparent and participants reported that this can be difficult to achieve. Previous semantic editing systems have included functionality to remove umms (Berthouzoz, Li, and Agrawala 2012) and breaths (Rubin et al. 2013), however these were made possible because the manually generated transcripts explicitly transcribed those items. ASR systems are normally trained to ignore umms/breaths rather than transcribe them, which prevented us from including this functionality. A transcription system that includes these would allow us to add this functionality, however further research is needed into the extent to which de-umming can be automated in this way.

5.5.2 Annotation

All of the participants used a script document to structure and assemble their programme, and as a medium to inform and gather feedback from the presenter about the content and layout of the programme. Although the clipboard of our semantic editing system acted much like a programme script, the participants did not use it in that way because it was missing some key functionality for annotation and collaboration.

Annotation features were an important requirement that we did not pick up on during the design specification, and which have not been included in previous semantic speech editing systems. Two participants in our study deviated from the expected workflow in order to annotate the transcript, and the other participants noted the absence of such functionality. Participants wanted to be able to annotate the transcripts as they would with a word processor, in order to highlight or rate particularly good parts of their recordings, add personal comments, and to segment and label the content.

A simple change to achieving this would be to allow the transcripts to be formatted, and for textual comments to be inserted and edited. Furthermore, the drag-and-drop editing could be replaced with cut/copy/paste similar to Whittaker and Amento (2004) and Rubin et al. (2013). An alternative approach could be to add semantic speech editing functionality to a word processor, rather than adding word processing functionality to a semantic speech editor.

5.5.3 Collaboration

Scripts are used as a tool for collaborating with colleagues such as the presenter because the programme’s content and structure can be quickly reviewed and commented on by others without them having to spend time downloading and listening to the audio. Our semantic editing system was designed for individual access to transcripts and edits, however this meant that they could not be shared with the presenter. A better approach would be to allow multiple users to navigate and edit the same material. This could be achieved using operational transformation (Sun et al. 2004) which can support concurrent users editing the same content. Participants were also interested in tighter integration with the DAW. The same technology could be used to create bi-directional integration with DAWs, so that any edits made in the DAW are automatically updated in the semantic editor and vice-versa.

5.5.4 Portability

Participants reported that the open-plan office environment in which they worked was often noisy and distracting, and that they had difficultly working on screens for extended periods. As a result, many reported that they work from home to get away from the office or print transcripts so they can get away from the screen. A more portable semantic speech editing system would allow producers the flexibility to work where they wanted.

Digital pen interfaces such as the Anoto system could be used to create a paper-based semantic editor that can be used anywhere and does not involve screens. Additionally, it naturally supports freehand annotation and may be a better medium for face-to-face collaboration. Klemmer et al. (2003) has previously explored how speech can be navigated using paper transcripts and Weibel et al. (2008) describes how an Anoto system can be used to edit digital documents, however these approaches have yet to be combined.

5.5.5 ASR transcripts

Participants reported that the automatically-generated transcripts were sufficiently accurate for editing, supporting similar previous findings from Whittaker and Amento (2004) and Sivaraman, Yoon, and Mitros (2016). This is helped by the fact that radio producers record the audio themselves, and can use their memory to cope with inaccuracies. Most participants were only interested in correcting errors that were distractingly wrong, which were often names or locations related to the story. However, as these are known ahead of time, they could be provided to the ASR system as a way to tweak or expand the language model.

Currently transcripts of each programme are not published due to the high cost and overhead, however several participants were interested in fully correcting their transcripts so they could do this. The availability of ASR transcription could have the potential to extend the scope of radio production to include publication of transcripts. This could help to improve discoverability of programme content, especially if word timings were included.

5.5.6 Outcome

Based on the results of this work, we developed the prototype further to take into account the feedback from the producers in our study. We handed the prototype over to a development team at the BBC who turned it into an officially supported production tool. This has allowed producers from around the BBC to use the tool as part of their normal workflow. As of October 2016, the system had 45 active users and had processed 265 audio recordings.

Independently of our research, in late 2017, a commercial semantic speech editing system called Descript was releasedhttps://www.descript.com/

. Descript is an audio production interface that uses ASR and manual transcription to allow users to transcribe and edit their audio. The interface, shown in Figure 5.5.6.1, includes annotation features such as bold, italic, highlighting and time markers. Rather than using a drag-and-drop technique for editing the audio, Descript uses strikethrough annotation to remove segments of audio. The transcript can be corrected by switching from editing mode to correction mode, and the transcript includes speaker diarization. In addition to these features, Descript includes an integrated waveform editor that can be used for fine editing and inserting cross-fades. This recent commercial interest in semantic speech editing suggests that there is interest in using this technology for audio production, and that ASR systems are now sufficiently accurate to support it.

6.1.1 Barcodes

Barcodes printed on paper transcripts have been explored as a method of navigating video recordings by using a device to scan the barcode and play the video from that position. Video Paper (Hull et al. 2003) was a system that printed video keyframes and barcodes down the side of a paper transcript. Each barcode linked to a position in a video, which was downloaded from a database and played on the scanning device. Books with Voices (Klemmer et al. 2003) was a similar system that tested this approach on oral historians who found it effective for assisting a transcript editing task. Erol et al. (2007) went a step further by embedding the video data in the barcode, removing the need for a database. HotPaper (Erol, Antúnez, and Hull 2008) removed the need for barcodes by using a camera to measure the whitespace between words and matching that to unique patterns in the text.

Barcode-based systems provide a link between text and media. They use real paper, can be annotated freely, are easy to generate and are robust to photocopying. However, they do not provide a convenient method of capturing annotations. It would be possible to use a handheld device to capture annotations and link them to a particular barcode. However, this would require the annotations to be entered into a handheld device, rather that just written on the paper. Additionally, the size of barcodes means that they cannot be used for each word, which affects the precision of the system.

6.1.2 Digital pens

A digital pen looks and functions as a normal pen, but includes an on-board infrared camera that tracks the position of the pen while it writes on paper. Digital pens must be used in combination with paper that has a unique non-repeating dot pattern printed onto it using a standard colour laser printer. By reading this pattern, the pen can calculate exactly where it is when touching the page. The pen records its position up to 100 times a second. Depending on the pen and software, this information can either be streamed live via Bluetooth, or downloaded as a batch onto a computer. The digital pens that use this patented technology (Fåhraeus 2003) are exclusively manufactured and licensed by Anoto Group. As such, this technology is commonly referred to as the Anoto dot pattern.

ChronoVis (Fouse et al. 2011) was a note-taking system that used the Anoto pattern for recording synchronised hand-written notes during playback of a video. An accompanying screen interface allowed users to click on the digital display of the handwritten notes to navigate to that position in the video. Weibel et al. (2012) conducted a longitudinal study of ChronoViz for use in observational research. The results show that notes became a mixture of linear notes and symbolic representations. Asterisks, stars, lines and simple shapes were used as bookmarks for later referral, or for counting events. The flexibility of freehand notes also enabled use of arrows in various contexts, such as to indicate direction and actions.

PADD (Guimbretière 2003) was a concept for a system of editing documents that used the Anoto pattern to allow users to move from digital to paper and back again. ProofRite (Conroy, Levin, and Guimbretière 2004) was the first full implementation of a PADD system, which overlaid annotations made on paper into a word processor. The annotations are anchored to the text, such that they “reflow” when the text is moved. Through informal feedback, users suggested that their annotations should translate into actions such as delete. PaperProof (Weibel et al. 2008) interpreted edit annotations and automatically applied them to the document. Gestures for delete, insert, replace, move and annotate were translated into modifications in a word processor, and intelligent character recognition was used to digitise any hand-written text. The interpretation of annotations allows for a two-way interaction between the digital and paper representations. We could not find any user studies of the PaperProof system.

6.1.3 Digital ink

Digital ink refers to technology that digitally captures and responds to the movements of a pen, such as a stylus. Typically, digital ink systems use a device with a backlit screen and a touch-sensitive interface like a tablet PC. Several systems have experimented with using a stylus with interactive sliders to provide advanced control for navigating video content. Examples include LEAN (Ramos and Balakrishnan 2003), Zlider (Ramos and Balakrishnan 2005) and MobileZoomSlider/ScrollWheel (Hürst and Götz 2008). However, these systems are limited to the navigation of content, without changing or labelling it. As we will see in this section, digital ink interfaces can also be used to annotate and edit media.

Marquee (Weher and Poon 1994) synchronised handwritten notes with a live video recording by using a horizontal line gesture to mark a timestamp. Dynomite (Wilcox, Schilit, and Sawhney 1997) synchronised handwritten notes to a live audio recording, and allowed the user to categorise their annotations using keywords, and to highlight regions of audio using a button. In the evaluations of each of their systems, Weher and Poon (1994) and Wilcox, Schilit, and Sawhney (1997) both found that users took fewer notes when using the digital ink system, and that they wanted to go back and use the audio/video to improve the notes afterwards.

Videotater (Diakopoulos and Essa 2006) was another digital ink interface for segmenting and annotating pre-recorded video clips. A vertical line gesture on a video timeline split the video into a clip, which could be labelled with handwritten notes. WaCTool (Cattelan et al. 2008) also included features for annotation, but added real-time collaboration and editing tools. Users could assign a “skip” command, which is analogous to removal, by pressing buttons at the start and end of an unwanted region. Video as Ink (Cabral and Correia 2016) allows users to “paint” video frames onto the tablet interface and then edit the video by erasing unwanted frames. Videotater, WaCTool and Video as Ink all rely on the manipulation of video thumbnails, which are unavailable in radio production.

Finally, as we saw in Section 2.4.3, RichReview (Yoon et al. 2014) allowed users to trim or tidy voice recordings by drawing a line through words or pauses to remove them. An evaluation with 12 students found that the editing features were considered easy to use and efficient for removing “umm”s and long pauses.

6.1.4 Summary

In this section, we have seen that barcodes, digital pens and digital ink have been used to link paper to digital media. Barcodes are a simple way to achieve this using real paper, which can be annotated freely and is easier to read. Although an additional device is needed to capture annotations, a camera on a mobile phone could be used to read the barcodes and play the digital content. However, barcodes only provide a one-way link from paper to media as annotations on the paper are not captured. Barcodes also occupy space on the page, which limits the precision with which they can be used.

Digital ink interfaces have both a screen and a stylus. This allows them to both capture freehand annotations, and respond by replaying the original media, or erasing mistakes in the annotations. However, as digital ink interfaces use screens, they do not benefit from the improved reading speed, comprehension and cross-referencing of paper. They are often bulky, have a short battery life, and in the event of battery or device failure, the transcript and annotations are lost. Electronic paper is a technology that attempts to emulate the benefits of reading from paper. Although it has been commercially successful through its use in e-readers, studies have found that e-paper has a higher reading time, worse comprehension and higher eye fatigue than reading from normal paper (Jeong 2012; Daniel and Woody 2013). Additionally, we could not find any systems that provided the level of interaction that would be needed to mark-up a transcript.

Digital pen interfaces combine many of the benefits of both barcode and digital ink interfaces. They use physical paper, which is better for reading, but also allow a two-way interaction by capturing freehand annotation. The pen-based interface is natural and familiar, and because the annotations are made on the paper itself, information is both accessible and backed-up in the event of device or battery failure. However, a colour laser printer must be used with proprietary software to print the required dot pattern, and the printouts cannot be photocopied. There is no easy way to undo or erase annotations, although this is an inherent problem with pens in general. We have seen that digital pen technology has successfully been applied to text editing (Weibel et al. 2008) and media annotation (Fouse et al. 2011), but we could not find any previous literature which has combined these approaches to allow semantic editing of speech content.

6.2.1 Mock-up design

In the results from Chapter 5, we saw that radio producers annotated paper transcripts using underlining (for selecting words), strikethrough (for removing words) and drawing a line down the side of the page (for selecting whole lines). We used this information as the basis for the design of our mock-up system, shown in Figure 6.2.1.1.

We used an ASR transcript and included the additional information that was generated by the ASR system. We wrote a timestamp at the beginning of each line in minute:second format, and used confidence shading (Vemuri et al. 2004) to “low-light” words with a low confidence score by shading them grey. We also put a paragraph break at speaker boundaries and wrote the speaker label at the start of each paragraph. To distinguish speaker gender, we coloured the speaker label blue for males and red for females.

To be able to capture timed edit commands using the LiveForms system, we designed our layout to use rectangular active zones that aligned with the location of each word. We placed an invisible active zone over each word to capture strikethrough, a shaded active zone under each word to capture underlining and a square shaded active zone at the end of each line to capture lines down the side.

6.2.2 Mock-up evaluation method

To evaluate our proposed layout, we recruited five radio producers (P1–P5) from BBC Radio to use our inactive prototype to annotate real transcripts as if they were editing them. Two of the participants worked in current affairs, two in science and one in documentaries. The participants had between 7 and 13 years experience in working as radio producers. Producers are very busy, so to recruit enough participants in the time available, we designed the experiment to take less than one hour. To make the study as realistic as possible, we asked each participant to provide a recent interview recording they had made, which we used to generate an ASR transcript.

To help explore our questions about annotation and editing gestures, we directed participants to employ three different strategies when using the prototype. This forced them to try different ways of interacting with the prototype, which they could later reflect on and compare. As part of the evaluation, we were interested in learning what gestures producers currently use, or want to use, without being influenced by the design or constraints of the prototype. We were also interested in directly comparing the underlining and strikethrough strategies.

We instructed the participants to follow the directions below for the first three pages of their transcript.

• Page 1: Undirected — Edit the speech by annotating the transcript as you would normally.

• Page 2: Underlining only — Edit the speech only by underlining words that you want to keep.

• Page 3: Strikethrough only — Edit the speech only by putting a line through words you don’t want to keep.

To evaluate speaker labelling, we excluded the labels from the first three pages, then included them on page 4 and asked the participant to edit the speech how they wished. Timestamps, line selection and confidence shading were included with all of the prototypes as we expected participants to be able to judge their value in situ.

After the editing task, we conducted a semi-structured interview with each participant. We asked the following questions, but also allowed participants to talk freely.

• How do you normally use a pen to edit the transcript?

• Do you prefer to select parts you want to keep, or remove parts you don’t want to keep?

• Which features of the prototype did you find useful?

• Were there any features missing that you would want added?

We categorised their responses into natural gestures, edit gestures and additional features. We counted the frequency of each response within the categories to compare the popularity of the features and editing strategies.

6.2.3 Mock-up evaluation results

The reaction to the system was overwhelmingly positive. All of the participants could immediately see the value of such a system and most remarked that it would save them significant amounts of time.

Table 6.2.3.1 lists the gestures that the participants used when editing undirected on pages 1 and 4. Each participant naturally used a different mixture of gestures for selection, removal, correction and labelling. The most common gestures for selection were underlining and line down side, with strikethrough being the most common removal gesture. Most participants combined line down side for large selections with underlining and strikethrough for finer edits.

We asked each participant whether they preferred selecting or removing words when editing the transcript. P1, P3 and P4 reported that they preferred selecting, with P2 and P5 preferring to remove words. P1 commented that selecting “felt more natural” to them, and P4 said deleting felt “counter-intuitive”. P2 and P5 reported that they preferred deleting words. P2 commented that “the challenge is to nibble away” and it was “the way my brain works”. P5 said they prefer to “get stuff out of the way”. All of the participants were certain about which they preferred, but there was no overall consensus. Additionally, Table 6.2.3.1 shows that most participants used a mixture of select and delete gestures during the undirected stage.

Four of the five participants said that they found the paragraphs and speaker information useful. Typically, interviews are recorded with a presenter and contributor, and the participants said they found it valuable to know when the presenter is asking a question. Three of the participants said that they were able to find the questions much more easily with this feature enabled. However, P2 said they found the speaker diarization to be “distracting”, particularly when it was inaccurate.

All participants said they found the timestamps and confidence shading features useful, but P2 said that the timestamps are “not needed on every line” and P5 suggested that one timestamp per page would be sufficient.

All of the participants liked being able to select whole lines at a time. P5, who prefers to remove words, asked whether a similar function could be available to delete content.

During our testing, some participants suggested adding features that were not included, or used the prototype in a way it was not designed. P3, P4 and P5 remarked that they often highlight important bits of transcripts, usually with asterisks or stars. P1 and P3 also suggested extending the underlining gesture so that underlining twice marked words as being more important. Three participants used what little space there was at the side to label the content and make notes for themselves, and P1 and P5 corrected mistakes in the transcript by writing over or above the incorrect word.

6.2.4 Requirements

The results of our mock-up evaluation showed how radio producers currently annotate transcripts, that there are mixed opinions on whether to select or remove content, and that they valued the additional features tested. We also discovered additional requirements for margins and highlighting that were not initially identified. We will now use these results to produce a set of design requirements for our system.

Edit gestures The most common edit gestures used by participants were underlining, strikethrough and line down side, with the other gestures being used less than half as often. This confirms our initial assumptions, so our design should use these gestures for editing operations.

Select vs remove There were mixed but strong opinions on whether participants preferred to select or remove content, and most used a mixture of both. This indicates that both selection and removal should be made available. In cases where both are used, remove should override select as this would allow users to draw a line down the side to select large chunks and to cross-out individual words within those.

Additional features Most participants valued speaker diarization, timestamps and confidence shading, so these additional features should be included. However, some participants reported that timestamps on every line are unnecessarily frequent and that one per page would suffice.

Margin Three participants made notes on the side of the page to label their content or to make a note for themselves, and two suggested adding a margin. With the LiveForms system, writing notes on the transcript itself would trigger the active zones that are on and below each word. Our design should include an inactive margin, as this would allow users to make freehand notes without inadvertently making edits to the speech.

Highlighting The ability to highlight regions of interest was a desired feature. Two participants suggested double underlining could be used to achieve this, however, this may be hard to detect using the LiveForms system. Alternatively, by giving the user an option to keep all of the content except for deleted words, underlining could then be used to highlight parts of interest. This mode-switching design would give the user greater flexibility in how they use the system.

6.3.1 Paper interface

Based on our findings, we used underlining, strikethrough and line down side as the edit gestures, and included speaker labelling and confidence shading. We retained the timestamps, but reduced their frequency to one per paragraph, rather than on every line. We added an inactive margin to allow users to write freehand notes without the risk of accidentally editing the audio. We did this by drawing a rectangular box on the right side to indicate where the user could safely write. We set the width of the margin to be approximately 25% of the width of the page, based on informal feedback from producers. Due to the wide variety of annotations that producers use, we did not attempt to capture structured notes.

We collaborated with Anoto to implement PaperClip using their LiveForms platform. As this platform did not allow us to combine underlining and strikethrough gestures with handwriting recognition, we could not include correction functionality. We did not include integrated playback control, as this would require a wireless link to the digital pen, which the platform did not support.

We used two active zones for each word — one on the word to detect a strikethrough, and one below the word to detect underlining. Drawing inside a zone would mark that word as either removed or selected, respectively. We lightly shaded the zone below each word to make the boundary visible. We drew a long thin rectangle between the transcript and the margin for capturing a line down the side. The final design is shown in Figures 6.3.1.1 and 6.3.1.2.

Editing was performed using an Anoto Live Pen 2 digital pen, which tracked and digitally recorded the gestures made on the transcript. When the pen was connected to a computer via a USB dock, the gestures were processed and translated by the Anoto system into edit commands. We integrated PaperClip with our screen interface (see Section 6.3.2) to handle audio import, printing transcripts, viewing/changing edits, viewing the margin notes and exporting the edits. A diagram of this integration is shown in Figure 6.3.1.3.

We supported two export formats — audio as a WAVE file, or an edit decision list (EDL) for the SADiE or StarTrack DAWs. PaperClip also created a PDF document of the transcript that showed the user’s annotations, which could be viewed through the screen interface.

6.3.2 Screen interface

For the screen interface, we updated the system we developed in Chapter 5 to implement some of the changes suggested by our findings. The original design can be seen in Figure 5.2.5.1, and the updated design is shown in Figure 6.3.2.1. The original design used a drag-and-drop system for creating clips from selected text. We replaced this with underlining and strikethrough gestures to provide better support for large selections, and to align with the design of PaperClip.

We added a double-speed playback feature to allow faster than real-time listening, and a “save-as” feature to allow multiple edits of the same material. We implemented this by using collapsible sidebars to separate the original “media” on the left, from the modified “edits” on the right (see Figure 6.3.2.2). We also included speaker diarization using a label and line down the side of each paragraph, coloured by gender, and included confidence shading using a dotted red underline to match the style of word processors.

The screen interface included integrated playback, which allowed the user to listen to and navigate the audio while they edit. The current playback position was shown in the text and the user could jump to a word by double-clicking it on the transcript. Any edits made to the transcript were reflected in the audio. The user could also correct any mistakes in the transcript by editing the text as they would in a word processor.

6.4.1 Protocol

The protocol for our study had three stages.

Stage 1: Training Firstly, the participant was briefed on the study and asked to sign a consent form. The participant used a test recording to perform a scripted series of tasks that used all of the features of each interface. This allowed them to use and experience all of the features for themselves. The participant was then given an opportunity to ask questions and become familiar with each interface until they felt comfortable with using them.

Stage 2: Observation The participant completed three editing tasks, each under one of the three conditions (C1, C2 or C3). The order of conditions was balanced to avoid carryover effects. We designed the experiment so that the editing tasks overlapped with the work the producers already needed to do. This ensured that the tasks were genuine and part of a real production. The participant provided three recent speech recordings that they needed to edit so that the content was fresh in their mind. We needed to use different recordings for each condition, but we asked the participant to choose recordings from the same programme to ensure they were as similar as possible. In Chapter 5, we found that there was no benefit in using transcripts for short recordings, so each recording was at least 20 mins in length.

The investigator observed the task, made written notes about their behaviour, and logged the duration of each audio file and the time taken to edit it, excluding any interruptions. Items of interest at this stage included editing workflow, tools used, data generated, usability challenges and problems, navigation and edit actions, time taken to complete tasks, unexpected reactions and unanticipated usage. During any “down-time”, we conducted ad hoc, in situ interviews to clarify the process and any decisions that were made. The observation took place at the participant’s normal work environment, which in all cases was a desk in an open plan office. We considered recording the task using a video camera, but due to the open-plan nature of the offices, there were insurmountable issues with privacy and information security.

After each task, the participant filled out a questionnaire to measure the usefulness and usability of the interface, using the Perceived Usefulness scale (Davis 1989) and the Software Usability Scale (SUS) (Brooke 1996), respectively. After completing all three tasks, the participant was asked to select which system they would prefer to continue using.

Stage 3: Interview The investigator conducted a semi-structured interview using the following questions. The order of questions 2–4 was adjusted to match the order in which the conditions were presented to the participant. An audio recording was made of the interview for later analysis.

1. Can you please describe your existing process for editing audio?

2. What did you like or dislike about the pen-based system?

3. What did you like or dislike about the screen-based system?

4. What did you like or dislike about using normal paper?

5. Overall, which of these systems would you most prefer to continue using, and why?

6.4.2 Analysis

We transcribed the interview recordings and corrected the words manually using the screen interface described in Section 6.3.2. Using thematic analysis (Braun and Clarke 2006), the investigator then openly coded the transcripts and observation notes using RQDA (Huang 2016), which produced 229 initial codes. The investigator then used FreeMind mind-mapping software to group the codes into categories, and the categories into themes.

The time taken to edit an audio file depends upon its length. As recommended by Dewey and Wakefield (2014), we divided the edit speed of each task by the audio file duration to calculate the “normalised task completion time”. Following the procedures described in Davis (1989; Brooke 1996), we converted the questionnaire data measuring the usefulness and usability into individual scores between 0 and 100. We used within-subjects one-way ANOVA (Rouanet and Lépine 1970) to test for differences between the systems in the relative edit time, perceived usefulness and usability (SUS) metrics. For the system preference data, we simply report the count of the preferences for each system.

6.5.1 Metrics

When asked which system they would prefer to continue using, four of the eight participants chose PaperClip, two (P3 and P6) chose the screen interface and two (P1 and P4) chose the normal paper transcript. Although it did not include any semantic editing functionality, P1 and P4 said they preferred the normal paper transcript as it allowed them to use their existing workflow and tools, which they found easiest and most comfortable. This demonstrates that ASR transcripts themselves are beneficial to radio production.

Figure 6.5.1.1 shows the mean average scores of the usefulness and usability of the three systems. A one-way within-subjects ANOVA showed that there was no statistically significant difference between the systems for usefulness [\(F(2,14)=0.788, p>0.05\)], nor usability [\(F(2,14)=1.068, p>0.05\)].

The SUS metric produces a numeric score between 0 and 100, which can be used to directly compare the usability of different systems. The reported SUS scores of systems from other studies can be used to convert our scores into a percentile figure that shows how they compare to other interfaces in general. Sauro and Lewis (2016, p. 204) proposed translating this percentile score into a grade between F and A+ to describe the result in human terms. The grades for our normal paper, screen and pen interfaces were A, B and B–, respectively, which shows that all of the tools appear to perform well overall. However, as this technique does not take into account the purpose of the interfaces, it cannot tell us how usable our interfaces are compared to other semantic speech editors.

For each task, we divided the edit time by the audio duration to calculate the relative edit time. The screen and normal paper interfaces had the same mean relative edit time (\(\times 0.99\) real-time), but PaperClip was 16% faster (\(\times 0.83\) real-time). This was surprising, as we expected the screen interface to be faster than both the pen and normal paper due to its integrated playback feature. However, a one-way within-subjects ANOVA did not find any statistically significant difference [\(F(2,14) = 0.931, p > 0.05\)].

The metrics results show that although half of participants preferred the PaperClip interface and it had the fastest relative edit time, it was rated least useful and least usable. To try to better understand these ratings, we now turn to the interview and observational data.

6.5.2 Editing

6.5.2.5 Technique

P1, P2, P6 and P8 reported that editing was an iterative process. P2 said this was because they are not sure what they need in the early stages, so they select too much then reduce it later. P8 said that what they select, or how much they select, depends on what was said in other interviews, and P1 said they often have to go back to re-edit clips in a different way.

P1, P6 and P8 reported that all three systems we tested were only suitable for the first iteration, known as a “rough edit”, because they were missing two features — re-ordering and labelling. Re-ordering is used to to see and hear how different clips from separate interviews would work together, and labelling is used to help the producer navigate, organise and structure their content.

P5 used annotations to segment and label the transcript (see Figure 6.5.2.5.1), which helped to structure the material.

“I was just labelling by summarising a paragraph in about two or three words — just who is speaking and the substance of it — or maybe just putting a cue to say that was a question.” (P5)

By capturing this information in a structured way, it could be exported as part of the EDL to guide the producer in later stages. P3 suggested that it might be possible to automatically generate labels using the text of a selected clip.

“If it was to dump those separate clips in your [system] and name them according to the text, then that would save twenty minutes suddenly in a single go.” (P3)

6.5.3 Transcript

6.5.4 Listening

6.6.1 Paper vs screen

We found that there were no overall preferences between the paper and screen interfaces, but that there were advantages and disadvantages of both in different uses and circumstances. Influential factors included the complexity of the edit, familiarity with the audio, accuracy of the transcript, user location and collaboration. Broadly speaking, we found that our pen interface was better for making simple edits involving quick decisions, using familiar content with a high-quality transcript, for producers working away from their desk, or with others in the same room. By contrast, we found that our screen interface was better suited to more complicated editing involving complex decisions, using less familiar content with a lower accuracy transcript, for producers working at their desk, or with other people remotely.

Participants reported that using paper rather than a screen made it it easier to read transcripts, remember information, think widely and orientate themselves. This aligns with previous research that has compared reading on paper to screens (O’Hara and Sellen 1997; And, Kurniawan, and Zaphiris 2001; Mangen, Walgermo, and Brønnick 2013; Singer and Alexander 2017). These results show that the benefits of paper-based working can translate to radio production using ASR transcripts.

On average, editing using the pen interface was 16% faster than using the screen, but this result was not statistically significant. However, three of the participants reported that they could edit faster and more easily using the pen interface compared to the screen. This may have been partially due to the faster reading speed of paper and the ability to underline while reading or listening. Some participants also suggested that the lack of integrated playback and correction features may have caused them to focus more on the task at hand, rather than being distracted by correction or navigation.

The screen interface included integrated playback and correction, which made it a more powerful tool. P7 reported that this made it suitable for more challenging editing tasks or exploring less familiar content, where the producer needs to be able to listen and navigate efficiently. The integrated listening also made it better for working with less familiar or lower accuracy transcripts as participants could tolerate the errors by using their memory of what was said, by listening or by correcting the errors. The screen makes it easier to listen and correct the errors, so is a better choice for editing less familiar content or lower quality transcript.

Three participants reported that both interfaces lacked features for labelling or re-ordering material, which meant that they were only currently suitable for creating a rough edit. This limits the usefulness of semantic editing in the later stages of production. For the pen interface, handwriting recognition could be used to label a specified region, or the nearest selected content. Two participants requested that the labels should be included in the exported EDL, so that they integrate with their existing tools.

Half of the participants reported that they like to work away from their desk or office as it helps them to focus. Pen and paper is naturally very portable and can be used almost anywhere. As it is does not use a screen, it is smaller, lighter, easier on the eyes and has a longer battery life. Several participants reported that this makes it more suitable for travel, and would allow them to work in more comfortable places. However, the requirement to print the transcripts makes it unsuitable for working “on the road”, where new material is recorded outside the office. By using a laptop or tablet device, the screen interface is also portable. The screen has the advantage of integrated playback and correction, and could be used on the road as it doesn’t need a printer.

Producers do not work alone and need to collaborate with others during production. The physical nature of paper made the digital pen interface suitable for working with others in the same room, as it allowed them to spatially arrange the pages and refer to the transcript by pointing. However, it is not possible for the pen interface to be used remotely. The digital nature of the screen interface makes it easy to work with remote collaborators. There is also potential to extend the screen interface to use operational transformation (Sun et al. 2004), which would allow multiple users to edit the same content simultaneously.

Our choice of technology for implementing the pen interface introduced some constraints that affected our design. Users edited the content by underlining or striking text within a set of rectangular boxes. These gestures were interpreted literally, which created a potential source of errors, and forced the participants to draw carefully. This design also prevented us from including correction and undo features. The batch-mode operation of the digital pen also prevented us from including integrated playback. By overcoming the constraints of our implementation, a pen interface with integrated listening, correction and undo could allow users to combine the benefits of working with paper with the full feature-set of the screen interface. However, there is also a risk that adding more features to the pen interface could introduce distractions, as we saw with the screen interface.

Electronic paper may provide a technical solution that could bypass these constraints. At the time we developed our pen interface, there were no e-paper devices that supported digital ink interaction, but these will become available in the near future. For example, the reMarkable tablethttps://remarkable.com/

is an e-paper device that includes a digital ink interface. Although e-paper displays do not seem to currently perform as well as paper for reading speed, comprehension or eye fatigue (Jeong 2012; Daniel and Woody 2013), they are likely to improve over time and may provide a good middle-ground between paper and screen interfaces.

6.6.2 ASR transcripts

The accuracy of a transcript has a direct effect on the performance and usage of semantic editing tools. We identified five different areas that were affected by transcript accuracy: correction, reading speed, reliance on listening, transcript longevity and edit granularity.

Three participants reported that errors in the transcripts slowed down their reading speed, with some errors being more distracting than others. Clearly, the more errors that occur, the more likely it is that corrections will be needed. However, the majority of participants were only interested in correcting errors that were particularly distracting. None of the participants needed or wanted to fully correct the transcripts, as this is only required if the transcript needs to be published. Although publication of transcripts is not currently part of the production workflow, doing so would make the programmes more easily discoverable and searchable.

The ASR system we used often mistranscribed unknown words into a variety of different words, which prevented the use of search-and-replace. This usually occurred with words specific to the programme, such as names of contributors or locations. To avoid this, producers could add unknown words to the dictionary of the ASR system prior to transcription. By providing additional contextual information, such as the programme topic or number of speakers, the ASR system could improve the transcript accuracy by using this to better calculate the likelihood of certain words occurring or by limiting the number of unique speaker segments.

The participants in our study using listening and memory to tolerate errors in the transcript. Remembering what was originally said increased the readability of the transcript and reduced the need to replay the audio. One participant reported that transcripts are often retained to help producers search through previously recorded material. However, as the producer’s memory fades, the errors in the transcript of previously recorded material become more of a problem, and the usefulness of the transcript deteriorates over time.

One participant reported that they selected more material than they needed due to the number of errors in the transcript. This suggests that the accuracy of the transcript may affect edit granularity. Selecting too much material creates more work for the producer at a later stage, as they have to edit their programme down to a specific time.

Two participants showed an interest in using semantic editing to remove “umm”s and breaths. The ASR systems we used were designed to ignore these sort of noises, rather than transcribe them. When they did appear, they were transcribed into a variety of words, which made it difficult to find and remove them. By explicitly including “umm”s and breaths in the ASR training data, these noises could be highlighted in the transcript, which could give users the option to remove them automatically.

6.6.3 Listening

We found in Chapter 5 that listening was an important part of the editing process. Through our study, we learned more about how and why the participants listened to the audio for semantic speech editing. Participants gave three main reasons for listening — processing information, judging sound quality and identifying non-speech sounds.

Listening allowed participants to hear what the transcript could not tell them, such as identifying mistakes or omissions in the transcript, finding where there are environmental sounds and working out whether the quality of the speech is sufficiently good for inclusion in their programme. Two participants said that simultaneously reading and listening made it easier to process the information in the speech, making the edit process more efficient. This falls in line with previous findings that providing transcripts allows users to process voicemail messages more efficiently (Whittaker et al. 2002), and improves the comprehension of time-compressed audio (Vemuri et al. 2004).

Half of the participants reported that they sometimes edit audio without a transcript, as they can remember most of what was said, and when it was said, for audio recordings less than 15–25 minutes long. For some programmes that focus on the auditory experience, editorial decisions will be led by the quality of the sound rather than what was said. In these cases, the cost and overhead of generating a transcript may not be worthwhile, however, it is unclear exactly where this threshold lies.

Two participants reported that they like to re-listen to their recordings in full to refresh their memory and start making decisions on what to use for their programme. Another participant suggested that the current process of manual transcription gave them an opportunity to re-listen. Re-listening in full adds overhead to the editing process, but some participants considered it to be a worthwhile process. The introduction of ASR transcription removes this opportunity. There is a risk that semantic editing may cause producers to base their decisions on transcripts rather than audio, which may affect the quality of the programme.

Listening is an important part of editing, but it is a time-consuming process. This could be reduced by using time compression techniques to increase the playback speed, as discussed in Section 2.5.1, or using sound event detection to identify and label regions of environmental noise (Duan et al. 2014; Kroos and Plumbley 2017). Including this information in the transcript could help producers identify sounds they do or do not want. Producers may also benefit from tools that help them see which edits would work or not. For example, a graphical representation of the intonation of speech may help producers identify whether editing two pieces of speech together would sound acceptable.

7.1.1 Semantic audio visualization

Our initial investigation looked at using pseudocolour to visualize a semantic audio feature to support audio editing using waveforms. We measured the user performance for a simple editing task using our semantic audio visualization, compared to a normal waveform. The results showed that when using the semantic audio waveform, the participants completed the task faster, with less effort and with greater accuracy than the normal waveform. This suggests that there is value in the pseudocolour approach to semantic audio visualization taken by Rice (2005), Akkermans et al. (2011) and J. Loviscach (2011a), which had previously been untested.

Our experiment only focused on a single task of segmenting music from speech so there are many opportunities for applications beyond the task we chose. Audio visualizations can either be designed for specific tasks, or for general use to cover a range of tasks. Visualizations that focus on an individual task may produce better results, but are only suitable for that task. Designing a general audio visualization is much more challenging, but has the potential to create a greater impact as it could be used for a variety of applications.

The semantic audio visualization we designed and tested used a rudimentary semantic audio feature rather than the state-of-the-art. We did not attempt to create the best possible visualization as we wanted there to be an element of human judgement in the measured task. There is potential to make better visualizations by including more and better semantic audio features, and using more advanced methods of visualization. For example, the false colour approach taken by Tzanetakis and Cook (2000) and Mason, Evans, and Sheikh (2007) allowed multiple features to be displayed simultaneously in a human-readable way.

Although we found that users required less effort to complete the task with a normal waveform than without, they did not complete the task significantly faster nor more accurately. We found this surprising as waveforms are widely used in audio editing software, so it was expected that waveforms would improve the performance of users in completing audio editing tasks. This poor performance is concerning as the widespread use of audio waveforms means that this affects a large community. However, this finding highlights an opportunity to increase the efficiency of audio production software by making improvements to the waveform visualization.

7.1.2 Semantic speech editing

We conducted two user studies in which semantic speech editing was successfully used by professional radio producers to create real radio programmes that were subsequently broadcast. Our results support previous work in finding that semantic speech interfaces help users navigate and edit speech (Whittaker et al. 2002), and that semantic editing is faster than, and preferable to, using audio waveforms (Whittaker and Amento 2004; Sivaraman, Yoon, and Mitros 2016). Our research went further by conducting user evaluations in a natural working environment, and directly comparing semantic speech editing to the existing editing workflow. This allowed us to gain insights into its limitations in the context of radio production.

Our semantic speech editor was similar to the system described by Rubin et al. (2013). However, rather than using verbatim transcripts, which are slow and expensive to produce, we used automatic speech recognition (ASR). The high speed and low cost of ASR make it better suited for use in broadcast environments, but the erroneous transcripts it produces affect the usability of semantic editing. Crucially, we discovered that the quality of modern ASR systems was sufficient to allow for semantic speech editing in radio production. This aligns with similar findings for the semantic editing of voice messages (Whittaker and Amento 2004; Sivaraman, Yoon, and Mitros 2016). Our studies also revealed that the accuracy of the transcript affected the need for correction, reading speed, reliance on listening, longevity of the transcript and the edit granularity.

The producers we tested reported that they only found semantic editing useful for creating a “rough edit”, where large segments of audio are selected from the original material. This contradicts findings by Sivaraman, Yoon, and Mitros (2016) for the editing of voice messages. We found that the main reason for this limitation was the lack of annotation and re-ordering features, which made it difficult for producers to organise, structure and arrange their material in the later stages. However, this limitation was not an obstruction to the producers we tested, as our semantic speech editing tools integrated with several digital audio workstations (DAWs), which allowed the producers to seamlessly transition to their normal tools to complete their production.

We tested a variety of additional features to support semantic editing, including transcript correction and confidence shading. We found that many of the transcript errors encountered were specific to the programme content, such as names and topic-specific words. However, most producers were only interested in correcting errors that were particularly distracting. Most producers we tested found confidence shading to be useful overall, which supports Burke, Amento, and Isenhour (2006), but contradicts Suhm, Myers, and Waibel (2001) and Vemuri et al. (2004).

DAWs are powerful, feature-rich tools, but to novice users they can be intimidating to work with. Yoon et al. (2014; Sivaraman, Yoon, and Mitros 2016) found that semantic speech editing tools are easier to use and more accessible than waveform-based tools. Podcasting has already democratised the distribution of audio content, and semantic speech editing may have the potential to achieve something similar for audio production. This would allow more people to access audio production, which could hopefully lead to the production of more and better audio content.

Alternative methods of interacting with audio content may also give rise to new creative opportunities for more innovative programme making. For example, the comedy duo known as “Cassetteboy” (Perraudin 2014) patiently trawl through hours of video footage to re-order the words of famous speeches in amusing ways. This could be achieved much more easily using a semantic speech editor.

The efficiency savings from using semantic speech editing tools and ASR could provide cost and time savings over traditional editing and manual transcription. This should free up time and budget for more valuable production activities, which may lead to improvements in the quality of programmes and/or reduction in the cost of production.

7.1.3 Reading vs listening

Transcripts display the words that were spoken in a recording, but much of the content’s true meaning is hidden in the audio. Although transcripts show what was said, they do not reveal how it was said, which is crucial when producing radio programmes. As one producer pointed out, “radio is made with your ears”.

The only way to fully comprehend audio recordings is through listening. We found that radio producers used listening to process information, judge the quality of the sound and identify non-speech sounds. Some producers reported that reading and listening at the same time enabled them to comprehend information more easily, supporting the findings from Vemuri et al. (2004). Producers listened to identify any low quality audio, such as “umm”s and “err”s, and long or heavy breaths. These are visible in verbatim transcripts, as used by Berthouzoz, Li, and Agrawala (2012) and Rubin et al. (2013), but are not included in ASR transcripts. Producers also listened to identify any high quality moments that may not be identifiable using the transcript, and non-speech sounds that they might want to include or exclude in their programme.

The existing radio production workflow involves producers “logging” recordings by listening to the audio and typing rough quotes and labels. This process is time-consuming and tedious, but it helps the producer to review and organise their content. ASR transcription replaces this logging process, which saves producers time and effort. However, it also means that producers don’t have to listen to the audio before editing, which may prevent some producers from listening to the material before making editorial decisions. There is a risk that over-reliance on transcripts may result in missing out on good quality content or failing to avoid poor quality content, which would affect the overall programme quality.

Providing an easy way to listen to the audio underlying the transcript is important. Our screen-based semantic speech editor included integrated playback, which allowed users to navigate the audio using the text. Time compression, such as described in Section 2.6, would also allow producers to listen faster, and producers valued this feature when we added it to our screen-based editor. Reading the transcript while listening also increases the speed at which time-compressed audio can be comprehended (Vemuri et al. 2004).

However, we found that in some situations, transcripts are not necessary or useful. With recordings shorter than 15-25 mins, producers reported that they can remember what was said, and when it was said. Some programmes have a greater focus on the sound design, which limits the value of a transcript.

We saw that transcripts of radio programmes are not normally published at the BBC, as they are not written during the programme’s production. ASR and transcript correction tools could make it possible for producers to create a verbatim transcript as part of their production process. Publishing these could allow audio content to be more easily searchable and discoverable through Internet search engines, for example. Word-level timings could also be used to link directly to segments of audio. For example, NPR’s online clipping tool “Shortcut” (Friedhoff 2016) allows radio listeners to create a short clip from a radio programme and share it with their friends.

7.1.4 Paper vs screen

Our ethnographic case studies in Chapter 3 identified that many radio producers work on paper. Our user study in Chapter 5 also found that many producers want to be able to work away from their screens. This led us to develop a novel semantic speech editing system that used digital pens to allow producers to edit speech recordings using a paper transcript. We conducted a qualitative user study with professional radio producers that compared our paper interface to a screen-based semantic speech editor. The results of our study provide insights into the relative benefits of editing speech on paper compared to screens. We found that both the paper and screen interfaces worked well, but that each had advantages and disadvantages in different contexts.

Overall, we found that the paper interface was better suited to quick and simple edits where listening is not as critical, such as with high accuracy transcripts, or very recent recordings. Our screen-based semantic speech editor included integrated playback and correction features, which made it better for more complex editing with less familiar audio and less accurate transcripts.

Producers reported that paper transcripts were easier to read and remember, and made it easier for them to think widely and orientate themselves. They reported that our digital pen interface was simple, intuitive, precise and allowed edit decisions to be made quickly and easily. However, producers had concerns over the cost of an additional device which would likely have to be shared amongst a team, and could easily be lost.

The physical nature of the digital pen and paper made it better suited to travel and working away from the desk. This gives producers greater flexibility to work in more comfortable locations and while commuting. However the digital pen interface uses considerable amounts of paper, involves carrying the pen around, and requires access to a colour laser printer, which would make it difficult to work “on the road”. The screen interface could be used on a laptop or tablet, which have the added benefit of integrated playback. However, screens are heavier, bulkier and have a much shorter battery life than digital pens.

Radio is usually produced by a team, so tools that facilitate collaboration are valuable in such an environment. We found that the physical nature and pagination of paper made it better suited to face-to-face collaboration than the screen. However, the screen interface is capable of remote collaboration, and could be used over the Internet for real-time collaborative speech editing.

The limitations of the digital pen system we used prevented us from including features for integrated playback, correction and undo. The lack of integrated playback forced producers to replay and navigate audio using a separate device, which made it more challenging to identify errors in the transcript. This was a bigger issue with low quality transcripts, which could also not be corrected using the digital pen. The integrated playback of the screen interface made it easier for the participants to find and fix mistakes in the transcript. This also made it easier to edit content that required more listening, such as old or unfamiliar recordings.

A.1.1 System overview

The structure and data flow of Dialogger is shown in Figure A.1.1.1. We divided our system into a presentation layer (front end) and a data management layer (back end). The front end used HTML, CSS and Javascript so that the system could be accessed through a web browser. We used Semantic UI as a user interface framework, Backbone as a model-view-controller framework and Dropzone to handle file uploads.

For the back end, we used Node.js and Express as our framework, and MongoDB for our database. We authenticated users using Passport.js, logged errors using Bunyan and used Mimovie to extract metadata from uploaded media files.

A.1.2 Word processing

We included word processing functionality in Dialogger to allow users to navigate the speech, correct mistakes in the transcript and mark up edit decisions. We used the CKEditorhttps://ckeditor.com/

library as the basis for the word processing functionality in our interface.

Timestamps In addition to the requirements of a normal word processor, our system had a unique requirement for each word to include a hidden timestamp, and for that timestamp to be retained throughout the editing process. We achieved this by using HTML formatting to add a tag to each word, and including the start and end timestamps as attributes of the tag. This linked the words to timestamps, but certain edit actions created problems, as explained below.

Split words Traditional word processors allow users to select text with the granularity of individual characters. This caused an issue where words could be split by cutting, moving or replacing only part of a word. To avoid this problem, we restricted editing to word level-granularity by automatically moving the user’s selections to the start and end of words.

Joined words Occasionally, ASR systems will transcribe a single word as two or more words. When the user corrects this by selecting both words and typing a replacement, then both words are replaced and their timestamps are lost. To avoid this, we added logic that detected this behaviour and created a new word with the start time of the first selected word and the end time of the last selected word.

Re-ordering Word processing gives users the freedom to move and edit words as they wish. However, when text is moved around it becomes very difficult for the user to keep track of the original location and order of the words. This is a problem with audio editing as some edits sound unacceptable, so users will often want to adjust the edit. Our solution was to retain the sequence of the words so that they stay in their original positions. This prevented re-ordering of the words, however, this could later be adjusted using a DAW. We implemented this restriction by disabling cut, copy, paste and drag-and-drop.

A.1.3 Media integration

We used timestamps to link each word in the transcript to a segment of a media file. To integrate the transcript edits with the media, we built systems to generate an edit decisions list (EDL) based on the user’s annotations, to instantly preview those edits in the browser, and to allow the user to download a copy of the final edit.

EDL generation In addition to including the start and end timestamp of each word in a HTML tag, we included the timestamp of the next word in the transcript. To detect edit points, we simply looked for any difference in the predicted and actual timestamp of the next word. We could then generate an EDL by filtering words based on their annotation, then looking for edit points. However, this approach relies on the words being in their original sequence. Our EDL format is simply an array of timestamps that denote the in-points and out-points of each edit.

Preview We wanted users to be able to preview their edits to quickly hear whether the edits sound acceptable or not. When a user uploads their media, we transcode a low-bitrate “preview copy” of the file, which we then use for live playback in the browser. To control the playback, we used HTML5 Video Compositorhttps://github.com/bbc/html5-video-compositor

— a media composition engine that can play edit decision lists in the browser. After the user makes an edit, we generate an EDL, convert it to the correct format, and pass it to the compositor, which dynamically adjusts the playback of the preview copy. To avoid the browser having to download the entire file before playback, we used byte serving to dynamically send only the portion of the preview copy that is needed.

Export Once the user has completed their production, they can export their edit as either a rendered media file, or an EDL file for a DAW. The EDL of the user’s edits is sent from the browser to the back end. Media is rendered and transcoded using the MLT Multimedia Frameworkhttps://www.mltframework.org/

that supports sample-accurate editing of audio and video files. To maximise the quality of the exported media, we use the original assets when rendering the file. To generate the EDL files for DAWs, we simply convert our internal EDL format into (AES31–3 2008) or another proprietary format.

A.2.1 Design

We built our plugin system on top of the well-known Vamp audio analysis plugin systemhttp://vamp-plugins.org/

, to make use of the current large collection of audio feature extraction plugins. An outline of the design of Vampeyer is shown in Figure A.2.1.1. We designed our plugin system to receive the output of one or more Vamp plugins as the input, and send a bitmap image as the output.

A Vampeyer plugin defines which Vamp plugin(s) it requires, including the block/step size and parameters. At least one Vamp plugin is required, but there is no restriction of the number of Vamp plugins that can be used. Both frameworks are written in C++ which allows for efficient processing. This is important when processing large collections, or generating visualizations “on-the-fly”. Vampeyer plugins are compiled into shared libraries, so they can be distributed without users having to re-compile locally and be integrated into third-party software.

A.2.2 Implementation

To illustrate the design and operation of Vampeyer plugins, we describe the data structures and functions that are used to convert the Vamp plugin output into an image. The following five data structures define how data should be sent and returned from the plugin’s functions.

• VampParameter is a name/value pair used to store a parameter

• VampParameterList is a vector of VampParameters

• VampPlugin stores the name of a Vamp plugin along with a
  VampParameterList and the preferred block and step sizes

• VampOutput stores a VampPlugin and output name

• VampOutputList is a vector of VampOutputs

Vampeyer plugins have two primary functions to describe the required input audio features, and then process those features into a bitmap image.

• getVampPlugins returns a VampOutputList variable that contains a list of Vamp plugin outputs which must be provided as input

• ARGB takes the Vamp plugin output data as a Vamp::FeatureSet variable, plus the sample rate of the audio and the desired width and height of the bitmap. It returns a bitmap image formatted in 32-bit ARGB format (alpha, red, green, blue).

Vampeyer plugins need to be run by a host program which reads the audio data, processes it using the Vamp plugins, generates the image using the Vampeyer plugin and then writes the image data. We have published such a program as a command-line tool that can either display the image in a window or write it to disk as a PNG file. This makes it useful for both prototyping and for back-end processing on a server, for example.

We have also published a number of example plugins to demonstrate the capabilities of Vampeyer, and to act as useful pieces of software in their own right. These include plugins for an audio waveform, a waveform colourised by low energy ratio (see Section 4.1.4), a waveform colourised by spectral centroid (similar to Akkermans et al. (2011)) and an MFCC visualization.

A.3.1 Design

We came across two challenges when designing BeatMap. Firstly, bitmap images of long audio files can be very large, which makes it impractical for use over the web. Secondly, audio visualizations need to be able to zoom horizontally so that users can view the visualization at different scales and levels of detail.

Tiling To bypass the problem of large image files, we designed BeatMap to use tiled bitmap images. This breaks the audio visualization into many individual chunks, so that only the portion of the image currently being viewed needs to be downloaded. Initial testing found that when playing, tiles were not being loaded in time, so we added a pre-loading function that predicts which tiles will be displayed next and downloads them in advance.

Zoom We added a horizontal zoom function to allow audio visualizations to be viewed at different scales. We achieved this by using multiple images, one for each scale, and allowing the user to switch between them. In addition to changing zoom level within the main display, we added support for a linked secondary display that can view the same audio visualization at a different zoom level (see Figure A.3.1). This enables users to simultaneously view the entire audio recording at once, while also being able to see the visualization of the current playback position in greater detail.