VISR Production Suite User documentationThis makes it the perfect audio tool for VR. The plugin has...

VISR Production Suite Userdocumentation

Release 0.12.0

The S3A project team

Oct 16, 2019

CONTENTS

1 Introduction 31.1 Object-based audio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.1.1 What is it? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.1.2 How does it work? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.1.3 Channel-based vs object-based . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

1.2 The VISR Production Suite . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51.2.1 What you can do with the VISR Production Suite . . . . . . . . . . . . . . . . . . . . . 51.2.2 The included Plugins . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

1.3 People . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2 Getting Started 112.1 Download . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112.2 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2.2.1 Windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112.2.2 Mac OS X . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142.2.3 Python configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3 Tutorial 1: Using VISR Production Suite 193.1 REAPER . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

3.1.1 OBJECT 1, OBJECT 2 Tracks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203.1.2 SCENE MASTER Track . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223.1.3 LOUDSPEAKER RENDERER Track . . . . . . . . . . . . . . . . . . . . . . . . . . . 223.1.4 BINAURAL RENDERER Track . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

3.2 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

4 Tutorial 2: Create your first object-based DAW session 274.1 Create a session from scratch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

4.1.1 REAPER . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284.2 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

5 Tutorial 3: Use a head position tracker with the Binaural Renderer 315.1 OSC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325.2 Internal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

6 Tutorial 4: Create your first Reverb-Object-Based DAW session 336.1 REAPER . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

6.1.1 Reverb_OBJECT 1 Track . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 336.1.2 OBJECT 2, OBJECT 3 Tracks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356.1.3 SCENE MASTER Track . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 356.1.4 LOUDSPEAKER RENDERER Track . . . . . . . . . . . . . . . . . . . . . . . . . . . 356.1.5 Optional Binaural Renderer Configuration . . . . . . . . . . . . . . . . . . . . . . . . . 37

i

VISR Production Suite User documentation, Release 0.12.0

This is a guide that will help you through the necessary steps to download, install and properly use the plugins inthe VISR Production Suite 0.12.0.

CONTENTS 1


2 CONTENTS

CHAPTER

ONE

INTRODUCTION

1.1 Object-based audio

1.1.1 What is it?

A novel paradigm and technology to produce and distribute audio, which allows an audio content creator to targetdifferent playback systems and formats using a single production. Therefore you can create scalable productions,being also future-proof by being able to easily support future formats.

Fig. 1: object-based production

1.1.2 How does it work?

The fundamental elements of an object-based production are:

• Objects: audio objects are audio signals with associated metadata. The metadata of all the objects togetherform the

• Scene: a way to represent the metadata of the set of objects, which is neutral in regard to the output audioformat and system. Those information are sent to the

• Renderer: a component that uses the scene metadata to mix the signals of the objects in order to generatethe output audio signals for different types of devices (Loudspeakers, Headphones, . . . ) and layouts ( stereo,binaural, 5.1, 7.1, 22.1, . . . ).

An important feature of object based is that having an all-purpose scene representation means that the renderingpart can be in the DAW as usual, or postponed in any further step of the audio chain, including the end user device.

3


The latter scenario paves the way to new possibilities in the interaction and personalisation of the audio content,e.g. allowing the end user to adjust the singer’s voice only, while listening to a song on a smartphone.

1.1.3 Channel-based vs object-based

This is a quick comparison between the channel-based approach and the object-based one:

channel-based (Fig.1) object-based (Fig.2)Production for one format only Scalable and future proof productionsNo adaptation possible after mix New end user experiences: interaction and personalisation

Fig. 2: Fig.1 typical channel-based production

Fig. 3: Fig.2 object-based production

See the following webpages for more info on object-based audio:

https://lab.irt.de/demos/object-based-audio/

https://www.bbc.co.uk/rd/object-based-media

4 Chapter 1. Introduction

https://lab.irt.de/demos/object-based-audio/

https://www.bbc.co.uk/rd/object-based-media


1.2 The VISR Production Suite

The VISR Production Suite is the first open-source set of DAW plugins for producing and reproducing object-based audio.

The fundamental elements of an object-based production are:




We provide those elements in the form of:

• an Object Editor plugin on each track to transform your tracks into audio objects

• a so called Scene Master plugin to manipulate the whole object-based audio scene

• and integrated renderer plugins to reproduce the scene on various loudspeaker layouts, sound bars, orheadphones.

The production suite is based on the JUCE framework, which is a famous tool for easily generate DAW plug-insin different formats, and on the VISR, a modular open-source framework for audio processing.

1.2.1 What you can do with the VISR Production Suite

• Create an object-based session:

• Spatialise and control your audio objects

• Easy Binaural content creation with different sets of HRIR and BRIRs in SOFA format

• Head tracking and head orientation compensation capabilities, suitable for VR

• Create loudspeaker layout agnostic spatial audio content and render it for different loudspeaker systemswith just one click

• Exploit the features of JUCE and the VISR frameworks

1.2. The VISR Production Suite 5

https://juce.com

http://www.s3a-spatialaudio.org/visr


1.2.2 The included Plugins

Supported Formats

The DAW plugins of the VISR Production suite are provided in the following formats:

• VST®1 (Virtual Studio Technology) version 3

Supported DAWs

The plugins are currently fully supported by the following DAWs:

• Reaper ( see tutorials )

Although the plugins are already available in other formats (AU, AAX), it seems that at the time of writing manyDAWs such as Cubase, Nuendo, Logic Pro, Ableton Live, Pro Tools are not capable of routing many objects in asingle rendering plugin as separate input channels. Therefore for the time being Reaper seems the only candidateand AU and AAX are therefore not shipped with the installer.

Object Editor

The Object Editor is the plugin that transforms your track into an audio object, by generating metadataassociated to the sound of the track.It contains 3D panning parameters as automatable metadata which are sent to the Scene Master while the audiosignal is routed to one of the available renderers.In this way you can have complete control on the spatialisation of your tracks and you can create yourobject-based scene at the same time.

Features:

• Transform your track into an audio object generating associated metadata

• Spatialise your track changing the object position in virtual space

• Use the sliders or drag the object around with your mouse/trackpad

• The GUI sphere representation allows for a full-resolution 3D movement on the surface of a sphere

1 VST is a trademark of Steinberg Media Technologies GmbH, registered in Europe and other countries.



Scene Master

The role of this plugin is to have a centralised place where you can see, control and edit the whole object-basedaudio scene.All the audio objects created with the usage of the Object Editor plugin are shown here in a table, and you cancontrol them by changing the associated metadata.The object-based audio scene is format agnostic and therefore the plugin keeps all the information on the soundobjects without defining a reproduction system, which can be chosen at a later stage using one of the renderingplugins.

Features:

• View the whole object-based audio scene

• Collect all the audio objects of your session in one place

• Edit and reorder all the single objects through the object table

• Control the output format agnostic audio scene and send it to the renderers

• Live sending of the objects using VISR object vector format

• Take scene-level decisions and perform scene-level changes from one single plugin

1.2. The VISR Production Suite 7


Loudspeaker Renderer

With this plugin you can render the object-based audio scene using different loudspeaker layouts which you canchoose from the dropdown menu, such as stereo, 5.1, 7.1 and so on.In this way, with just one click, you can playback the same audio production using many different audioreproduction formats and loudspeaker layouts.

Features:

• Render for different loudspeaker layouts using the same object-based scene description received by theScene Master plugin

• Dynamically switch between different loudspeaker layouts

• Visualise the Loudspeakers and the objects in the same 3D sphere view

• Take advantage of state-of-the-art VBAP rendering under the hood, provided by the VISR framework

Binaural Renderer



With the Binaural Renderer you will be able to easily create binaural content. It allows you to switch betweendifferent SOFA-compliant HRIR and BRIR datasets and to try different binaural rendering approaches.Moreover this plugin is capable of receiving head orientation information through OSC to exploit dynamicbinaural cues by applying head rotation compensation. This makes it the perfect audio tool for VR.The plugin has been made as a wrapper around the functionalities of the existing VISR BST (BinauralSynthesis Toolkit).

Features:

• Listen to your object-based scene with just a pair of headphones

• Explore different binaural rendering approaches in the same plugin

• Use different datasets of HRIR and BRIRs under the form of SOFA files

• Head tracking and head orientation compensation through usage of Razor AHRS 9DOF tracker

• Head tracking and head orientation compensation through OSC using external trackers

• Use the python package VISR BST (Binaural Synthesis Toolkit) inside a DAW Plugin

1.3 People

The VISR Production Suite has been made as part of the S3A project and it is based on the JUCE framework,which is a famous tool for easily generate DAW plug-ins in different formats, and on the VISR, a modular open-source framework for audio processing.

The people who were directly involved in the plugin development are:

• Giacomo Costantini - main developer ([email protected])

• Andreas Franck - VISR development and support ([email protected])

• Will Morris - Reverb Object development and support ([email protected])

Moreover, many of the S3A researchers were involved in the scientific research aspects behind the development ofsome of the plugins, and helped the developers providing concrete use cases and participating to the beta testing.

1.3. People 9

https://juce.com

http://www.s3a-spatialaudio.org/visr

mailto:[email protected]



CHAPTER

TWO

GETTING STARTED

2.1 Download

Installation packages are available for the following platforms:

Windows (x86_64) Recent versions (Windows 8 and Windows 10) 64 Bit only

Mac OS X Version 10.9 and above

Download the VISR Production Suite installer from the download page .

Note: One of the plugins in this Suite, the Binaural Renderer, requires Python to be installed. If you don’t needto use that plugin, choose the standard installer .

Otherwise, select the full (Python x.x) installer, being careful to select the one corresponding to the python x.xversion you have installed or you want to install. e.g. if you are on MacOS and have Python version 3.6, installMac OS X 10.9 or higher (64-bit) - full ( Python 3.6 ).

If you have python already on your machine, you can see the python version by opening a command line promptand typing the command:

python -V

2.2 Installation

Software Requirements:

• only for Binaural Renderer plugin, i.e. if you executed one of the full (Python x.x) installers: Python3(64 bit).We recommend Anaconda3 Python distribution .

2.2.1 Windows

The guided installer is provided as an .exe file, and is for 64-bit versions of Windows only.The figure shows the component selection dialog of the installer.

If required, install the “Microsoft Visual C++ Redistributable for Visual Studio 2017”, package, for examplefrom the Visual C++ downloads page.

11

http://www.s3a-spatialaudio.org/plugins

https://www.anaconda.com/download/

https://support.microsoft.com/en-us/help/2977003/the-latest-supported-visual-c-downloads


Fig. 1: Graphical Windows installer.

On Windows, it is necessary to add the directory containing the VISR libraries (DLLs) and the third-partylibraries shipped with the VISR installer to the PATH variable.

Warning: If you have a previous installation of VISR, you may already have some<VISR-directory>\3rd and <VISR-directory>\lib folders in PATH, this could cause a problemif used in conjunction with the <install-directory>\3rd in case there is some library mismatch.

To this end, open the environment variable editor (Settings -> System -> Advanced system settings ->Environment variables).

Append the value <install-directory>\3rd to the path variable, where install_directory is thedirectory specified during the installation. When using the default installation path, that directory isC:\Program Files\VISR-Production-Suite\3rd .Depending on your system permissions and whether your VISR Production Suite shall be used by all users of thecomputer, you can either set the PATH user variable or the PATH system variable.

Note: Any applications using the plugins of the VISR Production Suite (e.g. a DAW) must be closed andreopened before the changed paths take effect.

Note: Future versions of the installer might adjust the paths automatically. However, as pointed out in NSIS Path

12 Chapter 2. Getting Started

http://nsis.sourceforge.net/Path_Manipulation



Fig. 2: Environment variable editor on Windows 10.

2.2. Installation 13




manipulation, this needs an extremely cautious implementation to avoid potential damage to users’ systems.

If you need to use the Binaural Renderer plugin, i.e. if you executed one of the full (Python x.x) installers,you need to perform some additional python configurations, described in Python configuration. If you are using aStandard installation package, no other steps are required and you can start following the Tutorial 1.

2.2.2 Mac OS X

The guided installer is provided as a .pkg file.When double clicking on the installer, it may be that you get a pop-up window by MacOS operating system,telling you that the installer is unsafe:

in that case open “System Preferences” -> “Security & Privacy” and press the button “Open anyway”. Then press“Open” in the pop-up window.

If your machine does not have the command line developer tools, you will be prompted to download them (seefollowing Figure). Just accept that and it should automatically install those tools as part of the installation.

To access the component selection dialog, use the button “Customize” on the “Installation Type” screen

The figure shows the component selection dialog of the installer.





Fig. 3: “Installation type” screen of Mac OS X installer. Use “Customize” to get to the component selection.

Fig. 4: Component-based installer for Mac OS X.



Normally you do not need to change the installation components, and please do that only if you know what youare doing.Follow the installation procedure until you get a message of successful installation.

If you need to use the Binaural Renderer plugin, i.e. if you executed one of the full (Python x.x) installers, youneed to perform some additional python configurations, described in Python configuration. If you are using aStandard installation package, no other steps are required and you can start following the Tutorial 1.

2.2.3 Python configuration

Note: Follow this section of the guide only if you need to use the Binaural Renderer plugin, i.e. if you executedone of the full (Python x.x) installers

Two environment variables must be set to ensure the working of the VISR Python subsystem.

PYTHONPATH

This variable is used to add the directory containing the VISR python modules to the system path.The Binaural Renderer plugin uses the python package VISR BST (Binaural Synthesis Toolkit) whichis provided as part of the installation. In order for it to work, the Python interpreter must know where thispackage is on the current machine.To this end, the python/ subdirectory of the installation folder must be added to PYTHONPATH.

PYTHONHOME

This variable is needed to locate the files and libraries of the Python distribution. This is especiallyimportant if there are more than one distributions on the system, most often on Mac OS X.This variable has to be set to the root directory of the Python distribution, i.e., one level of hierarchy abovethe bin/ folder conatining the Python interpreter.Depending on the platform and the distribution, the correct value might be:

Windows with Anaconda C:\ProgramData\Anaconda3

Mac OS X with Anaconda $HOME/anaconda3/

To find that path, you can use the following command on a command line prompt:

python-config --prefix

or

python3-config --prefix

OPENBLAS_NUM_THREADS ( Optional )

It is advisable, in many cases, to set the value of this environment variable to 1. It controls how numpynumerical algebra functions are distributed to multiple CPU cores.numpy is used by the VISR Python integration as well as in many Python-based VISR componentsperforming mathematical or DSP operations.For the matrix/vector sizes typically encountered in our code, the overhead for distributing the work overmultiple cores typically exceeds the potential gains.Multithreading is disabled by setting the maximum number of cores (or threads) to 1:

OPENBLAS_NUM_THREADS = 1



How to set the variables

Depending on the operating system, these variables can be set as follows:

Windows Add the environment variables either as user or system variables, as described in Windows section. Thecorrect settings are (assuming the default installation directory and the Anaconda distribution):

PYTHONPATH=c:\Program Files\VISR-Production-Suite\pythonPYTHONHOME=c:\ProgramData\Anaconda3OPENBLAS_NUM_THREADS=1

Note that if there is already a PYTHONPATH variable, the recommended value should be appended, using asemicolon as a separator.

Mac OS X

In order to set the environment variables system-wide, without requiring that the applications in question isstarted from a shell, (e.g., a command-line terminal), we recommend a custom launchd property list file,as detailed, e.g., in this StackExchange thread. In this way you can add a LaunchAgent that sets thisvariable for you.

For convenience, the installers provide a pre-configured VISR-Production-Suite.plist file in theresources subdirectory of the installation directory (e.g., /Applications/VISR-Production-Suite/resources/VISR-Production-Suite.plist).By convention, these files should to be stored in /Library/LaunchAgents/ but that’s not mandatoryand in the following commands we assume you don’t do that.

Warning: if you used this same mechanism in the past for registering PYTHONPATH andPYTHONHOME (e.g for installing VISR ), it might be necessary to unload ( launchctlstop <nameOfOldPlist> and then launchctl unload <nameOfOldPlist> ) and thenremove the files beginning with VISR in /Library/LaunchAgents/ and in ~/Library/LaunchAgents, if any. Then start the Terminal application (restart it if it is already open) and ensurethat echo $PYTHONPATH gives you an empty result. If not, a restart of the machine and re-check issuggested.

To activate the settings, call:

launchctl load /Applications/VISR-Production-Suite/resources/VISR-Production-→˓Suite.plist

launchctl start /Applications/VISR-Production-Suite/resources/VISR-Production-→˓Suite.plist

To take effect, all applications using these settings (e.g. DAWs) must be quit and reopened.

These settings are preserved if the machine is restarted. To deactivate them, the property list file must beunloaded:

launchctl unload /Applications/VISR-Production-Suite/resources/VISR-Production-→˓Suite.plist

launchctl stop /Applications/VISR-Production-Suite/resources/VISR-Production-→˓Suite.plist

If you made changes to the settings, you have to perform the unload command followed by a load.


https://apple.stackexchange.com/questions/106355/setting-the-system-wide-path-environment-variable-in-mavericks


Now, to test that the path is set correctly, restart the Terminal application and check the PYTHONPATH bytyping:

echo $PYTHONPATH

The output should be /Applications/VISR-Production-Suite/python .

Additional required modules

By default anaconda3 does not come with the pyserial module installed, which is needed by the BinauralRender plugin. To do so open Terminal application and type:

conda install pyserial

Another module that might not be present in your python distribution , is h5py. To install it run the followingcommand:

conda install h5py

If your python distribution does not include numpy and scipy you have to install them as well. To sum up, youcan install all the needed packages in one line with the following command:

conda install numpy scipy h5py pyserial

or, if you do not have Anaconda distribution, you can achieve the same with pip or pip3:

pip install numpy scipy h5py pyserial

You can now start following the Tutorial 1 to know how to use the plugins.

Note: After the installation, to see the Binaural Renderer, normally a re-scan of the plugins is needed on theDAW you are using.


CHAPTER

THREE

TUTORIAL 1: USING VISR PRODUCTION SUITE

Note: Reading the content of Introduction is strongly suggested before starting this tutorial.

First of all: what is an object-based session?It’s a DAW session/project that contains all the fundamental elements of an object-based production, as describedin the Object-based audio section, are:




All the provided plugins belong to one of the three categories above.

Within the installer, pre-populated object-based DAW sessions are currently available for REAPER

3.1 REAPER

Open <install-directory>/resources/DAW_Sessions/REAPER folder, whereinstall_directory is the path specified during the installation. The default position is /Applications/VISR-Production-Suite on MacOS and C:\Program Files\VISR-Production-Suite onWindows.

There you will find two sub-folders: loudspeaker_renderer and binaural_renderer, eachone containing two Reaper project called example_project_VST_loudspeakers.RPP andexample_project_VST_binaural.RPP, respectively.

Note: if you installed the VISR Production Suite through the standard installer you will not have the BinauralRenderer plugin nor the relative REAPER session, therefore you will only see the loudspeaker_rendererfolder. You can simply ignore the Binaural Renderer section in this documentation in that case. The followinginstructions are made considering the full (Python x.x) installations.

The only change between the two sessions is that the first renders the object-scene for a loudspeaker layout ofyour choice, whereas the second renders to binaural, which can be used by headphones or CTC systems.

Note: The two sessions are identical apart from the rendering plugins, which means that potentially, thanks toobject-based audio, you could have just one object-based session and switch between different renderer plugins

19


to reproduce the sound in different environments. However be aware that, for the time being, only one rendererinstance can be present in each session at a given time, therefore if you want to change the renderer you can easilydo it via removing the previous one and instantiating a new one (e.g. deleting the Loudspeaker Renderer andinstantiating a Binaural Renderer).

Open one of the two Reaper projects.

In each of the two Reaper sessions you will see four tracks in total:

example_project_VST_loudspeakers.RPP example_project_VST_binaural.RPP

• OBJECT 1 • OBJECT 1

• OBJECT 2 • OBJECT 2

• SCENE MASTER • SCENE MASTER

• LOUDSPEAKER RENDERER • BINAURAL RENDERER

Note: On MacOS, if you run the Reaper session from the /Applications/(...) folder, it is likely that youcannot save it in that place. So please “Save as ..” in another folder, or copy it beforehand in another place in thefilesystem.

Note: Should you be asked to provide the folder containing some audio files used in the ses-sion, Click on “Search . . . ” and provide the <install-directory>/resources/DAW_Sessions/common_material folder.

3.1.1 OBJECT 1, OBJECT 2 Tracks

Those are usual DAW audio tracks, containing audio items.As soon as they are equipped with the Object Editor plugin they become audio objects (Objects category) andstart to be part of the object-based scene .

Pressing the FX button on one of the two tracks, you can have a look at the loaded Object Editor plugin that wealready introduced in the corresponding Introduction section.

20 Chapter 3. Tutorial 1: Using VISR Production Suite


This plugin contains 3D panning parameters as automatable metadata.The right part of the GUI consists of a 3D sphere grid centred in a small grey sphere which represents thelistener. The coloured small sphere is the representation of the object, and a same coloured ellipse represents thehorizontal plane in which the object lays.

If you drag the coloured sphere around you actually change the position of the object in the scene:

• left-click + drag around : To move the source’s azimuth, which is the angle on the horizontaln planebetween the object and the reference front-facing point on the sphere surface, from the listener point ofview.

• right-click + drag up/down : To move the source’s elevation, which is the angle on the vertical planebetween the object and the reference front-facing point on the sphere surface, from the listener point ofview.

A value increment/decrement widget and three knobs are also provided:

• ID: this is the object identifier, which must be unique among the many object tracks that you may have in asession.

Note: Important: normally you must ensure that each instance of the plugin has a different ID number bymanually adjusting the ID values. However this is already ensured in the provided session, so you don’t have todo anything about that.

• Azimuth: changes azimuth (same as left-click + drag around)

• Elevation: changes elevation (same as right-click + drag up/down)

• Radius: is the radius of the sphere, which means the distance between the listener (you) and the virtualaudio object.

Note: None of the current renderers uses the radius information for rendering, which means that differentvalues of radius make currently no audible difference. Nonetheless it is a conceptually important metadataand will be used in the future releases.

Everything that is done on a per-object-track level can be seen in real time in the following plugin, which collectsall the positional information of the Object Editor plugins.

3.1. REAPER 21


3.1.2 SCENE MASTER Track

This track is a stub track which conceptually only serves to contain this plugin and which is not used to performany audio input/output.The role of this plugin is to allow you to control and edit the whole object-based scene.

Press the FX button on the track, you will open up the Scene Master plugin that we already introduced in thecorresponding Introduction section.

The right part of the GUI consists of the same kind of 3D visualisation as the Object Editor, where here you cansee all the objects present in the session under the form of differently coloured small spheres.

Note: unlike in the Object Editor plugin, here you cannot drag the objects around within the 3D sphere visuali-sation, but you can achieve the same effect by editing the position from within the object table on the left.

On the left part there is a table with the complete list of all the tracks that are also audio objects (through theusage of the Object Editor plugin).You can control their positional metadata by changing the associated polar coordinates (azimuth, elevation,radius), and you can do that either entering the value in the corresponding text box fields or by clicking/draggingthe corresponding “+” and “-” buttons.

All these data about the objects, and therefore about the entire scene, are sent to the renderer plugin you are using( Loudspeaker Renderer in theloudspeaker_renderer/example_project_VST_loudspeakers.RPP session, BinauralRenderer in the binaural_renderer/example_project_VST_binaural.RPP session)

3.1.3 LOUDSPEAKER RENDERER Track

(only in loudspeaker_renderer/example_project_VST_loudspeakers.RPP)

You can consider this track as what’s normally called Master track in DAW environments, because it is where allthe channels of the tracks containing the audio object are routed to and where the actual rendering is performed.



This track also receives the object-based scene data from the Scene Master plugin in order to take decisions onhow to simulate the position of the objects with the given loudspeaker layout.

Pressing the FX button on the track, you will open up the Loudspeaker Renderer plugin that we alreadyintroduced in the corresponding Introduction section.

The right part of the GUI consists of the same 3D visualisation as the Scene Master, but with the addition of arepresentation of the chosen loudspeaker layout. Loudspeakers are displayed as green squares with numbers thatindicate the channel number to which each loudspeaker is assigned to.

The default loudspeaker layout is stereo, but you can choose a different one via the drop-down menu on the left.To do so, click on it, and then click on the “Select from folder . . . ” drop-down menu item.Then go to <install-directory>/resources/loudspeaker_layouts folder to try one of theprovided loudspeaker configurations.

Note: unlike in the Object Editor plugin, here you cannot edit the position of the objects (e.g. dragging them onthe 3D sphere visualisation) because at this point the scene is already completed and this plugin just performs therendering of the given scene.

3.1.4 BINAURAL RENDERER Track

(only in binaural_renderer/example_project_VST_binaural.RPP)

Note: to see and instantiate the Binaural Renderer, you might need to perform a “clear cache/re-scan” of theplugins after installing. To do that press the corresponding button in REAPER “Preferences” window (REAPER/Preferences... on mac, Options\Preferences... on Windows), under “VST” tab.

Like for the Loudspeaker renderer track (in the corresponding session), you can consider this track as what’snormally called Master track in DAW environments, because it is where all the channels of the tracks containingthe audio object are routed to and where the actual rendering is performed.

3.1. REAPER 23


This track also receives the object-based scene data from the Scene Master plugin in order to take decisions onhow to simulate the position of the objects with the given HRIR (Head Related Impulse Response), HOA (HighOrder Ambisonics) to binaural filter, or BRIR (Binaural Room Impulse Response).

Pressing the FX button on the track, you will open up the Binaural Renderer plugin that we already introducedin the corresponding Introduction section.

When you first instantiate the plugin, the provided Koln HRIR L2354 Dataset is used as default for HRIR binauralrendering approach. We will explain how to configure the renderer with some options, including the SOFAdatasets to be used, in the Binaural Renderer properties configuration subsection.

On the left part of the plugin interface you can see a drop-down menu where you can choose between threedifferent binaural rendering approaches:

• HRIR: This approach uses a set of Head Related Impulse Responses to dynamically select a Left and Rightear filter pair to be applied to the signals of the objects based on their position.

• HOA: the audio objects are encoded in High Order Ambisonics, and HOA signals are created applying thecalculated spherical harmonics coefficients. Finally Ambisonics signals are convolved with special filters toget a binaural output.

• BRIR: similarly to the first one, this approach uses a set of Binaural Room Impulse Responses to dynami-cally select a Left and Right ear filter pair to be applied to the signals of the objects based on their position.

You can find more detailed information about the approaches in the VISR BST (Binaural Synthesis Toolkit)documentation. In fact the plugin is actually a wrapper around that python package, made to provide its binauralrendering in a DAW plugin.

Binaural Renderer properties configuration

As we said, we provided a HRIR dataset to be used by default, but for using the other two approaches (HOA andBRIR) you need to provide a compatible SOFA dataset.Here are some datasets that you can download and use:

• HRIRs: Koln HRIR L2354 Dataset (already provided)

• BRIRs: BBC R&D BRIRs (info at https://github.com/bbc/bbcrd-brirs)

If you press the gear button next to the drop-down menu you will see a set of configuration parameters that can bechanged to customise your experience:


http://sofacoustics.org/data/database/thk/HRIR_L2354.sofa

http://sofacoustics.org/data/database/thk/HRIR_L2354.sofa

http://data.bbcarp.org.uk/bbcrd-brirs/sofa/

https://github.com/bbc/bbcrd-brirs


• Max # objects : is a field that lets you optimise the rendering in case you are not exceeding a certain numberof audio objects. As for this session, you can set it to 3 or more.

• Tracking properties: you can find detailed information on how to use a tracker in Tutorial 3: Use a headposition tracker with the Binaural Renderer. If you don’t have a tracker or you don’t want to use it, simplytick the “Off” checkbox.

• SOFA file path fields : for setting the corresponding HRIR dataset, HOA-to-binaural filters, BRIR dataset,respectively. The SOFA (Spatially Oriented Format for Acoustics) file format is a standardised (AES69-2015) file format for acoustic data.

Set those text fields to the path of the SOFA datasets you want to use, e.g. BBC R&D BRIRs for theBRIR approach.

• Ldspk Config file path : when using BRIR approach you should provide the loudspeaker configuration usedfor generating the BRIR dataset you are using. To do so you must provide that configuration through anXML configuration file. You can find the loudspeaker configuration files for the BBC R&D BRIRs datasetin <install-directory>/resources/loudspeaker_layouts/bbcrd-brirs folder.

• dynamic ITD : whether the delay part of the HRIR/BRIR is applied separately to the (delay-free)HRIR/BRIR. That requires preprocessed HRIR/BRIR data.

• interpolation : Whether HRIR/BRIR grid points should be interpolated for the current head oriention. Ifunchecked means nearest neighbour (no interpolation) is used, when checked enables barycentric interpo-lation.

• crossfade : whether dynamic HRIR/BRIR changes are crossfaded (checked) or switched immediately(unchecked)

• brir truncation length : to save computational resources when using BRIRs: BRIR datasets can be verylarge because contain the reverberation information of a room, this field is needed for a responsive rendering.

When the tracking capabilities are not enabled (see Tutorial 3: Use a head position tracker with the BinauralRenderer), the right part of the GUI consists of the same 3D visualisation as the Scene Master.

Note: unlike in the Object Editor plugin, here you cannot edit the position of the objects (e.g. dragging them onthe 3D sphere visualisation) because at this point the scene is already completed and this plugin just performs therendering of the given scene.

3.1. REAPER 25

https://www.sofaconventions.org/mediawiki/index.php/SOFA_(Spatially_Oriented_Format_for_Acoustics)




3.2 Conclusion

Now that you know in practice the basic features of the various plugins contained in the suite, listen to the providedsession by starting the playback, and try to change and hear the position of the two audio objects.

If you want to try your own material in an empty session, you can also use the templates that you find in<install-directory>/resources/DAW_Templates. If you want to understand how to use them orhow you can create a session from scratch try the Tutorial 2: Create your first object-based DAW session


CHAPTER

FOUR

TUTORIAL 2: CREATE YOUR FIRST OBJECT-BASED DAWSESSION

If you have followed the previous tutorial, now you are ready to create your own object-based DAW Session.

First of all, to avoid having to create an object-based session from scratch everytime, you can use the providedtemplates inside <install-directory>/resources/DAW_Templates folder, whereinstall_directory is the directory specified during the installation. The default position is/Applications/VISR-Production-Suite on MacOS and C:\ProgramFiles\VISR-Production-Suite on Windows.

There you will find two sub-folders: projecet_templates and track_templates. In REAPER you candefine project templates and track templates, the latter being sets of tracks that you can use in different projects.

In each of the two folders you will find files with the following naming patterns:

• <x>obj_session_binaural_renderer : this is a full object-based session with a Binaural Renderer and xas the number of objects (with the corresponding tracks) being used.

• <x>obj_session_loudspeaker_renderer : this is a full object-based session with a Loudspeaker Rendererand x as the number of objects (with the corresponding tracks) being used.

• <x>obj_session_no_renderer : this is a full object-based session with no renderer istantiated and x asthe number of objects (with the corresponding tracks) being used. Note that while the renderer is notinstantiated, a “RENDERER” track is present in order to have the object routing in place. In that way youcan setup these project/track templates by just instantiating your preferred renderer in the “RENDERER”track.

If you want those templates to be available by default in REAPER templates, you need to copy/pastethe content of projecet_templates into <REAPER folder>/ProjectTemplates andthe content of track_templates into <REAPER folder>/TrackTemplates. By de-fault <REAPER folder> is ~/Library/Application Support/REAPER on MacOS, andC:\Users\<username>\AppData\Roaming\REAPER on Windows.

To use a project template, just open it as you would normally do with a REAPER session and you’re ready to usethat just by following Tutorial 1: Using VISR Production Suite. In order to use track templates you just have toopen a REAPER session and go to Track menu and chose the template from Insert track from template dropdown menu.

27


4.1 Create a session from scratch

Note: While you can skip this tutorial and start using the templates, it is important for you to understand themanual configurations that need to be performed, because you will likely need them in the future.

Once again, you are about to create a DAW session/project that contains all the fundamental elements of anobject-based production (see Object-based audio section), which are:




This tutorial is currently provided for REAPER

4.1.1 REAPER

As a starting point, create a new project and create 4 audio tracks.On the first two tracks place one Object Editor plugin each, opening the FX window on each track and clickingthe “add” button.

As said in the previous tutorial, each Object Editor, and thus every track that becomes an audio object, has anID field which is the object identifier, that must be unique.

28 Chapter 4. Tutorial 2: Create your first object-based DAW session


When you add a new Object Editor the default ID is always 1. You must ensure that each instance of the pluginhas a different ID number by manually adjusting the ID values.In this case, having just two object tracks, it is sufficient to change the ID value of the Object Editor in thesecond track, from 1 to 2. Following the same logic, if you need more objects you will increase the object IDprogressively in the new Object Editor instances.

Once you have done that, a Scene Master plug-in is needed, which will collect the information regarding the twoaudio objects. To do that, add a Scene Master plugin through the FX button on the third track.Now you should already see that the changes performed on the Object Editors are mirrored in the Scene Master.

Then we need a renderer in place to eventually use those object information for rendering.Therefore place one renderer of your choice (Loudspeaker Renderer or Binaural Renderer) in the remainingtrack.Remember that if you want to use more than two output channels (default for Reaper tracks) you always have tochange the number of channels of the track, via the “Route” button (on the track) -> “Track channels”. This is thecase for the Loudspeaker Renderer, where you may want to support audio formats with more channels than juststereo: the suggestion is to set the number of channels to 64, which is the maximum number of channels for atrack in Reaper.

Note: to see and instantiate the Binaural Renderer, you might need to perform a “clear cache/re-scan” ofthe plugins after installing. To do that press the corresponding button in REAPER “Preferences” window, under“VST” tab.

Now add two mono files of your choice to the two object tracks.At this point there is one thing missing: the audio content from each object track needs to be routed to one of therenderer’s channels.To do so, press the “Route” button in the first object track and go to “Add new send. . . ” drop-down menu toselect the renderer track, as shown in figure.

Also remember to untick the “Master send” checkbox on each object track, since we do not want to send it tothe master track, but just to one of the channels of the object-based renderer track.It is crucial that each audio object is routed to a different input channel of the track where the object-basedrenderer is:

4.1. Create a session from scratch 29


4.2 Conclusion

Now you should have everything in place and you can start the playback and listen to your first object-basedsession!

You can follow the same procedure for session with many more objects, being aware of the necessary attentionthat needs to be paid to the objects IDs and routing channels configurations.

30 Chapter 4. Tutorial 2: Create your first object-based DAW session

CHAPTER

FIVE

TUTORIAL 3: USE A HEAD POSITION TRACKER WITH THEBINAURAL RENDERER

This tutorial is about integrating a head tracker with the Binaural Renderer plugin, in order to perform real-timehead orientation compensation and therefore to take advantage of dynamic cues in the binaural rendering.If you have followed the Tutorial 1: Using VISR Production Suite, now you are ready to go on with this tutorial.

As you can see from the figure below, in the settings panel the tracking is disabled by default ( Off ).

You can enable it by selecting one of the two mechanisms of passing the head compensation information.

31


5.1 OSC

(Coming soon)

5.2 Internal

(Coming soon)

32 Chapter 5. Tutorial 3: Use a head position tracker with the Binaural Renderer

CHAPTER

SIX

TUTORIAL 4: CREATE YOUR FIRST REVERB-OBJECT-BASEDDAW SESSION

Note: to use the Reverb Object you must enable the ReverbObjectEditorVST3 and thereverb_resources check boxes during the installation of the VISR Production Suite.

If you have followed the 1st and 2nd tutorials, now you are ready to create your own object-based DAW Sessionwith Point Source objects. In addition to creating these ‘dry’ objects using the Object Editor VST, you can alsocreate reverberant objects using the Reverb Object Editor plugin. There are however, some things you shouldknow before getting started with this.

Within the installer, a pre-populated object-based DAW session is currently available for REAPER

6.1 REAPER

To get you started, open the resources file example_project_reverb_VST_loudspeakers.RPPsession, provided in<install-directory>/resources/DAW_Sessions/REAPER/loudspeaker_renderer.

6.1.1 Reverb_OBJECT 1 Track

Similarly to what you see in Tutorial 1: Using VISR Production Suite this is an usual DAW audio track,containing audio items.Pressing the FX button on one of the track, you can have a look at the loaded Reverb Object Editor plugin.

33


It is conceptually the same thing as the Object Editor but this plugin contains also a drop-down menu containingdifferent room environments to choose from.

Changing Rooms

Per Reverb Object Editor, there are a set of room to choose from. These are pre-recorded impulse responses, allof which have different reverberant qualities. Currently the rooms available are:

• emm_main

• null (dry point source)

• emm_old

• vislab

• st_andrews

• maes_howe

• york_min

• st_patricks

• tynd_bruce

• sports_hall

• nuclear_hall

• bbc_ul

• bbc_listening_room

• tunnel

34 Chapter 6. Tutorial 4: Create your first Reverb-Object-Based DAW session


• outside_rockface

• courtyard

• cvssp_kitchen

• dwrc

• bridgewater_hall

Whilst in playback, you can move between rooms in real-time, whether you are using the Binaural Renderer orLoudspeaker renderer.

If you cannot hear anything, make sure you check the Loudspeaker Renderer or Binaural Renderer is not throwingyou any errors. If there are, make sure you have set up the Settings configuration in the Loudspeaker Renderer, orthe Settings configuration and object renderer options correctly in the Binaural Renderer.

Adding New Rooms

If you are a developer or researcher, you can use the RSAO by Luca Remaggi (On S3A Organisation’s GithubRepository) to calculate your own room impulse responses, and add these rooms to the<install-directory>/resources/RoomIRs.json file. Currently this is only supported via changingthe RoomIRs.json file (it is not available through the UI). Be sure to make a backup of this file somewhere elsebefore you start tampering with it.

You can do this by taking the output room JSON file you get from RSAO, and appending it to the"roomlibrary", which is an array in JSON format. There are various online JSON editors and tree views thatcan both help you do this, and also validate the JSON. Failing to do so will throw unexpected errors in the ReverbEditor UI window.

On reloading a REAPER session, you should now see this room in the dropdown menu.

6.1.2 OBJECT 2, OBJECT 3 Tracks

See track explanation in Tutorial 1: Using VISR Production Suite.

6.1.3 SCENE MASTER Track

See same track explanation in Tutorial 1: Using VISR Production Suite.

6.1.4 LOUDSPEAKER RENDERER Track

The Loudspeaker Renderer plugin uses some settings to render Reverb Objects, which are available via aspanner icon in the UI. Clicking on this spanner will show you some reverb configuration rendering options.

6.1. REAPER 35


• Max # reverb objects : The amount of reverb objects should be equal to the amount of Reverb ObjectEditors you have in the session ( e.g. if you have 3 Reverb Editors on 3 REAPER tracks, you should specify3 objects in this configuration)

• Number of discrete reflections : The Renderer supports a minimum of 6 reflections, and a maximum of 20reflections, per Reverb object.

• Late reverb length (seconds) : The late reverb length is the length of the reverb tail. This is how long ittakes for the reverb to fade out. Longer tail lengths will consume more CPU resources. I suggest between3-6 seconds as a good starting point.

• Late reverb decorrelation filters : You can specify a decorrelation filter path using this field. There is adefault decorrelation filter path already set in the renderer, so it is not strictly required to set a new one.

• Crossfade Filter length : This is how long it takes in seconds to fade between one room and another. Thisallows seamless transitions between rooms in real-time, as if you were walking from one room to another.The default is 2 seconds, and you can set this to higher or lower. Higher transition lengths will also consumemore CPU resources.

Depending on your CPU resources, more Reverb Objects will consume more resources, and you may start toexperience some audio latency and artifacts. In practice you can’t currently use a lot of Reverb Objects becausethey are very CPU-expensive. You can try the following to reduce this consumption though:

• Reduce the number of reflections.

• Reduce the late tail (in seconds)

36 Chapter 6. Tutorial 4: Create your first Reverb-Object-Based DAW session


6.1.5 Optional Binaural Renderer Configuration

There is also some support for binaural rendering too. Follow the instructions in Tutorials 1, 2 and 3 on usingnormal Objects and the Binaural Renderer together.

Reverb and Binaural is currently only available to use with the BRIR approach, so please follow the specifiedinstructions in tutorial 1 on how to do that. Within the same BRIR section, there is an option file called objectrenderer options. This configuration is specified in JSON format. Copy and paste this configuration, andadjust if necessary:

{"reverbConfig": "{ \"numReverbObjects\": 1, \"discreteReflectionsPerObject\":20, \"lateReverbFilterLength\": 4 }" }

This would equal 1 reverb object with 20 reflections and a late tail length of 4 seconds. Refer to the LoudspeakerRenderer Configuration section on what these settings mean and how to modify them.

6.1. REAPER 37

VISR Production Suite User documentationThis makes it the perfect audio tool for VR. The plugin has...

Documents

Transcript of VISR Production Suite User documentationThis makes it the perfect audio tool for VR. The plugin has...