Forensic Video Analysis – Step by Step 

Through an examina on of the underlying digital data, a Forensic Video Analyst is able to make the correct decisions when 

dealing with a piece of proprietary video. 

The Supreme Court of Canada once stated, “The video camera is never subject to stress. Through tumultuous events it con n-

ues to record accurately and dispassionately all that come before it. Although silent, it remains a constant, unbiased witness

with instant and total recall of all that it observed”.

Upon ini al reading you may believe this to be en rely accurate. It could be argued however, that with inferior lenses and

poor quality imaging chips, that ‘stress’ is put into the camera by the manufacturer. The most important part though is a sys-

tem’s ability to record and retain accurately the images captured by a camera. The responsibility for this task is given to the

Digital Video Recorder (DVR). If the DVR cannot record accurately, and then export that in an eviden al manner, it is up to the

analyst to reveal the truth.

In this walkthrough of an inves ga on, involving an overt surveillance video file, David Spreadborough highlights some of the

challenges faced and how video analysis now start with the data.

My case starts with the arrival of an op cal disk. Unlabelled, and without a protec ve case, the ini al few moments are always

of interest whilst I establish the presence of actual data. Unfortunately, blank disks are an all too common occurrence. With

the presence of data files confirmed, I now rou nely scan the media for viruses before copying the contents to my work-

sta on. I may also create a series of file hash values, if it is likely that the file is also to be examined by another member of

staff. This ensures that errors caused during any copy process are iden fied. In this instance though, hashing is not required

due to the export already being supplied with a series of values. Although export hashing is quite uncommon, the disk struc-

ture is very familiar, consis ng of a video file and an associated player.

There are hash values for the video file and the player but,

also included, is a .txt file that contains the names for each

camera. Now that I have understood what is contained with-

in the disk, it’s me to have a look at the player and preview

the video.

It has a simple interface with basic

controls for window layout, search,

speed and playback. The eject icon

opens a search dialogue box with a

useful piece of informa on. Within

the ‘files of type’ box, it describes

a .ps file as an MPEG Video file type.

MPEG, standing for Mo on Picture

Experts Group, is a commi ee that

set standards in Video compression.

The compression is required in or-

der to ensure that the footage com-

plies with a desired delivery or stor-

age method.

By selec ng the .ps file supplied, the

player comes alive and seven cam-

era boxes are filled with video.

Firstly, the textual overlay on the video displays a camera name. It is apparent that this is taken from the .txt file on the disk.

Next, you may no ce that the me and date are displayed out of the main window matrix. This usually highlights that there is

a separate me index, either in a separate file or forming part of the video stream. Finally, one of the windows has the word-

ing “[aud]” inside.

Before I do anything else, I am required to test certain func ons of the player to assess its capabili es. We also now have the

possible presence of audio [aud] to be confirmed, and then establish how to deal with the video streams for further inves ga-

on.

In mul -camera view, I hear nothing through my speakers. When selec ng the single camera view of the camera with the

[aud] overlay, the low hum of an audio track is heard. The worrying part is that other audio streams are there! By selec ng

single view for cameras 1,3 & 4, audio is also heard. These are not iden fied in the mul -camera view and could be easily

overlooked. Now that the presence of audio is confirmed, it’s over to dealing with the visual elements first.

The preview window in single camera mode starts off displaying a small view.

By right clicking, the camera view gets bigger, to fill the en re pane. The popular ‘floppy disk’ save icon is on a bu on near the

bo om of the interface. Unfortunately, this only gives me single image saving. More problema c however, is that it does not

ma er what view is chosen, all images are exported as they are displayed. Either as a small image with a large black border, all

the way up to a large, highly pixelated image. All images are produced at a pixel size of 771px x 547px.

By measuring the player interface I confirm that the preview pane is 771px x 547px. This means that the exported images are

merely screen shots and NOT a true frame extrac on from the video stream. Lastly, during our assessment of the player’s ca-

pabili es, there is no method to deal with the video out of the player.

My assessment of the player and footage has raised a number of tasks:

Iden fy video pixel size

Iden fy the Stream type.

Iden fy methods to extract the Video and the Audio (if required)

Iden fy video encoding issues

Iden fy clarifica on possibili es

If you remember, we have started with a single .ps file containing seven video streams and have now iden fied a further 4

audio streams. Media Data Analysis detects the presence of an MPEG 4 Visual File.

MPEG 4 compression has been split into a number of parts, with each part defining further advancements in compression

technology. MPEG 4 Visual is Part 2. Interes ngly, the so ware has only been able to detect a single video stream and as such,

further analysis and verifica on is required.

FFprobe is a command line analysis and repor ng tool for mul media data streams. It is compiled alongside FFmpeg and

FFplay, with the libraries used by many mul media players and programs. By opening a command prompt from within the

FFprobe folder, I am able to examine a given file and produce a text file report.

My command window and the resul ng .txt file now reveals 11 streams, with stream 0,1,9 & 10 being Audio. The .txt file con-

tains a lot of useful informa on including the stream mapping structure. Stream2 example:

[STREAM] index=2 codec_long_name=MPEG-4 part 2 width=352 height=288 sample_aspect_ra o=12:11 display_aspect_ra o=4:3 pix_fmt=yuv420p start_pts=5541901566 dura on=0:03:59.839467 nb_read_frames=N/A [/STREAM]

To visualise each stream, FFplay can be used to only play a specific stream number from the desired file by including the de-

sired stream number to the ffplay command.

You may have no ced from our

ini al FFprobe output that

there was no frame count

(nb_read_frames). This is due

to all streams being encapsu-

lated into the single proprie-

tary file. In order to assess each

camera stream, they have to

be extracted.

Extrac on of streams can, in

many cases, be completed us-

ing FFmpeg. A er reading the

ini al .ps file, it’s possible to

create new files containing

only a desired camera stream.

The benefit of conduc ng the

stream extrac on this way is

that the footage retains its

original digital structure and, as

such, can be analysed further.

A er each video stream is ex-

tracted, individual file analysis

on each stream can be com-

pleted and this me we have

frame counts.

Our last piece of analysis here

is to assess each individual

frame.

For this we use FFprobe again, but this me sending the report to an .xml table, best viewed with MS Excel or similar.

The Presenta on Time Stamp and the Decoding Time Stamp for each frame is visible and I can see that there should be approx.

6 Frames per Second. When analysing mo on in footage, this informa on can be highly important in iden fying variable frame

rate recording. If a piece of footage has been recorded with differences between frames but is played back, or transcoded, with

a constant frame rate, considerable misinterpreta on could be caused.

I can see that each frame has a set width and height of 352 x 288. It is important to verify a video’s size parameter by analysing

the informa on at a frame level. It is common for video files to have an increased size se ng, thus forcing a player to upscale

footage. Obviously, this is something to be very wary of, if planning to conduct any image clarifica on.

The Sample Aspect Ra o is shown as 12:11. This is important as a player uses this informa on to display it correctly. The math

to calculate S.A.R to D.A.R:

352 pixels (width) x 12 = 4224

288 pixels (height) x 11 = 3168

4224/3168 = 4:3 Aspect Ra o

The problem though is that 352x288 gives an aspect ra o of 11:9. We must remember this so a er our image analysis work,

the final image can be corrected to ensure the files 4:3 Display Aspect Ra o is maintained. It is worth poin ng out that this is

just for the recorded file and does not take into account any distor on caused at the start by the lens and/or camera.

You may also no ce that each frame type is visible. MPEG video can consist of a number of different frame types and these are

an important part of the compres-

sion standard. The subject of mpeg

compression is another ar cle in

itself, but in this instance we have I

frames and P frames. I frames, or

Intra frames, do not rely on any oth-

er frame in order to be constructed.

P frames, or Predicted frames how-

ever, are ‘predicted’ from a previous

I or P frame. It is impossible for them

to be constructed without the previ-

ous frame. The set of pictures be-

tween I frames is called a Group of

Pictures or “GOP”. Looking back to

Image 7 we can see that we have a

GOP of 8, with the structure visual-

ised here:

In many cases it is important to not only iden fy the specific frame types, but to also look deeper into the predicted frames.

This assists in clarifying any movement across predicted frames to visualise what components are new, what have been copied

and moved from a previous frame and also what has been held over from a previous frame. These components are called mac-

ro-blocks and are made up of blocks of pixels.

This P frame has an area of movement in the bo om le . The encoding condi on of each macro-block can be visualised. The

arrows indicate mo on vectors. This is the path taken of a macro-block from one frame to the next. As an object moves across

pictures, if the encoding algorithm detects a change of loca on but no change in its pixel structure, it only has to be moved and

not newly encoded. The condi ons and movement can assist greatly in iden fying what parts of an image have been ‘le over’

from a previous image and, more importantly, pinpoint newly encoded macro-blocks a er items enter a picture before the

next I frame.

We are now star ng to get somewhere with our analysis. In a short period of me we have established the encoding method,

extracted the individual camera streams and have reports containing their frame structure. A er a visual review of specific key

parts, we have reviewed the encoder’s ability to correctly capture detail, predict mo on and its accuracy across frames.

When relying on informa on provided by individual so ware it is important to compare any results against other so ware.

Visual verifica on can also be completed. In this case I have used one of each.

In order to validate my findings on the individual camera streams I have used another piece of so ware. Although primarily

designed for the iden fica on of video streams in raw data, Defraser can also be used to iden fy camera streams in a proprie-

tary file.

Defraser also iden fied 7 Video streams and 4 Audio streams.

A Visual verifica on was made to confirm the I frames. By moving frame by frame through a por on of the video stream in

Virtualdub, the higher quality I frames could be easily seen. Virtualdub also has the capability to display the frame type and the

frame number. All of these details were compared and verified against my original FFprobe reports.

When comparing the results from one piece of so ware to another it is important to ensure that they are not using the same

‘engine’. If both pieces of so ware were using the FFmpeg libraries then any comparison could be negated.

My visual analysis, and presenta on prepara on, is now complete. It’s me to briefly head back to the audio. A review of all

four streams present within the player determined that all four streams were merely noise. This suggests that no microphones

were a ached to the system. If Audio had been recorded though, the end result may not have been as successful as the visual

side. Ini al tests reveal that although the MPEG audio frame headers are visible, the streams do not contain the required infor-

ma on for them to be decoded effec vely. Informa on such as sample rate, bit rate and encoding type usually follow the

header but this informa on was not present. Again, this is fairly common in surveillance recordings, where certain decoding

informa on is contained within the player rather than the video/audio file.

Before individual frame or image analysis is carried out, I have been requested to author a Video DVD for presenta on. Trans-

coding digital mul media from one format to another, or even crea ng a number of versions for different purposes, is a com-

mon task and one fraught with possible challenges. Any change in structure must be documented and referenced back to the

original, exhibited file. For this reason it is good prac ce to keep any process as simple as possible and be confident in either

the so ware being used or the process applied. A er transcoding, new file(s) must be checked and referenced with the origi-

nal to ensure that no visual eviden al loss has occurred.

Next a visual viewing log is completed using the extracted camera streams. This could be completed in a Video Non Linear Edi-

tor (NLE), or with other playback so ware and a report writer. Today though I am going to use ForeVid.

The individual streams extracted through FFmpeg can all be imported directly into the so ware. As no transcoding occurred I

can be assured that what I see, and refer to, is exactly what was recorded. You would be surprised at how many players apply

a hidden de-blocking and sharpening filter, thereby changing the image to appease the human eye. During playback with

ForeVid, frame bookmarks can be taken and annotated. The bookmarks can later be exported as a series of images, but in my

case, I wish to export the en re report as a PDF. This includes all frame images and the notes made for each one.

By comple ng the Video DVD, followed by the annotated frame report, I have established a good knowledge of the video con-

tent and from here it is me to complete our image analysis.

Due to comple ng the ini al format analysis and iden fying frame types, it is now possible to export only the frame types I

desire to aid in my inves ga on. These can either be extracted as a batch using FFmpeg or, as is the case here, a small selec-

on chosen. It is the ‘I’ frames that are needed in order to eliminate any predic on changes in the image. A er using Pho-

toshop’s ‘Smart Stack’ technique, a great deal of noise is reduced in the image, enabling a much higher clarifica on of sta c

objects. This method is highly useful during night me footage to reduce the added noise caused by IR illumina on.

Finally, a er clarifica on of my images of interest, they were resized ensuring my final aspect ra o kept the 4:3 Display Aspect

Ra o dictated in my original video stream. All images were then compiled into sequences and then exported to Image PDF.

As all my reports are in PDF format, they are easily shared and viewed by all persons within the inves ga on.

Without the prior understanding of the video technology and then how to iden fy the

influen al informa on, any judgements made on the visual elements could turn out to

be wrong. The significance of visual evidence within the legal system is increasing dra-

ma cally, as people expect to see something rather than just hear about it. If a court is

to trust what they are seeing, the person producing that item must be able to explain

what it is and what they have done. Gone are the days of the, “push bu on forensics”.

I hope that this walk through an inves ga on has highlighted some of the changes in

Visual Forensics and how it’s the data that points us in the right direc on and not the

image. If I had relied on what the player was telling me, I would have been wrong. My

opportuni es for clarifica on would have been reduced and any judgements made on

an image could be ques oned. Some of the analysis methods used in this case are not

going to work with every proprietary video file. It all comes down to iden fying the

stream type and understanding what tools will be suitable in that case. Automated

tools may also not be able to iden fy the format, and it may come down to establish-

ing the raw frame headers in hex. Whatever the file is, if it’s from a video surveillance

device, you can pre y much guarantee that the so ware supplied is going to leave you

with ques ons, and it will be down to you to find the answers.

The following two pages contain some of the FFprobe, FFplay and FFmpeg commands used during this examina on.

About David

David has been a Police Officer for 23 years. 10 years ago, a er iden fying a need for local video support for officers, he de-

vised the first Divisional CCTV Inves ga on Unit in Cheshire Constabulary. He now works within the Force Visual Forensic Unit,

conduc ng Forensic Video Analysis.

So ware used:

MediaInfo

FFmpeg, FFprobe, FFplay

Microso Excel

Defraser

Virtualdub

AVS to DVD

ForeVid

Adobe Photoshop

Adobe Acrobat

Can the FFmpeg library read a file? 

ffmpeg -i thefile.ps

The ‐i denotes the input file. It is some mes necessary to force a file to be read as a specific codec. To force 

a file to be read as h264, you would simply add ‐f h264 before the ‐i . 

 

Ini al probe of file: 

ffprobe -show_format -show_streams -count_frames -pre y thefile.ps > thefilePROBE.txt

This command will output an easy to read text file showing the detected format, the detected streams, 

and if possible, the total number of frames.  

 

To play individual streams within the file (example shows stream number 2): 

ffplay -vst 2 thefile.ps

To play only a selected stream, ‐vst (number) can be placed before the file. ‐vst is short for Video Stream.   

 

Extract streams from encapsulated file (example shows stream number 2):    

ffmpeg -i thefile.ps -map 0:2 -vcodec copy -vsync 0 thefile_stream2.mp4

A er iden fying the the input file, the ‐map flag, followed by stream number, states that the output file 

must only contain that specific stream. The ‐vcodec copy directs the so ware not to transcode the output. 

The ‐vsync 0 directs that each frames  mestamp is passed with its original  mestamp. Finally the output 

file loca on and name is given. 

 

Individual frame analysis of file: 

ffprobe –i show_frames –print_format xml thefile_stream2.mp4 > thefilePROBE.xml

This will output a detailed list of every frame in a video file and place it into an xml datasheet. 

 

Create macro block analysis Mo on JPEG video: 

ffmpeg -debug vis_mb_type -vismv pf -i inputstream.mp4 -vcodec mjpeg -q:v 2 -f avi

thefile_stream2_Analysis.avi

Prior to iden fying the input stream, the two flags here mark it to be read whilst showing the Macroblock 

type and the Predicted Frame Mo on Vectors. A er the input stream, the output codec has been marked 

as Mo on JPEG with the highest quality parameter. Finally the –f avi directs the output format.  

 

 

Create High Quality Mpeg2 files for PAL DVD Authoring: 

ffmpeg -i inputstream.mp4 -vf "scale=iw*2:ih*2,pad=720:576:8:0,setdar=4:3" -c:v mpeg2video -b:v

6000k -minrate 4000k -maxrate 6000k -bufsize 2000k -dc 9 -flags +ilme+ildct -alternate_scan 1 -top 1

outpu ile.mpg

A er the Input stream, the ‐vf stands for video filter. The filter chain specified in this instance begins with 

the scale filter to increase the width and height by 2. Next, as the increased width does not meet the 

720x576 PAL requirement size, the output has been padded by 8 pixels on either side. Lastly the Display 

Aspect Ra o has been set to 4:3. Following on from the Filter chain, I have a series of flags to set the en‐

coding type and quality. The last set informs the muxer to create the output as interlaced.     

 

Batch export of all ‘I’ frames from an MPEG video file into TIFF Images: 

ffmpeg -i inputstream.mp4 -vf select=’eq(pict_type\,I)’ -vsync drop -f image2 -pix_fmt rgb24 C:/folder/

video/framesfolder/frame%05d. ff

Another Video Filter has been used here to select only the frames with a Picture Type of ‘I’. This is followed 

by ‐vsync drop to not include frames with a duplicate  me code. The output has been set to a RGB24 TIFF 

Image format.