Data Stories

Sony PXW-Z190, ffmpeg batch transcode to ProRes Quicktime: Explained

Specific guide on using ffmpeg to transcode Quicktime (ProRes) from the native MXF as recorded by Sony PXW-Z190 cameras. Also of interest to users of the PXW-Z280. For those of you desperate to cut and paste an incantation into Terminal – here you go:

 for i in *.MXF; do ffmpeg -i "$i"   -map 0:0 -map 0:1 -map 0:2 -map 0:3 -map 0:4 -c:v prores_ks -profile:v 1 -quant_mat:v 3 -qscale:v 13   "./output/${i%.*}.mov";done

Read on to understand what it does and why it works. If you’re not technical or used to Terminal – don’t worry we do it step by step.

So for the transcode we want to operate on the video stream particularly which is in h264, and transcode that to ProRes. We could do things to the audio stream but here we are just going to copy the four tracks of audio straight over.

Video transcode

Using ffprobe find the information about the zeroth stream which in this camera is the video…

Stream #0:0: Video: h264 (High), yuv420p(tv, bt709, progressive), 3840x2160 [SAR 1:1 DAR 16:9], 29.97 fps, 29.97 tbr, 29.97 tbn, 59.94 tbc    

So lets write a ffmpeg command to just make a QuickTime video file with no audio to illustrate what we are about.

ffmpeg -i INPUT.MXF -c:v prores_ks -profile:v 1 -quant_mat:v 3 -qscale 20  OUTPUT.MOV    

Exploding the command term by term

  • -c:v Selects the zeroth stream. If your video is somewhere else or there are n video streams then you need to map it / them formally
  • prores_ks selects the codec – This is one of number of ProRes options but is the one that I’ve had most luck with. Here’s a slightly despairing blog by one of the original authors: ProRes KS
  • -profile:v a preconfigured profile in prores_ks options are specified by an integer thus: -profile:v 3
    • 0 ‘proxy’
    • 1 ‘lt’ (suprisingly useful especially when output to web)
    • 2 ‘standard’
    • 3 ‘hq’
    • 4 ‘4444’
    • 5 ‘4444xq’
  • -quant_mat a preconfigured matrixes in prores_ks options are specified by an integer thus: -profile:v 3
    • 0‘auto’
    • 1‘default’
    • 2‘proxy’
    • 3‘lt’
    • 4 ‘standard’
    • 5 ‘hq’
  • -qscale Quantiser. Very broadly this is about the way the encoder picks the amount of compression per frame. Setting a fixed qscale speeds up the encode because a whole chunk of processing to get best quality is short-circuited.
  • bits_per_mb higher values will improve speed. I don’t recommend specifying both this and -qscaleat the same time because predicting the outcome gets tricky if you aren’t intimate with the maths

Audio copy

ffmpeg -i INPUT.MXF -map 0:1 -map 0:2 -map 0:3 -map 0:4 -c:a copy  OUTPUT.MOV    

Creating a straight copy over of the audio as PCM – i.e. without transcoding needs the use of the map command. There’s a better explanation of what’s going on here than I can give ffmpeg wiki some systems may struggle with the raw audio.

 Synthesising the command

ffmpeg -i INPUT.MXF -map 0:1 -map 0:2 -map 0:3 -map 0:4 -c:v prores_ks -profile:v 1  -quant_mat:v 3 -qscale:v 12  -c:a:0  pcm_alaw  -c:a:1  pcm_alaw -c:a:2 pcm_alaw -c:a:3 pcm_alaw

Note that we now need to map the video channel explicitly.

Now some magic to make I process a directory of files.

This works in OSX and probably works in most Linux. I’m not going to explain because why ‘$I’ works isn’t straightforward…

The only requirement here is that you probably should pre-create a directory called /output in the active directory. So if you’re working dir is
$usr/video then $usr/video/output

 for i in *.MXF; do ffmpeg -i "$i"   -map 0:0 -map 0:1 -map 0:2 -map 0:3 -map 0:4 -c:v prores_ks -profile:v 1 -quant_mat:v 3 -qscale:v 13  -c:a:0  pcm_alaw  -c:a:1  pcm_alaw -c:a:2 pcm_alaw -c:a:3 pcm_alaw "./output/${i%.*}.mov";done

Use your friend ffprobe

use ffmpeg without a solid understanding of the file you are feeding it as an input is fairly futile for all but the most straightforward cases. So first steps is always to use ffprobe to examine the input.

The bits we need from the screed of the Metadata report are the stream definitions – since they help with specifying the right options for the transcode.

 Video definition

 Stream #0:0: Video: h264 (High), yuv420p(tv, bt709, progressive), 3840x2160 [SAR 1:1 DAR 16:9], 29.97 fps, 29.97 tbr, 29.97 tbn, 59.94 tbc    

The first stream is the video – h264 encoded, colour depth 4:2:0,

N audio streams

 Stream #0:1: Audio: pcm_s24le, 48000 Hz, 1 channels, s32 (24 bit), 1152 kb/s

Then we get to the audio streams of which there are at least four – two external microphones, and two internal mics. I haven’t been able to test the Sony special interface

Mysterious fifth stream

There is a data stream which it isn’t quite clear what it does – It’s timecode related but I don’t do enough TC specific stuff to have worked out what.

Data Stories

Ordinands: disabled under-represented?

Only four percent of candidates sponsored for a BAP (the gateway process in selection for ordination) declare a disability. Relative to national proportions of disability in working age adults there seems to be massive under-representation in the selection process. Is pernicious, cumulative and hidden discrimination to blame, or does poor measurement and lack of focus by the Ministry Division bear the blame?

4% of those sponsored for a BAP declared a disability… 

Bishop Seeley (St Edmundsbury and Ipswich), as chair of the Ministry Council

In written questions at Synod on 20th February 2019 Mr Williams asked the Chair of the. Ministry Council about the proportion of candidates with disability at four points in the process of discernment surrounding ordination and appointment

  • Entering the vocations process for ordained ministry;
  • Being recommended for training;
  • Completing training; and
  • Receiving a stipendiary vs non-stipendiary title post?

His question is perceptive since since the potential for filtering as a function of the protected characteristics the candidate expresses during the discernment process could poses a discriminatory force on ordinations, and the published data does not suggest the issue has high visibility for the Church of England.

A further necessary question Mr Williams neglects

Based on my personal experience of the Vocations system in the Church of England Mr Williams might also have useful added a fifth point at which the proportion of disabled candidates could usefully be recorded:

  • when their Vicar recommends them to meet with the DDO

This is necessary because this might in fact be the point at which the greatest effort needs to be focused on combatting discrimination.

 Response at Synod from Bishop Seeley

In a written response the Bishop Seeley (St Edmundsbury and Ipswich), as chair of the Ministry Council said

In the 2017/2018 academic year, 4% of those sponsored for a BAP declared a disability, the same percentage were recommended for training and a slightly higher percentage, 5%, sponsored for a stipendiary post.

I have no doubt that the answer given by Mr Seeley is accurate in the terms given. It is in fact those terms which causes most concern

Self declaration results in under identification of disabled candidates

Specifically the answer refers to candidates ‘declaring a disability’. In the 2014 report on Ordained Vocation Statistics it was clear that disability was being assessed based on ‘self declaration’ for reasonable adjustment at a panel. Assessing on this basis is known to result in under reporting, as the report acknowledges

As those declaring a disability are only doing so if it is relevant to BAP, the numbers are low and do not reflect the likely number of ordinands with a disability. This makes the data unreliable and indicates a need to take steps to improve the monitoring process.

The similarity in the 2014 and 2019 language ‘declaring a disability’ suggests a continued reliance on self declaration as a method for monitoring levels of disabled candidates in the discernment process. Relying on self declaration almost always fails to identify significant proportions of the disabled community. The reasons for under-declaration are complex but are thought to include applicants:

  • Believing their condition or disability has nothing to do with your ability to perform the role
  • not wanting an ‘employer’ to focus on the disability during a selection process
  • assuming that the ‘employer’ will expect you to have difficulties, need more time off, or support.

Achieving a realistic estimate

It’s almost certain that there is a problem with the statistics being reported, both in the 2014 report and Mr Seeley’s answer. The Papworth Trust’s disability stats reference 2018 suggests 18% of the working age population has a ‘disability’ of some sort.

 Do we know what pool the church is picking its ordinands from?

We don’t know what proportion of worshippers experience a disability. That matters, because we can’t assess whether the figures given in 2014 and by Mr Seeley are helpful without understanding the population the process is selecting from. Sadly discrimination in ‘right to access worship’ may mean that the pool the Church of England is choosing its ordinands from differs substantially in respect of type and severity of disability in respect to the wider population.

What should we want the answer to be?

I hope I worship in a church where there is a commitment to the poor, the excluded and the disabled. That care for, and responsibility to the ‘outsider’ is something that jumps out at me from the whole of scripture. So for the church to be finding that only four percent of those sponsored for a BAP declared a disability fills me with horror. Even if that answer is 100% below the actual, i.e. 8% of the BAP population is disabled, the proportion of candidates with a disability is still far below the proportion of disabled people in the working age population. Something is probably going very badly wrong

Is sufficient attention being paid to this by the Church of England?

Since it’s been known that the Church of England’s understanding of the potential problem of discrimination in it selection of candidates for ordination is incomplete since 2014 (nine years after the Disability Discrimination Act) passed, reasonable questions should be asked as to whether sufficient priority has been given to what is an important statistic.

Data Stories

Childhood obesity in the Diocese of Winchester

Obesity correlates with issues related issues the bible talks about as matters of justice and mercy. Unofficially for Church of England leaders in Winchester Diocese I’ve mapped childhood obesity prevalence in a way which allows an estimate of the situation in their parish. Click the picture to link to a larger file better formatted for printing