Recover deleted MP4 files from Sony PMW-F3 update

My last post reported good progress on recovery of Sony PMW-F3 files.  I am now pleased to report some extremely good results.

The final approach involved scanning the complete disk for possible audio and video elements.  As this is fairly machine intensive it was very satisfying from the programming point of view to make use of parallel processing.  While one cluster was being read from the physical device, the previous cluster was examined for audio data, and in parallel, video data.

Once the disk has been scanned it is then quick to isolate any pre scanned video structure with it's location within a cluster.  On a small memory device one is normally luck that each structure will have a unique offset.  For large memory device there will be duplicate entries.  In these cases the physical location of the matching cluster will be used to determine the correct one.

When a video file is deleted it is normal for the data to remain, although may well be non sequential.  The critical section to find is the moov atom.  This contains pointers to frame starts and audio buffers.  By tying up the moov pointers with the physical clusters a valid file can be reconstructed.

Simple?  In some respects yes, but recognising patterns is not a simple computer task.  It is very much an exercise in fitting together the best matches.  It is a bit like doing a jigsaw that has no picture.  When it works, the results are fantastic.

Many videos are recorded in two sections, the Sony sometimes uses over 100 sections, so joining these together is a great success.

Sony Video Camera PMW-F3 deleted MP4 recovery

A major and on going CnW Recovery development is processing fragmented video files.  In particular these are ones that have been deleted on the camera.  When a FAT32 file is deleted, the allocation table is cleared down and so the order of clusters is lost.  For may applications, this does not matter as the file is sequential and so can be recovered - and this is often true for photos.  Videos however tend to be long and an MP4 /3GP / MOV file has three main sections (or atoms).

• A short 'ftyp' header
• A mid size 'moov' that stores pointers
• A big 'mdat' that stores the audio and video

Logically the sequence is normally either ftyp-moov-mdat  or ftyp-mdat-moov.

When recording it is impossible to know the length of the moov or mdat atom.  For this reason the mdat is stored on the memory chip, and often the moov is stored in the camera RAM until the end.  On finalisation it is then written to the memory chip.  If the required sequence is ftyp-moov-mdata some 'clever' fiddling of the FAT is performed by the camera to make the file logically sequential.

The CnW program has been developed to handle the above for many video types but then  the Sony PMW-F3 format was found.  The big difference this time is that the 'mdat' is stored in numerous chunks, and not always in sequence on the memory chip.  The challenge that CnW is working on is to find these chunks and reconstruct the video.  This is performed in several stages

• Chip is scanned for all ftyp, mdat and moov headers
• Chip is scanned for all MP4A audio clusters
• If required a fragmented moov is reconstructed
• The video and audio frames are located based on offset within a cluster.  Special routines are required when there are multiple audio or video frames stored at the same location within a cluster
• Often the frame pointers point to clusters several clusters later than the previous one. In these cases the gap between the known cluster locations has be filled in by working forward and backwards from known good locations

The current results are very reasonable, but still more tweaking is required.  However one customer reported that CnW recovery does recover viewable video, and no other program got anywhere near this. 

CnW is very sceptical of many adverts that claim video recovery - it may work from a hard disk, but CnW has major doubts about working form camera memory chips