Sony Video Camera PMW-F3 deleted MP4 recovery

A major and on going CnW Recovery development is processing fragmented video files.  In particular these are ones that have been deleted on the camera.  When a FAT32 file is deleted, the allocation table is cleared down and so the order of clusters is lost.  For may applications, this does not matter as the file is sequential and so can be recovered - and this is often true for photos.  Videos however tend to be long and an MP4 /3GP / MOV file has three main sections (or atoms).

• A short 'ftyp' header
• A mid size 'moov' that stores pointers
• A big 'mdat' that stores the audio and video

Logically the sequence is normally either ftyp-moov-mdat  or ftyp-mdat-moov.

When recording it is impossible to know the length of the moov or mdat atom.  For this reason the mdat is stored on the memory chip, and often the moov is stored in the camera RAM until the end.  On finalisation it is then written to the memory chip.  If the required sequence is ftyp-moov-mdata some 'clever' fiddling of the FAT is performed by the camera to make the file logically sequential.

The CnW program has been developed to handle the above for many video types but then  the Sony PMW-F3 format was found.  The big difference this time is that the 'mdat' is stored in numerous chunks, and not always in sequence on the memory chip.  The challenge that CnW is working on is to find these chunks and reconstruct the video.  This is performed in several stages

• Chip is scanned for all ftyp, mdat and moov headers
• Chip is scanned for all MP4A audio clusters
• If required a fragmented moov is reconstructed
• The video and audio frames are located based on offset within a cluster.  Special routines are required when there are multiple audio or video frames stored at the same location within a cluster
• Often the frame pointers point to clusters several clusters later than the previous one. In these cases the gap between the known cluster locations has be filled in by working forward and backwards from known good locations

The current results are very reasonable, but still more tweaking is required.  However one customer reported that CnW recovery does recover viewable video, and no other program got anywhere near this. 

CnW is very sceptical of many adverts that claim video recovery - it may work from a hard disk, but CnW has major doubts about working form camera memory chips

GoPro Hero 3 recovery of deleted files

I have often said of data structures, 'if it can be done, it will be done'.  ie almost anything is possible, and so expect many variations.  In the words of the 'Hitch Hikers Guide to the Universe' - Expect the unexpected.

A recent view of a GoPro Hero 3 camera confirmed the above.  As I have written in the past, video camera often record the data not in a different physical to logical sequence.  The slight of hand is that the FAT defines the logical sequence.  Thus many camera record the video data physically earlier than the file start sector.  On the GoPro Hero 3 camera another twist has been discovered.  Two video streams can be recorded at the same time - a high resolution, and a smaller low resolution.  The physical sequence on the disk could be as below



Just to complicate the above, there can also be jpeg thumbnails, and text status files inserted in the above stream.  Standard data carving is totally useless as nothing is in sequence.

The CnW MP4/3GP Wizard works hard, but will recover, with a high success rate videos that have been deleted.  www.cnwrecovery.com/html/mp4_wizard.html

More fragmented deleted video files

I came across another variation of ways that video files are created on a camera.  A Quick time video (eg .mov, .3gp, .mp4) has three main sections, a 'ftyp' 'moov' and 'mdat'.  The mdat is the video data, and 'moov' the indexing information.  The 'mdat' can be large, ie 100s of MBs, while the moov may only be 10s of KBs or a few MBs.  The 'moov segement is a variable size and can only be created when the complete 'mdat' has been recorded, which can make logical storage difficult when the final sequence of 'ftyp'-'moov'-'mdat' is required.  The way this can be overcome in a camera is to record the 'ftyp' as a cluster, then all of the 'mdat' as complete clusters, and then the 'moov' also as complete clusters.  By modifying the FAT on a FAT32 disk, the logical sequence can be made as 'ftyp'-'moov'-'mdat'.

When the files are deleted (maybe by accident) the logical sequence information is lost making recovery by data carving impossible.  CnW Recovery have developed routines to detect this type of fragmentation and hence recover otherwise fragmented files.  A previous blog discussed a similar problem, but on that occasion the 'ftyp' and 'moov' segements were joined together, and not in separate clusters.