My last post reported good progress on recovery of Sony PMW-F3 files. I am now pleased to report some extremely good results.
The final approach involved scanning the complete disk for possible audio and video elements. As this is fairly machine intensive it was very satisfying from the programming point of view to make use of parallel processing. While one cluster was being read from the physical device, the previous cluster was examined for audio data, and in parallel, video data.
Once the disk has been scanned it is then quick to isolate any pre scanned video structure with it's location within a cluster. On a small memory device one is normally luck that each structure will have a unique offset. For large memory device there will be duplicate entries. In these cases the physical location of the matching cluster will be used to determine the correct one.
When a video file is deleted it is normal for the data to remain, although may well be non sequential. The critical section to find is the moov atom. This contains pointers to frame starts and audio buffers. By tying up the moov pointers with the physical clusters a valid file can be reconstructed.
Simple? In some respects yes, but recognising patterns is not a simple computer task. It is very much an exercise in fitting together the best matches. It is a bit like doing a jigsaw that has no picture. When it works, the results are fantastic.
Many videos are recorded in two sections, the Sony sometimes uses over 100 sections, so joining these together is a great success.
Sunday, December 29, 2013
Sunday, December 1, 2013
Sony Video Camera PMW-F3 deleted MP4 recovery
A major and on going CnW Recovery development is processing fragmented video files. In particular these are ones that have been deleted on the camera. When a FAT32 file is deleted, the allocation table is cleared down and so the order of clusters is lost. For may applications, this does not matter as the file is sequential and so can be recovered - and this is often true for photos. Videos however tend to be long and an MP4 /3GP / MOV file has three main sections (or atoms).
When recording it is impossible to know the length of the moov or mdat atom. For this reason the mdat is stored on the memory chip, and often the moov is stored in the camera RAM until the end. On finalisation it is then written to the memory chip. If the required sequence is ftyp-moov-mdata some 'clever' fiddling of the FAT is performed by the camera to make the file logically sequential.
The CnW program has been developed to handle the above for many video types but then the Sony PMW-F3 format was found. The big difference this time is that the 'mdat' is stored in numerous chunks, and not always in sequence on the memory chip. The challenge that CnW is working on is to find these chunks and reconstruct the video. This is performed in several stages
CnW is very sceptical of many adverts that claim video recovery - it may work from a hard disk, but CnW has major doubts about working form camera memory chips
- A short 'ftyp' header
- A mid size 'moov' that stores pointers
- A big 'mdat' that stores the audio and video
When recording it is impossible to know the length of the moov or mdat atom. For this reason the mdat is stored on the memory chip, and often the moov is stored in the camera RAM until the end. On finalisation it is then written to the memory chip. If the required sequence is ftyp-moov-mdata some 'clever' fiddling of the FAT is performed by the camera to make the file logically sequential.
The CnW program has been developed to handle the above for many video types but then the Sony PMW-F3 format was found. The big difference this time is that the 'mdat' is stored in numerous chunks, and not always in sequence on the memory chip. The challenge that CnW is working on is to find these chunks and reconstruct the video. This is performed in several stages
- Chip is scanned for all ftyp, mdat and moov headers
- Chip is scanned for all MP4A audio clusters
- If required a fragmented moov is reconstructed
- The video and audio frames are located based on offset within a cluster. Special routines are required when there are multiple audio or video frames stored at the same location within a cluster
- Often the frame pointers point to clusters several clusters later than the previous one. In these cases the gap between the known cluster locations has be filled in by working forward and backwards from known good locations
CnW is very sceptical of many adverts that claim video recovery - it may work from a hard disk, but CnW has major doubts about working form camera memory chips
Monday, September 16, 2013
GoPro Hero 3 recovery of deleted files
I have often said of data structures, 'if it can be done, it will be done'. ie almost anything is possible, and so expect many variations. In the words of the 'Hitch Hikers Guide to the Universe' - Expect the unexpected.
A recent view of a GoPro Hero 3 camera confirmed the above. As I have written in the past, video camera often record the data not in a different physical to logical sequence. The slight of hand is that the FAT defines the logical sequence. Thus many camera record the video data physically earlier than the file start sector. On the GoPro Hero 3 camera another twist has been discovered. Two video streams can be recorded at the same time - a high resolution, and a smaller low resolution. The physical sequence on the disk could be as below
Just to complicate the above, there can also be jpeg thumbnails, and text status files inserted in the above stream. Standard data carving is totally useless as nothing is in sequence.
The CnW MP4/3GP Wizard works hard, but will recover, with a high success rate videos that have been deleted. www.cnwrecovery.com/html/mp4_wizard.html
A recent view of a GoPro Hero 3 camera confirmed the above. As I have written in the past, video camera often record the data not in a different physical to logical sequence. The slight of hand is that the FAT defines the logical sequence. Thus many camera record the video data physically earlier than the file start sector. On the GoPro Hero 3 camera another twist has been discovered. Two video streams can be recorded at the same time - a high resolution, and a smaller low resolution. The physical sequence on the disk could be as below
Just to complicate the above, there can also be jpeg thumbnails, and text status files inserted in the above stream. Standard data carving is totally useless as nothing is in sequence.
The CnW MP4/3GP Wizard works hard, but will recover, with a high success rate videos that have been deleted. www.cnwrecovery.com/html/mp4_wizard.html
Tuesday, September 3, 2013
Windows 8
It was time for a new PC and so Windows 8 - 64 seems the obvious option in order to keep up to date. I ordered reasonable spec system (3.6GHz, Core 7, 12GB RAM) and a week or so later it arrived.
I have had Windows 8-32 since it came out, but not as a main PC. This post will describe some of the issues I've had, and some solutions.
My work (data recovery) requires lots of disk space. I keep the main 2TB system drive as the system and development drive. All customer data is stored on other drives - or a new 9TB NAS RAID. My first job was to add a second 3TB drive to the PC. This was a reasonable quick job with the screwdriver, and the drive was seen in the BIOS. However, what ever I did, I could not see it in Windows 8. Google to the rescue and it turns out that Windows 8, being much more secure will not recognise extra hardware (though it could see USB drives). The solution was a BIOS option switch to remove the 'Secure boot' option. Drive now seen, and the secure boot can now be re-enabled.
The next problem was Norton antivirus. The PC came with McAfee, but the rest of my systems all have Norton, on a group licence. I uninstall McAfee and installed Norton 360 but then problems began. Various problems, mainly IE10 and other internet related products had issues. Worse still, each time I rebooted the PC, Norton stopped working and often came up with the error 8504,101. Google had several ideas for this, including running NPE.exe, uninstall and re-install. I tried many of these suggestions, but none worked. The error message is common on Google posts, and so I installed AVG and started to get some work done. Even the Beta version of N360 had the same problem.
On hardware, I have become a complete fan of multiple screens. The new PC has 3 video outputs, but 2 of them are HDMI, and I only have (old) VGA screens. I hope the eBay adaptor will arrive soon and let me work with 2, or maybe 3 screens at once.
A final issue I had was over a security dongle. The drive is not signed, and so Windows 8 will not load it. The solution was the result of another search and involves booting Windows in a mode to accept unsigned drivers. The instructions (with due credit to someone else) are as follows
4. System will restart to a Choose an option screen
5. Select Troubleshoot from Choose an option screen
6. Select Advanced options from Troubleshoot screen
7. Select Windows Startup Settings from Advanced options screen
8. Click Restart button
9. System will restart to Advanced Boot Options screen
10. Select Disable Driver Signature Enforcement
11. Once the system starts, install the drivers as normal
Conclusion
Looking forward to Windows 8.1 where I hope the metro screen will be less important, though I will start playing with some of the apps.
I have had Windows 8-32 since it came out, but not as a main PC. This post will describe some of the issues I've had, and some solutions.
My work (data recovery) requires lots of disk space. I keep the main 2TB system drive as the system and development drive. All customer data is stored on other drives - or a new 9TB NAS RAID. My first job was to add a second 3TB drive to the PC. This was a reasonable quick job with the screwdriver, and the drive was seen in the BIOS. However, what ever I did, I could not see it in Windows 8. Google to the rescue and it turns out that Windows 8, being much more secure will not recognise extra hardware (though it could see USB drives). The solution was a BIOS option switch to remove the 'Secure boot' option. Drive now seen, and the secure boot can now be re-enabled.
The next problem was Norton antivirus. The PC came with McAfee, but the rest of my systems all have Norton, on a group licence. I uninstall McAfee and installed Norton 360 but then problems began. Various problems, mainly IE10 and other internet related products had issues. Worse still, each time I rebooted the PC, Norton stopped working and often came up with the error 8504,101. Google had several ideas for this, including running NPE.exe, uninstall and re-install. I tried many of these suggestions, but none worked. The error message is common on Google posts, and so I installed AVG and started to get some work done. Even the Beta version of N360 had the same problem.
On hardware, I have become a complete fan of multiple screens. The new PC has 3 video outputs, but 2 of them are HDMI, and I only have (old) VGA screens. I hope the eBay adaptor will arrive soon and let me work with 2, or maybe 3 screens at once.
A final issue I had was over a security dongle. The drive is not signed, and so Windows 8 will not load it. The solution was the result of another search and involves booting Windows in a mode to accept unsigned drivers. The instructions (with due credit to someone else) are as follows
1. Windows Key + R
2. Enter shutdown.exe /r /o /f /t 00
3. Click the OK button
2. Enter shutdown.exe /r /o /f /t 00
3. Click the OK button
System reboots here
4. System will restart to a Choose an option screen
5. Select Troubleshoot from Choose an option screen
6. Select Advanced options from Troubleshoot screen
7. Select Windows Startup Settings from Advanced options screen
8. Click Restart button
9. System will restart to Advanced Boot Options screen
10. Select Disable Driver Signature Enforcement
11. Once the system starts, install the drivers as normal
Looking forward to Windows 8.1 where I hope the metro screen will be less important, though I will start playing with some of the apps.
Saturday, January 5, 2013
Encrypted drives
I was recently helping a potential customer with a data recovery problem. The problem was a Western Digital external drive (1TB) that had a damaged USB connector on the case. The drive was removed and placed in a USB caddy (for common practise). The drive appears physically OK, but very few files could be read.
A few scans and logs were transfered and it appears that there were files at the end of the disk, but the middle area was almost totally blank, ie very few file signatures recognised. The next stage was a a disk scan (a CnW Forensic feature) which scans the complete drive and shows the broad category of data in each sector. This includes text, blank, directory entry and compressed. Most of the middle was deteceted as compressed. A compressed sector (in this logic) is one with many different byte values in the sector and will detect Zip files, JPEGs, MPEGS, music files as well as encrypted data.
The customer had not used encryption, or a program such as TrueCrypt so the results did not make much sense. However, the more I thought about it, the more the data looked as the disk was compressed.
A bit of Google research did reveal something I was not expecting. WD do make external drives with built in compression, controlled by the internal controller board. This is enabled, even if a password is not entered and could explain the situation with this drive. The only solution is to read the drive with the original controller board. Otherwise, to read the encrypted data can vary between very difficult to impossible.
This configuration came a surprise for me, and I must now be aware that the dirve case may be important, and not just accept the naked drive.
A few scans and logs were transfered and it appears that there were files at the end of the disk, but the middle area was almost totally blank, ie very few file signatures recognised. The next stage was a a disk scan (a CnW Forensic feature) which scans the complete drive and shows the broad category of data in each sector. This includes text, blank, directory entry and compressed. Most of the middle was deteceted as compressed. A compressed sector (in this logic) is one with many different byte values in the sector and will detect Zip files, JPEGs, MPEGS, music files as well as encrypted data.
The customer had not used encryption, or a program such as TrueCrypt so the results did not make much sense. However, the more I thought about it, the more the data looked as the disk was compressed.
A bit of Google research did reveal something I was not expecting. WD do make external drives with built in compression, controlled by the internal controller board. This is enabled, even if a password is not entered and could explain the situation with this drive. The only solution is to read the drive with the original controller board. Otherwise, to read the encrypted data can vary between very difficult to impossible.
This configuration came a surprise for me, and I must now be aware that the dirve case may be important, and not just accept the naked drive.
Subscribe to:
Posts (Atom)