I recently received a 20 minute MP4 file were recovery of both video and sound was required. However, the file had not been finalised so there was no meta data. The file contained a header, 'ftyp' and the 'mdat' segement, but no 'moov' data.
Interestingly, Windows media player would view the file - but video only. Quick time and VLC would not touch it.
To process the file I recquested, and received a small MP4 file. From this all the meta data in the moov atom could be read. It was then a case of reconstructing the atoms that describe the file. The most important ones are 'stsz', 'stss' and 'stco'. Each codec has different coding systems, but on this video, each video chunck started with the string 0x00 0x00 0x01 0xb3. Each audio chunck was a fixed length, and preceeded the video chunk. The difficult one to find was the video frame, which started with 0x00 0x00 0x01 0x00 - although easy to find, the possiblity of false positives required careful attention.
After a fair bit work and investigation it was possible to get a viewable video with sound, but approx halfway through the video became distorted. The problem turned out to be the 'stss' atom (sync sample atom). All chunks seemed to have a fixed number of frames, so the 'stss' array was straight incrementing numbers. However, at about the 2GB mark of the file one chunk had extra frames. This was reflected in both the 'stss' and 'stsz' atoms, but the video did not display correctly. After a lot of head scratching a workable solution was found by ignoring the extra frames.
Curiously, the sample MP4 would not play with QuickTime, or produce sound with Windows Media Player. It is possible that the originally created file was valid, but VLC does not like variable length 'stss' atom elements.
Overall, a very satisfactory solution as file was reconstructed with a totally missing moov section.
Monday, June 11, 2012
Tuesday, April 17, 2012
Multiple screens
I don't know why I didn't do this before - add a second screen to my main development PC (win7-64). The benefits are dramatic. On one screen I can have visual studio for program development, and on the second a hex dump of the file I am looking at. Or the second screen can be a web page of documentation, or anything else.
The result is that is now much less likely necessary to print anything out for reference purposes, and one can cut and paste beween the two screens.
The setup was easy, just aquire a suitable monitor, and in my case a new lead to connect to the video card my PC. Configuration was through the control panel and the only hiccup was trying to get the second monitor to be logically (as well as physically) on the left of my desk. The tip is that the control panel will display two monitors with a big 1 and 2 on them. These icons can be dragged so that the main and second monitor are as required.
The result is that is now much less likely necessary to print anything out for reference purposes, and one can cut and paste beween the two screens.
The setup was easy, just aquire a suitable monitor, and in my case a new lead to connect to the video card my PC. Configuration was through the control panel and the only hiccup was trying to get the second monitor to be logically (as well as physically) on the left of my desk. The tip is that the control panel will display two monitors with a big 1 and 2 on them. These icons can be dragged so that the main and second monitor are as required.
Sunday, February 19, 2012
PSR
It has been known (although I would like to think very rarely) for my software to crash. As the user could be anywhere, it normally means several e-mails to track down what they have been doing, upto the point of failing.
I have just discovered a very useful Microsoft feature - part of Windows 7 - that may help considerably in the future. It is called PSR - and can be launched from the command prompt, or Windows start prompt. It will very simply record screens and produce a sensible size file that can be e-mailed for investigation. The user can see what is being sent so there can be no question of possible spying. There is also no requirement to load software on the user PC, or change security settings (beyond setting administrator rights for the PSR to run).
Credit for this information should really go to PC-PRO magazine in UK, April 2012 - thanks.
I have just discovered a very useful Microsoft feature - part of Windows 7 - that may help considerably in the future. It is called PSR - and can be launched from the command prompt, or Windows start prompt. It will very simply record screens and produce a sensible size file that can be e-mailed for investigation. The user can see what is being sent so there can be no question of possible spying. There is also no requirement to load software on the user PC, or change security settings (beyond setting administrator rights for the PSR to run).
Credit for this information should really go to PC-PRO magazine in UK, April 2012 - thanks.
Friday, February 10, 2012
4K sector size
I am sure we have all read about hard drives with sector sizes greater than 512 bytes (0x200). This week I saw my first one. A 1TB Iomega drive, with USB 3.0 connection. The customer had lost their data, and sure enough when plugged into a Mac, there was nothing.
Looking at the disk, the first surprise was the sector size was 4096 (0x1000) bytes. On first loading, my software did not like it but fortunately the fix was very quick to implement. The software was expecting the boot sector signature of 0x55 0xAA to be at the end of the sector, rather than the end of the first 0x200 area. By fixing this small problen, CnW recovery read the disk without problems. Logically, the Mac disk is still based on 0x200 byte sectors so the program has a simple read function that translates a logical sector number into a disk logical block address (LBA) and the extracts the sector.
Be warned, more of these disks will be on their way.
Looking at the disk, the first surprise was the sector size was 4096 (0x1000) bytes. On first loading, my software did not like it but fortunately the fix was very quick to implement. The software was expecting the boot sector signature of 0x55 0xAA to be at the end of the sector, rather than the end of the first 0x200 area. By fixing this small problen, CnW recovery read the disk without problems. Logically, the Mac disk is still based on 0x200 byte sectors so the program has a simple read function that translates a logical sector number into a disk logical block address (LBA) and the extracts the sector.
Be warned, more of these disks will be on their way.
Tuesday, January 31, 2012
Not enough server storage is available to process this command
I recently upgraded an XP system to have a larger hard drive for data storage. The new drive is a 1.5TB drive, replacing the previous 500GB drive. The PC is about 5 years old but still works well, and has 3GB of RAM.
On a recent disk imaging process, it came up with errors writing to the drive. It indicated not enough resources. I tried a reboot (to clear down any untidy memory) but errors kept appearing. I then tried writing from a different machine via the network. This displayed the error message
"Not enough server storage is available to process this command". As a second test I tried to copy a 350GB to the same drive. After seconds, this Windows only command failed.
I determined that the error being returned was m_loSError 1130, or 0x46a
The next stage was Google with this error message and sure enough I have not been the first to receive it. Also, very nicely there is a fix by changing a registry setting.
http://support.microsoft.com/kb/304101
Subsequent testing shows that the system is now stable, and I can copy a 350GB file to the same disk drive. As a spin off, I have improved the error reporting of my software to assist with other users hitting he same problem.
On a recent disk imaging process, it came up with errors writing to the drive. It indicated not enough resources. I tried a reboot (to clear down any untidy memory) but errors kept appearing. I then tried writing from a different machine via the network. This displayed the error message
"Not enough server storage is available to process this command". As a second test I tried to copy a 350GB to the same drive. After seconds, this Windows only command failed.
I determined that the error being returned was m_loSError 1130, or 0x46a
The next stage was Google with this error message and sure enough I have not been the first to receive it. Also, very nicely there is a fix by changing a registry setting.
http://support.microsoft.com/kb/304101
Subsequent testing shows that the system is now stable, and I can copy a 350GB file to the same disk drive. As a spin off, I have improved the error reporting of my software to assist with other users hitting he same problem.
Wednesday, January 18, 2012
How to read XFS on Windows
I often read about problems people have reading Unix formats on a PC. Linux has many features that allow a new file system to be loaded and hence read disks in non Ext/2/3/4 format.
One solution for a PC is to use CnW to just read the data files from the foreign file system. 95% of the time this will be a satisfactory solution, though there will be occasions were read and write are required. CnW does not write to non PC file systems.
By using a copy of CnW Recovery software files can be read from many Unix disks, including XFS and Reiser. It also included Macintosh. The demo will read the files, but not actually save any.
CnW is designed as a data recovery tool, rather than this general reading tool, so file selection is limited, for instance there is no drag and drop. However, there are extensive file filter tools so file of a certain date, size or name/location may be selected or skipped.
When presented with a problem of accessing data from a foreign drive, CnW may well provide a simple solution. As a bonus deleted files can also be found.
One solution for a PC is to use CnW to just read the data files from the foreign file system. 95% of the time this will be a satisfactory solution, though there will be occasions were read and write are required. CnW does not write to non PC file systems.
By using a copy of CnW Recovery software files can be read from many Unix disks, including XFS and Reiser. It also included Macintosh. The demo will read the files, but not actually save any.
CnW is designed as a data recovery tool, rather than this general reading tool, so file selection is limited, for instance there is no drag and drop. However, there are extensive file filter tools so file of a certain date, size or name/location may be selected or skipped.
When presented with a problem of accessing data from a foreign drive, CnW may well provide a simple solution. As a bonus deleted files can also be found.
Thursday, January 12, 2012
Ubuntu disk formatting
Recently I was using Ubuntu V11.10 to create a sample Unix disk. I am no greate fan of Linux, but I have to say that V11 is getting close to a system that could be used by 'normal' users and not just geeks that love long command lines with millions of options. On the negative side it does seem to be influenced by the Macintosh.
However, one currious problem I ran into was with formatting a disk. I wanted to reformat an old Windows disk into Unix. The old disk was a valid disk, so had a partition identifier set in the boot sector, and naturally it was set as 7 for NTFS. I formated the disk, copied some files over and went to use the disk. Rather oddly, it was recognised as an NTFS disk - but corrupted. I discovered this was because the partition ID was still 7, rather than 0x83 I would have expected for Reiser.
The solution was two fold - it is posible in Ubuntu to edit the partition info, and in CnW it is possible to select the correct type of disk.
The warning that I doubt I am the first to create this problem and it is possible that a corrupt NTFS disk is infact a Unix disk. Be aware!
However, one currious problem I ran into was with formatting a disk. I wanted to reformat an old Windows disk into Unix. The old disk was a valid disk, so had a partition identifier set in the boot sector, and naturally it was set as 7 for NTFS. I formated the disk, copied some files over and went to use the disk. Rather oddly, it was recognised as an NTFS disk - but corrupted. I discovered this was because the partition ID was still 7, rather than 0x83 I would have expected for Reiser.
The solution was two fold - it is posible in Ubuntu to edit the partition info, and in CnW it is possible to select the correct type of disk.
The warning that I doubt I am the first to create this problem and it is possible that a corrupt NTFS disk is infact a Unix disk. Be aware!
Saturday, December 31, 2011
Reiser and program development
Software development can be a slow process, but in many respects that is the best approach. But this slow process should ideally be a series of small developments, each tested and reviewed on a continuous basis.
An example is the Reiser handler for CnW Recovery. This was written in response to a an internal job to recover files from a failed HP Media Vault device. The disks were OK, but the RAID had failed. The development required was both support for JBOD (Job bunch of disks) and the complex Reiser format. The job was done, and all data recovered. ie Version 1 worked.
The next requirement came to read deleted files. Deleted files on Reiser, according to many Internet sites is not possible, so CnW developed a method that works to about 80% success, a figure that varies and depends on many factors.
When software is released, there are often problems as unseen configurations, or failures crop up. One by one these are fixed, and the product improves. Currently, there is a good engine for HP Mediavault recovery, and many successful recoveries have been performed.
Along with a good engine, an easy to use interface has to be developed, and here the problems can expand. Users can be extremely computer literate, or beginners. Most users are not experts in data recovery and so there are many terms and concepts unfamiliar to them. A good place to start is with the Help system, but not many users think they need this. There is also the expectation that because one can turn a computer on, one is an expert in all fields of computing, including data recovery.
One solution to this problem is to give no options, and only a single path through the software. For predictable failures, and known configurations this can be implemented, and an example in CnW is the wizard for failed mini DVDs, to produce a new video disk. For devices such as a HP RAID, there are several variations, and different requirements. To this end, the chosen approach has been to try and trap errors, and give possible solutions. At the same time, intelligence is built into the program to try and determine the original configuration, and point to a solution.
With HP Raid disks many users do not know if the disk is a mirrored disk, or a pair of disks. Routines can be built into the program to indicate that the single disk is probably the second of a pair, or that an extra disk is likely to be required, and configured as a JBOD.
The overall result of this development is a program that will result in fewer calls for support, and a quicker solution for users who do not want to read the manual, but just get a solution.
An example is the Reiser handler for CnW Recovery. This was written in response to a an internal job to recover files from a failed HP Media Vault device. The disks were OK, but the RAID had failed. The development required was both support for JBOD (Job bunch of disks) and the complex Reiser format. The job was done, and all data recovered. ie Version 1 worked.
The next requirement came to read deleted files. Deleted files on Reiser, according to many Internet sites is not possible, so CnW developed a method that works to about 80% success, a figure that varies and depends on many factors.
When software is released, there are often problems as unseen configurations, or failures crop up. One by one these are fixed, and the product improves. Currently, there is a good engine for HP Mediavault recovery, and many successful recoveries have been performed.
Along with a good engine, an easy to use interface has to be developed, and here the problems can expand. Users can be extremely computer literate, or beginners. Most users are not experts in data recovery and so there are many terms and concepts unfamiliar to them. A good place to start is with the Help system, but not many users think they need this. There is also the expectation that because one can turn a computer on, one is an expert in all fields of computing, including data recovery.
One solution to this problem is to give no options, and only a single path through the software. For predictable failures, and known configurations this can be implemented, and an example in CnW is the wizard for failed mini DVDs, to produce a new video disk. For devices such as a HP RAID, there are several variations, and different requirements. To this end, the chosen approach has been to try and trap errors, and give possible solutions. At the same time, intelligence is built into the program to try and determine the original configuration, and point to a solution.
With HP Raid disks many users do not know if the disk is a mirrored disk, or a pair of disks. Routines can be built into the program to indicate that the single disk is probably the second of a pair, or that an extra disk is likely to be required, and configured as a JBOD.
The overall result of this development is a program that will result in fewer calls for support, and a quicker solution for users who do not want to read the manual, but just get a solution.
Monday, November 28, 2011
Fake software
I recently needed to purchase another copy of Microsoft Office 2007 for a new laptop. Being fairly expensive, I searched Amazon for a deal. As the product has now been replaced with 2010 I hope to get some old stock. I found a copy at about £175 and purchased.
I received the package very quickly, but the CD was warped, and would not load. Having already got a package, I decided to load my old (and I think idenical CD) but use the new product code (registration number) to licence the software. However, the product code was not valid. After a very short correspondence with the supplier, I returned the package and received a full refund.
Attempt number 2 was no better. This time, eBay. The CD was good, product code worked when installing but when I came to online registration, I was told the code was a contfeit one. Again, the supply has promised to refund my purchase, and I have every confidence that this will happen.
My third attempt is as I have to pay the full cost, I have now purchased Office 2010 (just over £300) and am waiting for delivery.
One area that has surprised me is that both, probably fake items looked 100% genuine. I compared every thing with with my original genuine item and the only difference was manufacture in Ireland rather than Singapore. The full refund did eventually arrive.
Moral, you get what you pay for - but office professional is a bit expensive these days.
I received the package very quickly, but the CD was warped, and would not load. Having already got a package, I decided to load my old (and I think idenical CD) but use the new product code (registration number) to licence the software. However, the product code was not valid. After a very short correspondence with the supplier, I returned the package and received a full refund.
Attempt number 2 was no better. This time, eBay. The CD was good, product code worked when installing but when I came to online registration, I was told the code was a contfeit one. Again, the supply has promised to refund my purchase, and I have every confidence that this will happen.
My third attempt is as I have to pay the full cost, I have now purchased Office 2010 (just over £300) and am waiting for delivery.
One area that has surprised me is that both, probably fake items looked 100% genuine. I compared every thing with with my original genuine item and the only difference was manufacture in Ireland rather than Singapore. The full refund did eventually arrive.
Moral, you get what you pay for - but office professional is a bit expensive these days.
Saturday, November 26, 2011
More fragmented deleted video files
I came across another variation of ways that video files are created on a camera. A Quick time video (eg .mov, .3gp, .mp4) has three main sections, a 'ftyp' 'moov' and 'mdat'. The mdat is the video data, and 'moov' the indexing information. The 'mdat' can be large, ie 100s of MBs, while the moov may only be 10s of KBs or a few MBs. The 'moov segement is a variable size and can only be created when the complete 'mdat' has been recorded, which can make logical storage difficult when the final sequence of 'ftyp'-'moov'-'mdat' is required. The way this can be overcome in a camera is to record the 'ftyp' as a cluster, then all of the 'mdat' as complete clusters, and then the 'moov' also as complete clusters. By modifying the FAT on a FAT32 disk, the logical sequence can be made as 'ftyp'-'moov'-'mdat'.
When the files are deleted (maybe by accident) the logical sequence information is lost making recovery by data carving impossible. CnW Recovery have developed routines to detect this type of fragmentation and hence recover otherwise fragmented files. A previous blog discussed a similar problem, but on that occasion the 'ftyp' and 'moov' segements were joined together, and not in separate clusters.
When the files are deleted (maybe by accident) the logical sequence information is lost making recovery by data carving impossible. CnW Recovery have developed routines to detect this type of fragmentation and hence recover otherwise fragmented files. A previous blog discussed a similar problem, but on that occasion the 'ftyp' and 'moov' segements were joined together, and not in separate clusters.
Subscribe to:
Posts (Atom)