Software development can be a slow process, but in many respects that is the best approach. But this slow process should ideally be a series of small developments, each tested and reviewed on a continuous basis.
An example is the Reiser handler for CnW Recovery. This was written in response to a an internal job to recover files from a failed HP Media Vault device. The disks were OK, but the RAID had failed. The development required was both support for JBOD (Job bunch of disks) and the complex Reiser format. The job was done, and all data recovered. ie Version 1 worked.
The next requirement came to read deleted files. Deleted files on Reiser, according to many Internet sites is not possible, so CnW developed a method that works to about 80% success, a figure that varies and depends on many factors.
When software is released, there are often problems as unseen configurations, or failures crop up. One by one these are fixed, and the product improves. Currently, there is a good engine for HP Mediavault recovery, and many successful recoveries have been performed.
Along with a good engine, an easy to use interface has to be developed, and here the problems can expand. Users can be extremely computer literate, or beginners. Most users are not experts in data recovery and so there are many terms and concepts unfamiliar to them. A good place to start is with the Help system, but not many users think they need this. There is also the expectation that because one can turn a computer on, one is an expert in all fields of computing, including data recovery.
One solution to this problem is to give no options, and only a single path through the software. For predictable failures, and known configurations this can be implemented, and an example in CnW is the wizard for failed mini DVDs, to produce a new video disk. For devices such as a HP RAID, there are several variations, and different requirements. To this end, the chosen approach has been to try and trap errors, and give possible solutions. At the same time, intelligence is built into the program to try and determine the original configuration, and point to a solution.
With HP Raid disks many users do not know if the disk is a mirrored disk, or a pair of disks. Routines can be built into the program to indicate that the single disk is probably the second of a pair, or that an extra disk is likely to be required, and configured as a JBOD.
The overall result of this development is a program that will result in fewer calls for support, and a quicker solution for users who do not want to read the manual, but just get a solution.
Saturday, December 31, 2011
Monday, November 28, 2011
Fake software
I recently needed to purchase another copy of Microsoft Office 2007 for a new laptop. Being fairly expensive, I searched Amazon for a deal. As the product has now been replaced with 2010 I hope to get some old stock. I found a copy at about £175 and purchased.
I received the package very quickly, but the CD was warped, and would not load. Having already got a package, I decided to load my old (and I think idenical CD) but use the new product code (registration number) to licence the software. However, the product code was not valid. After a very short correspondence with the supplier, I returned the package and received a full refund.
Attempt number 2 was no better. This time, eBay. The CD was good, product code worked when installing but when I came to online registration, I was told the code was a contfeit one. Again, the supply has promised to refund my purchase, and I have every confidence that this will happen.
My third attempt is as I have to pay the full cost, I have now purchased Office 2010 (just over £300) and am waiting for delivery.
One area that has surprised me is that both, probably fake items looked 100% genuine. I compared every thing with with my original genuine item and the only difference was manufacture in Ireland rather than Singapore. The full refund did eventually arrive.
Moral, you get what you pay for - but office professional is a bit expensive these days.
I received the package very quickly, but the CD was warped, and would not load. Having already got a package, I decided to load my old (and I think idenical CD) but use the new product code (registration number) to licence the software. However, the product code was not valid. After a very short correspondence with the supplier, I returned the package and received a full refund.
Attempt number 2 was no better. This time, eBay. The CD was good, product code worked when installing but when I came to online registration, I was told the code was a contfeit one. Again, the supply has promised to refund my purchase, and I have every confidence that this will happen.
My third attempt is as I have to pay the full cost, I have now purchased Office 2010 (just over £300) and am waiting for delivery.
One area that has surprised me is that both, probably fake items looked 100% genuine. I compared every thing with with my original genuine item and the only difference was manufacture in Ireland rather than Singapore. The full refund did eventually arrive.
Moral, you get what you pay for - but office professional is a bit expensive these days.
Saturday, November 26, 2011
More fragmented deleted video files
I came across another variation of ways that video files are created on a camera. A Quick time video (eg .mov, .3gp, .mp4) has three main sections, a 'ftyp' 'moov' and 'mdat'. The mdat is the video data, and 'moov' the indexing information. The 'mdat' can be large, ie 100s of MBs, while the moov may only be 10s of KBs or a few MBs. The 'moov segement is a variable size and can only be created when the complete 'mdat' has been recorded, which can make logical storage difficult when the final sequence of 'ftyp'-'moov'-'mdat' is required. The way this can be overcome in a camera is to record the 'ftyp' as a cluster, then all of the 'mdat' as complete clusters, and then the 'moov' also as complete clusters. By modifying the FAT on a FAT32 disk, the logical sequence can be made as 'ftyp'-'moov'-'mdat'.
When the files are deleted (maybe by accident) the logical sequence information is lost making recovery by data carving impossible. CnW Recovery have developed routines to detect this type of fragmentation and hence recover otherwise fragmented files. A previous blog discussed a similar problem, but on that occasion the 'ftyp' and 'moov' segements were joined together, and not in separate clusters.
When the files are deleted (maybe by accident) the logical sequence information is lost making recovery by data carving impossible. CnW Recovery have developed routines to detect this type of fragmentation and hence recover otherwise fragmented files. A previous blog discussed a similar problem, but on that occasion the 'ftyp' and 'moov' segements were joined together, and not in separate clusters.
Tuesday, November 22, 2011
Flash memory address decoding
Flash memory chips are very different to logical hard drives, or even USB memory sticks. With a hard drive, acessing sector 0 always takes one to sector 0. Where the sector is stored on teh disk is actually irrlevant as the disk controller takes care of this.
For a flash memory chip, there is also a controller that takes care of where a sector is stored. However, this controller is also the interface to the USB port and can fail. The solution then is to remove the flash memory chip (with a hot air gun) and read the data directly. By missing out the flash controller chip, it is often possible to read data on the memory chip - BUT the flash controller chip has to be emulated.
Flash memory in some respects is a compromise as there are limitations on how it can be written and read. Data can only be read in blocks, often equivalent to 8 or 16 sectors and it is best to districute where the data is written on the chip. Blocks can fail, so have to be remapped, and they can also wear out. Another aspect is that before a block can be overwitten it has to be cleared down, so writing can be a two stage process, and hence slower than reading.
The flash controller chip controls all of the above and has methods to obtain optimum performance. This involves storing the data in areas that do not physically relate to the logical address. When recovering data it is necessary to decode this mapping . This blog entry describes one useful method that uses CnW to help.
Most memory chips are part of a FAT controller system. The problem to solve is to determine where blocks of data are stored. The solution described is ony a small part of the whole process but one that can help considerably. If the start of a memory device can be created, then the basic disk parameters can be determined, eg cluster size and address of cluster 2. The memory (working with a disk image) can be scanned using search for Directory Stubs and this will produce a log of file names and logical addresses. The same memory image can now be carved, and this will produce some valid files, often JPEGs that will be validated and have a valid date and length.
The clever bit now is to find a file that has been carved, and also read, by matching the file length exactly. This will then give the logical address from the directory scan and the physical address from the carving. If different, then the mapping is wrong, but you can see how far out the mapping is, and hopefully work out why.
For a flash memory chip, there is also a controller that takes care of where a sector is stored. However, this controller is also the interface to the USB port and can fail. The solution then is to remove the flash memory chip (with a hot air gun) and read the data directly. By missing out the flash controller chip, it is often possible to read data on the memory chip - BUT the flash controller chip has to be emulated.
Flash memory in some respects is a compromise as there are limitations on how it can be written and read. Data can only be read in blocks, often equivalent to 8 or 16 sectors and it is best to districute where the data is written on the chip. Blocks can fail, so have to be remapped, and they can also wear out. Another aspect is that before a block can be overwitten it has to be cleared down, so writing can be a two stage process, and hence slower than reading.
The flash controller chip controls all of the above and has methods to obtain optimum performance. This involves storing the data in areas that do not physically relate to the logical address. When recovering data it is necessary to decode this mapping . This blog entry describes one useful method that uses CnW to help.
Most memory chips are part of a FAT controller system. The problem to solve is to determine where blocks of data are stored. The solution described is ony a small part of the whole process but one that can help considerably. If the start of a memory device can be created, then the basic disk parameters can be determined, eg cluster size and address of cluster 2. The memory (working with a disk image) can be scanned using search for Directory Stubs and this will produce a log of file names and logical addresses. The same memory image can now be carved, and this will produce some valid files, often JPEGs that will be validated and have a valid date and length.
The clever bit now is to find a file that has been carved, and also read, by matching the file length exactly. This will then give the logical address from the directory scan and the physical address from the carving. If different, then the mapping is wrong, but you can see how far out the mapping is, and hopefully work out why.
Wednesday, November 2, 2011
Price of disk drives
I got a nasty shock yesterday when I cam to purchase a new disk drive. I've had a few in house failures recently, and a few weeks ago I purchased a new 2TB Western Digital drive from PC World. I was in a hurry for one so did not investigate much on price, but was happy with £79.99. Yesterday, I required another one, and checked on line with my normal supplier and it looked like the £79.99 was in fact a very good price. I drove back to PC World (about 10 miles away) only to discover that 2TB drives where now about £120. The best value I could see (GB per pound) was a 3TB WD Green drive for £169.99.
At check out I queried the price rise and was informed about floods in the Far East. A bit of Google research later in the day suggested that the Western Digital plant may be under water. There are press concerns about no availability, and price rises.
On drives, I personally like the WD Green range as they run very cool. Speed may not be as fast as some, but as much of my work still includes a USB2.0 connection, speed is not a major issue.
At check out I queried the price rise and was informed about floods in the Far East. A bit of Google research later in the day suggested that the Western Digital plant may be under water. There are press concerns about no availability, and price rises.
On drives, I personally like the WD Green range as they run very cool. Speed may not be as fast as some, but as much of my work still includes a USB2.0 connection, speed is not a major issue.
Tuesday, November 1, 2011
Corrupted Macintosh Disk
On many disks there is often a critical sector. On a Macintosh HFS disk it is often located at sector 0x40042, it starts with "H+" and points to the equally critical Catlog and Extents area of the disk. It is known as the Volume header. On a 'good' day there is a duplicate at the end of the disk.
A recent disk I received was missing both the 0x40042 sector and the one at the end of the disk and so recovery became a rather more manual process than normally happens. The disk was a 1TB drive containing over 900 GB of data, mainly music related. Various areas of the disk had failed and so I started with an incremental image.
The incremental image had a problem that when some sectors were attempted to be read, the drive failed in a way that only a power off and on reset would kick it back into life. This was tedious, but overall probably more than 98% of the disk was imaged.
The next problem was to reconstruct the parameters that sector 0x40042 would have provided. The area where the catalog is often found was searched, and a catalog could be found there, with a length of 0x2000 bytes (ie 16 sectors long). A Mac catalog entry starts with two pointers, pointing the entry before and after as a linked list chain. By looking for pointers that were different by the value of 2, it is a good guess that the one being looked at is the value in the middle. For example, pointers 0x12bb and 0x12bd were found which indicates that the actual catalog value is 0x12bc, and this was in sector 0x8ecf0. By subtracting 0x12bc0 sectors from 0x8ecf0 the location 0x7c130 was established as the catlog start location (in sectors).
By examining the disk, a few Volume header sectors were found, and these indicated that the catalog should start at cluster 0x781e. With a typical 8 sectors per cluster, this maps to offset 0x3c0c0. When this value is added to 0x40040 (the volume start location) we get to 0x7c130 and good assumption that the Volume header has senible values. The rather harder to determine extents map was then used from the dummy Volume header.
Once these values were entered by hand into the CnW Mac recovery option screen, data was recovered
A recent disk I received was missing both the 0x40042 sector and the one at the end of the disk and so recovery became a rather more manual process than normally happens. The disk was a 1TB drive containing over 900 GB of data, mainly music related. Various areas of the disk had failed and so I started with an incremental image.
The incremental image had a problem that when some sectors were attempted to be read, the drive failed in a way that only a power off and on reset would kick it back into life. This was tedious, but overall probably more than 98% of the disk was imaged.
The next problem was to reconstruct the parameters that sector 0x40042 would have provided. The area where the catalog is often found was searched, and a catalog could be found there, with a length of 0x2000 bytes (ie 16 sectors long). A Mac catalog entry starts with two pointers, pointing the entry before and after as a linked list chain. By looking for pointers that were different by the value of 2, it is a good guess that the one being looked at is the value in the middle. For example, pointers 0x12bb and 0x12bd were found which indicates that the actual catalog value is 0x12bc, and this was in sector 0x8ecf0. By subtracting 0x12bc0 sectors from 0x8ecf0 the location 0x7c130 was established as the catlog start location (in sectors).
By examining the disk, a few Volume header sectors were found, and these indicated that the catalog should start at cluster 0x781e. With a typical 8 sectors per cluster, this maps to offset 0x3c0c0. When this value is added to 0x40040 (the volume start location) we get to 0x7c130 and good assumption that the Volume header has senible values. The rather harder to determine extents map was then used from the dummy Volume header.
Once these values were entered by hand into the CnW Mac recovery option screen, data was recovered
Monday, October 31, 2011
Overwritten memory chip
Data recovery is all about get data back when it has been lost. Often this is possible but there is one case when it just cannot be achieved. The problem is when the physical area,sector has been overwritten.
Whe a file is deleted, normally the index, or directory entry is removed. When a disk is formatted, generally only the key areas on the disk are written to. In both of these cases, old data will exist, and programs such as CnW Recovery will find the data.
When a sector is overwritten, the data is lost. If there are no backups of the data, there is nothing that can be done. There are suggestions that with a hard drive, unless data is overwritten 7 times, then it could be recovered - the author is extremely sceptical of this due to the exceptionally high density of current disks. What may have been possible 20 years ago, is no longer the case. For memory chips, it is also very definate. There is no scope of reading slightly off track and getting data back. It could be argued that routines that process wear leveling could point to old data, but this would only be a maximum of about 5% of the data, from any period of the disk. It will not contain a complete photo.
With cameras, because there is no editing, there is no requirement for any backups. The conclusion is that an overwritten photo is lost. Take care.
Whe a file is deleted, normally the index, or directory entry is removed. When a disk is formatted, generally only the key areas on the disk are written to. In both of these cases, old data will exist, and programs such as CnW Recovery will find the data.
When a sector is overwritten, the data is lost. If there are no backups of the data, there is nothing that can be done. There are suggestions that with a hard drive, unless data is overwritten 7 times, then it could be recovered - the author is extremely sceptical of this due to the exceptionally high density of current disks. What may have been possible 20 years ago, is no longer the case. For memory chips, it is also very definate. There is no scope of reading slightly off track and getting data back. It could be argued that routines that process wear leveling could point to old data, but this would only be a maximum of about 5% of the data, from any period of the disk. It will not contain a complete photo.
With cameras, because there is no editing, there is no requirement for any backups. The conclusion is that an overwritten photo is lost. Take care.
Wednesday, August 17, 2011
More FAT32 delete problems
I have just come across a FAT32 memory chip with deleted files. There is a well known issue that when FAT32 has deleted files, the upper 16 bits of the cluster address are blanked, though the lower 16 bits remain valid. CnW has developed routines to make use of the lower 16 bits, and with intelligence can recreate the upper 16 bits for know file types. The new chip was from a video recorder, and all 32 bits have been deleted. This means there is no information on the location of the file.
Fortunately the FAT directory entry does contain the file length. The only way that file names can be associated with files is to data carve the disk and then try and match file lengths. This is far from optimum, but does provide a partial solution to an other impossible problem.
Fortunately, FAT32 is now largely used for removeable storage and typically for one type of file, eg video, music or photos. File names are not always critical, and data carving can produce reasonable results.
Fortunately the FAT directory entry does contain the file length. The only way that file names can be associated with files is to data carve the disk and then try and match file lengths. This is far from optimum, but does provide a partial solution to an other impossible problem.
Fortunately, FAT32 is now largely used for removeable storage and typically for one type of file, eg video, music or photos. File names are not always critical, and data carving can produce reasonable results.
Monday, July 4, 2011
More Digital Signature issues with IE9.0 and Norton
My last posting talked about digitally signing software for downloads. Unfortunately this is only half of the solution. I am sure that Microsoft and Norton are working in most people's best interests, but this does not help the way that CnW operates. With CnW, the idea is to give users the latest updates (and even bug fixes) as soon as possible. This is done by updating the main software package maybe once or twice a month. The changes are small, but incrementally ensure that the produce evolves, and responds to customer feed back.
Part of IE9.0 and Norton security is how long any download has been on the internet. This period is monitored to see if there has been any negative feedback about the product. Norton suggests there is a 1 week learning time, which would man that CnW software would be viewed as dubious for maybe 50% of it's time.
The next plan to try and resolve this issue is to move to the common pattern of having a installer tool, and then in effect an update program. In this case, the installer can be written, and will not be changed for a considerable period of time, maybe a year. The installer will then download the main, and frequently updated data from it's own secure environment. Everything will be digitally signed, but the installer can be recognised by IE9.0 and Norton as a stable, safe program. IE9.0 and Norton will only see the stable installer, though Norton will obviously also monitor any future updates.
Part of IE9.0 and Norton security is how long any download has been on the internet. This period is monitored to see if there has been any negative feedback about the product. Norton suggests there is a 1 week learning time, which would man that CnW software would be viewed as dubious for maybe 50% of it's time.
The next plan to try and resolve this issue is to move to the common pattern of having a installer tool, and then in effect an update program. In this case, the installer can be written, and will not be changed for a considerable period of time, maybe a year. The installer will then download the main, and frequently updated data from it's own secure environment. Everything will be digitally signed, but the installer can be recognised by IE9.0 and Norton as a stable, safe program. IE9.0 and Norton will only see the stable installer, though Norton will obviously also monitor any future updates.
Friday, July 1, 2011
Software signing and IE9.0
Internet security can be a large issue. To this end, IE9.0 has very 'tough' security measures added to try and protect users from downloading unsafe software. If a user tries to download an unsigned program, there are severe warning messages and it can be difficult to download or run the program.
To overcome the above problem the program has to be digitally signed, with a secure certificate. This can be an expensive process but fortunately there are solutions for small companies. One such company - that CnW have used - is Global Sign. This gives a digital signature which can then be traced back to Global Sign. Having this on the program now means that Internet Explorer V9 does not scream that the software could be very dangerous.
The other security measure that is causing problems is Norton Insight. This can warn that the software is new, and may be dangerous. The solution can be to get listed on their White List (is not a Black List). The problem here is that listing can take a few weeks, and with CnW this is when the next release may have been made. If a company does a single release each year this is not a problem, but for a small company doing regular updates to keep pace with new solutions, and customer requests, then Norton is not very friendly. Unfortunately, Norton does not seem to recognise the digital signature. I do not know what the ultimate solution will be.
To overcome the above problem the program has to be digitally signed, with a secure certificate. This can be an expensive process but fortunately there are solutions for small companies. One such company - that CnW have used - is Global Sign. This gives a digital signature which can then be traced back to Global Sign. Having this on the program now means that Internet Explorer V9 does not scream that the software could be very dangerous.
The other security measure that is causing problems is Norton Insight. This can warn that the software is new, and may be dangerous. The solution can be to get listed on their White List (is not a Black List). The problem here is that listing can take a few weeks, and with CnW this is when the next release may have been made. If a company does a single release each year this is not a problem, but for a small company doing regular updates to keep pace with new solutions, and customer requests, then Norton is not very friendly. Unfortunately, Norton does not seem to recognise the digital signature. I do not know what the ultimate solution will be.
Subscribe to:
Posts (Atom)