Software development can be a slow process, but in many respects that is the best approach. But this slow process should ideally be a series of small developments, each tested and reviewed on a continuous basis.
An example is the Reiser handler for CnW Recovery. This was written in response to a an internal job to recover files from a failed HP Media Vault device. The disks were OK, but the RAID had failed. The development required was both support for JBOD (Job bunch of disks) and the complex Reiser format. The job was done, and all data recovered. ie Version 1 worked.
The next requirement came to read deleted files. Deleted files on Reiser, according to many Internet sites is not possible, so CnW developed a method that works to about 80% success, a figure that varies and depends on many factors.
When software is released, there are often problems as unseen configurations, or failures crop up. One by one these are fixed, and the product improves. Currently, there is a good engine for HP Mediavault recovery, and many successful recoveries have been performed.
Along with a good engine, an easy to use interface has to be developed, and here the problems can expand. Users can be extremely computer literate, or beginners. Most users are not experts in data recovery and so there are many terms and concepts unfamiliar to them. A good place to start is with the Help system, but not many users think they need this. There is also the expectation that because one can turn a computer on, one is an expert in all fields of computing, including data recovery.
One solution to this problem is to give no options, and only a single path through the software. For predictable failures, and known configurations this can be implemented, and an example in CnW is the wizard for failed mini DVDs, to produce a new video disk. For devices such as a HP RAID, there are several variations, and different requirements. To this end, the chosen approach has been to try and trap errors, and give possible solutions. At the same time, intelligence is built into the program to try and determine the original configuration, and point to a solution.
With HP Raid disks many users do not know if the disk is a mirrored disk, or a pair of disks. Routines can be built into the program to indicate that the single disk is probably the second of a pair, or that an extra disk is likely to be required, and configured as a JBOD.
The overall result of this development is a program that will result in fewer calls for support, and a quicker solution for users who do not want to read the manual, but just get a solution.
Saturday, December 31, 2011
Monday, November 28, 2011
Fake software
I recently needed to purchase another copy of Microsoft Office 2007 for a new laptop. Being fairly expensive, I searched Amazon for a deal. As the product has now been replaced with 2010 I hope to get some old stock. I found a copy at about £175 and purchased.
I received the package very quickly, but the CD was warped, and would not load. Having already got a package, I decided to load my old (and I think idenical CD) but use the new product code (registration number) to licence the software. However, the product code was not valid. After a very short correspondence with the supplier, I returned the package and received a full refund.
Attempt number 2 was no better. This time, eBay. The CD was good, product code worked when installing but when I came to online registration, I was told the code was a contfeit one. Again, the supply has promised to refund my purchase, and I have every confidence that this will happen.
My third attempt is as I have to pay the full cost, I have now purchased Office 2010 (just over £300) and am waiting for delivery.
One area that has surprised me is that both, probably fake items looked 100% genuine. I compared every thing with with my original genuine item and the only difference was manufacture in Ireland rather than Singapore. The full refund did eventually arrive.
Moral, you get what you pay for - but office professional is a bit expensive these days.
I received the package very quickly, but the CD was warped, and would not load. Having already got a package, I decided to load my old (and I think idenical CD) but use the new product code (registration number) to licence the software. However, the product code was not valid. After a very short correspondence with the supplier, I returned the package and received a full refund.
Attempt number 2 was no better. This time, eBay. The CD was good, product code worked when installing but when I came to online registration, I was told the code was a contfeit one. Again, the supply has promised to refund my purchase, and I have every confidence that this will happen.
My third attempt is as I have to pay the full cost, I have now purchased Office 2010 (just over £300) and am waiting for delivery.
One area that has surprised me is that both, probably fake items looked 100% genuine. I compared every thing with with my original genuine item and the only difference was manufacture in Ireland rather than Singapore. The full refund did eventually arrive.
Moral, you get what you pay for - but office professional is a bit expensive these days.
Saturday, November 26, 2011
More fragmented deleted video files
I came across another variation of ways that video files are created on a camera. A Quick time video (eg .mov, .3gp, .mp4) has three main sections, a 'ftyp' 'moov' and 'mdat'. The mdat is the video data, and 'moov' the indexing information. The 'mdat' can be large, ie 100s of MBs, while the moov may only be 10s of KBs or a few MBs. The 'moov segement is a variable size and can only be created when the complete 'mdat' has been recorded, which can make logical storage difficult when the final sequence of 'ftyp'-'moov'-'mdat' is required. The way this can be overcome in a camera is to record the 'ftyp' as a cluster, then all of the 'mdat' as complete clusters, and then the 'moov' also as complete clusters. By modifying the FAT on a FAT32 disk, the logical sequence can be made as 'ftyp'-'moov'-'mdat'.
When the files are deleted (maybe by accident) the logical sequence information is lost making recovery by data carving impossible. CnW Recovery have developed routines to detect this type of fragmentation and hence recover otherwise fragmented files. A previous blog discussed a similar problem, but on that occasion the 'ftyp' and 'moov' segements were joined together, and not in separate clusters.
When the files are deleted (maybe by accident) the logical sequence information is lost making recovery by data carving impossible. CnW Recovery have developed routines to detect this type of fragmentation and hence recover otherwise fragmented files. A previous blog discussed a similar problem, but on that occasion the 'ftyp' and 'moov' segements were joined together, and not in separate clusters.
Tuesday, November 22, 2011
Flash memory address decoding
Flash memory chips are very different to logical hard drives, or even USB memory sticks. With a hard drive, acessing sector 0 always takes one to sector 0. Where the sector is stored on teh disk is actually irrlevant as the disk controller takes care of this.
For a flash memory chip, there is also a controller that takes care of where a sector is stored. However, this controller is also the interface to the USB port and can fail. The solution then is to remove the flash memory chip (with a hot air gun) and read the data directly. By missing out the flash controller chip, it is often possible to read data on the memory chip - BUT the flash controller chip has to be emulated.
Flash memory in some respects is a compromise as there are limitations on how it can be written and read. Data can only be read in blocks, often equivalent to 8 or 16 sectors and it is best to districute where the data is written on the chip. Blocks can fail, so have to be remapped, and they can also wear out. Another aspect is that before a block can be overwitten it has to be cleared down, so writing can be a two stage process, and hence slower than reading.
The flash controller chip controls all of the above and has methods to obtain optimum performance. This involves storing the data in areas that do not physically relate to the logical address. When recovering data it is necessary to decode this mapping . This blog entry describes one useful method that uses CnW to help.
Most memory chips are part of a FAT controller system. The problem to solve is to determine where blocks of data are stored. The solution described is ony a small part of the whole process but one that can help considerably. If the start of a memory device can be created, then the basic disk parameters can be determined, eg cluster size and address of cluster 2. The memory (working with a disk image) can be scanned using search for Directory Stubs and this will produce a log of file names and logical addresses. The same memory image can now be carved, and this will produce some valid files, often JPEGs that will be validated and have a valid date and length.
The clever bit now is to find a file that has been carved, and also read, by matching the file length exactly. This will then give the logical address from the directory scan and the physical address from the carving. If different, then the mapping is wrong, but you can see how far out the mapping is, and hopefully work out why.
For a flash memory chip, there is also a controller that takes care of where a sector is stored. However, this controller is also the interface to the USB port and can fail. The solution then is to remove the flash memory chip (with a hot air gun) and read the data directly. By missing out the flash controller chip, it is often possible to read data on the memory chip - BUT the flash controller chip has to be emulated.
Flash memory in some respects is a compromise as there are limitations on how it can be written and read. Data can only be read in blocks, often equivalent to 8 or 16 sectors and it is best to districute where the data is written on the chip. Blocks can fail, so have to be remapped, and they can also wear out. Another aspect is that before a block can be overwitten it has to be cleared down, so writing can be a two stage process, and hence slower than reading.
The flash controller chip controls all of the above and has methods to obtain optimum performance. This involves storing the data in areas that do not physically relate to the logical address. When recovering data it is necessary to decode this mapping . This blog entry describes one useful method that uses CnW to help.
Most memory chips are part of a FAT controller system. The problem to solve is to determine where blocks of data are stored. The solution described is ony a small part of the whole process but one that can help considerably. If the start of a memory device can be created, then the basic disk parameters can be determined, eg cluster size and address of cluster 2. The memory (working with a disk image) can be scanned using search for Directory Stubs and this will produce a log of file names and logical addresses. The same memory image can now be carved, and this will produce some valid files, often JPEGs that will be validated and have a valid date and length.
The clever bit now is to find a file that has been carved, and also read, by matching the file length exactly. This will then give the logical address from the directory scan and the physical address from the carving. If different, then the mapping is wrong, but you can see how far out the mapping is, and hopefully work out why.
Wednesday, November 2, 2011
Price of disk drives
I got a nasty shock yesterday when I cam to purchase a new disk drive. I've had a few in house failures recently, and a few weeks ago I purchased a new 2TB Western Digital drive from PC World. I was in a hurry for one so did not investigate much on price, but was happy with £79.99. Yesterday, I required another one, and checked on line with my normal supplier and it looked like the £79.99 was in fact a very good price. I drove back to PC World (about 10 miles away) only to discover that 2TB drives where now about £120. The best value I could see (GB per pound) was a 3TB WD Green drive for £169.99.
At check out I queried the price rise and was informed about floods in the Far East. A bit of Google research later in the day suggested that the Western Digital plant may be under water. There are press concerns about no availability, and price rises.
On drives, I personally like the WD Green range as they run very cool. Speed may not be as fast as some, but as much of my work still includes a USB2.0 connection, speed is not a major issue.
At check out I queried the price rise and was informed about floods in the Far East. A bit of Google research later in the day suggested that the Western Digital plant may be under water. There are press concerns about no availability, and price rises.
On drives, I personally like the WD Green range as they run very cool. Speed may not be as fast as some, but as much of my work still includes a USB2.0 connection, speed is not a major issue.
Tuesday, November 1, 2011
Corrupted Macintosh Disk
On many disks there is often a critical sector. On a Macintosh HFS disk it is often located at sector 0x40042, it starts with "H+" and points to the equally critical Catlog and Extents area of the disk. It is known as the Volume header. On a 'good' day there is a duplicate at the end of the disk.
A recent disk I received was missing both the 0x40042 sector and the one at the end of the disk and so recovery became a rather more manual process than normally happens. The disk was a 1TB drive containing over 900 GB of data, mainly music related. Various areas of the disk had failed and so I started with an incremental image.
The incremental image had a problem that when some sectors were attempted to be read, the drive failed in a way that only a power off and on reset would kick it back into life. This was tedious, but overall probably more than 98% of the disk was imaged.
The next problem was to reconstruct the parameters that sector 0x40042 would have provided. The area where the catalog is often found was searched, and a catalog could be found there, with a length of 0x2000 bytes (ie 16 sectors long). A Mac catalog entry starts with two pointers, pointing the entry before and after as a linked list chain. By looking for pointers that were different by the value of 2, it is a good guess that the one being looked at is the value in the middle. For example, pointers 0x12bb and 0x12bd were found which indicates that the actual catalog value is 0x12bc, and this was in sector 0x8ecf0. By subtracting 0x12bc0 sectors from 0x8ecf0 the location 0x7c130 was established as the catlog start location (in sectors).
By examining the disk, a few Volume header sectors were found, and these indicated that the catalog should start at cluster 0x781e. With a typical 8 sectors per cluster, this maps to offset 0x3c0c0. When this value is added to 0x40040 (the volume start location) we get to 0x7c130 and good assumption that the Volume header has senible values. The rather harder to determine extents map was then used from the dummy Volume header.
Once these values were entered by hand into the CnW Mac recovery option screen, data was recovered
A recent disk I received was missing both the 0x40042 sector and the one at the end of the disk and so recovery became a rather more manual process than normally happens. The disk was a 1TB drive containing over 900 GB of data, mainly music related. Various areas of the disk had failed and so I started with an incremental image.
The incremental image had a problem that when some sectors were attempted to be read, the drive failed in a way that only a power off and on reset would kick it back into life. This was tedious, but overall probably more than 98% of the disk was imaged.
The next problem was to reconstruct the parameters that sector 0x40042 would have provided. The area where the catalog is often found was searched, and a catalog could be found there, with a length of 0x2000 bytes (ie 16 sectors long). A Mac catalog entry starts with two pointers, pointing the entry before and after as a linked list chain. By looking for pointers that were different by the value of 2, it is a good guess that the one being looked at is the value in the middle. For example, pointers 0x12bb and 0x12bd were found which indicates that the actual catalog value is 0x12bc, and this was in sector 0x8ecf0. By subtracting 0x12bc0 sectors from 0x8ecf0 the location 0x7c130 was established as the catlog start location (in sectors).
By examining the disk, a few Volume header sectors were found, and these indicated that the catalog should start at cluster 0x781e. With a typical 8 sectors per cluster, this maps to offset 0x3c0c0. When this value is added to 0x40040 (the volume start location) we get to 0x7c130 and good assumption that the Volume header has senible values. The rather harder to determine extents map was then used from the dummy Volume header.
Once these values were entered by hand into the CnW Mac recovery option screen, data was recovered
Monday, October 31, 2011
Overwritten memory chip
Data recovery is all about get data back when it has been lost. Often this is possible but there is one case when it just cannot be achieved. The problem is when the physical area,sector has been overwritten.
Whe a file is deleted, normally the index, or directory entry is removed. When a disk is formatted, generally only the key areas on the disk are written to. In both of these cases, old data will exist, and programs such as CnW Recovery will find the data.
When a sector is overwritten, the data is lost. If there are no backups of the data, there is nothing that can be done. There are suggestions that with a hard drive, unless data is overwritten 7 times, then it could be recovered - the author is extremely sceptical of this due to the exceptionally high density of current disks. What may have been possible 20 years ago, is no longer the case. For memory chips, it is also very definate. There is no scope of reading slightly off track and getting data back. It could be argued that routines that process wear leveling could point to old data, but this would only be a maximum of about 5% of the data, from any period of the disk. It will not contain a complete photo.
With cameras, because there is no editing, there is no requirement for any backups. The conclusion is that an overwritten photo is lost. Take care.
Whe a file is deleted, normally the index, or directory entry is removed. When a disk is formatted, generally only the key areas on the disk are written to. In both of these cases, old data will exist, and programs such as CnW Recovery will find the data.
When a sector is overwritten, the data is lost. If there are no backups of the data, there is nothing that can be done. There are suggestions that with a hard drive, unless data is overwritten 7 times, then it could be recovered - the author is extremely sceptical of this due to the exceptionally high density of current disks. What may have been possible 20 years ago, is no longer the case. For memory chips, it is also very definate. There is no scope of reading slightly off track and getting data back. It could be argued that routines that process wear leveling could point to old data, but this would only be a maximum of about 5% of the data, from any period of the disk. It will not contain a complete photo.
With cameras, because there is no editing, there is no requirement for any backups. The conclusion is that an overwritten photo is lost. Take care.
Wednesday, August 17, 2011
More FAT32 delete problems
I have just come across a FAT32 memory chip with deleted files. There is a well known issue that when FAT32 has deleted files, the upper 16 bits of the cluster address are blanked, though the lower 16 bits remain valid. CnW has developed routines to make use of the lower 16 bits, and with intelligence can recreate the upper 16 bits for know file types. The new chip was from a video recorder, and all 32 bits have been deleted. This means there is no information on the location of the file.
Fortunately the FAT directory entry does contain the file length. The only way that file names can be associated with files is to data carve the disk and then try and match file lengths. This is far from optimum, but does provide a partial solution to an other impossible problem.
Fortunately, FAT32 is now largely used for removeable storage and typically for one type of file, eg video, music or photos. File names are not always critical, and data carving can produce reasonable results.
Fortunately the FAT directory entry does contain the file length. The only way that file names can be associated with files is to data carve the disk and then try and match file lengths. This is far from optimum, but does provide a partial solution to an other impossible problem.
Fortunately, FAT32 is now largely used for removeable storage and typically for one type of file, eg video, music or photos. File names are not always critical, and data carving can produce reasonable results.
Monday, July 4, 2011
More Digital Signature issues with IE9.0 and Norton
My last posting talked about digitally signing software for downloads. Unfortunately this is only half of the solution. I am sure that Microsoft and Norton are working in most people's best interests, but this does not help the way that CnW operates. With CnW, the idea is to give users the latest updates (and even bug fixes) as soon as possible. This is done by updating the main software package maybe once or twice a month. The changes are small, but incrementally ensure that the produce evolves, and responds to customer feed back.
Part of IE9.0 and Norton security is how long any download has been on the internet. This period is monitored to see if there has been any negative feedback about the product. Norton suggests there is a 1 week learning time, which would man that CnW software would be viewed as dubious for maybe 50% of it's time.
The next plan to try and resolve this issue is to move to the common pattern of having a installer tool, and then in effect an update program. In this case, the installer can be written, and will not be changed for a considerable period of time, maybe a year. The installer will then download the main, and frequently updated data from it's own secure environment. Everything will be digitally signed, but the installer can be recognised by IE9.0 and Norton as a stable, safe program. IE9.0 and Norton will only see the stable installer, though Norton will obviously also monitor any future updates.
Part of IE9.0 and Norton security is how long any download has been on the internet. This period is monitored to see if there has been any negative feedback about the product. Norton suggests there is a 1 week learning time, which would man that CnW software would be viewed as dubious for maybe 50% of it's time.
The next plan to try and resolve this issue is to move to the common pattern of having a installer tool, and then in effect an update program. In this case, the installer can be written, and will not be changed for a considerable period of time, maybe a year. The installer will then download the main, and frequently updated data from it's own secure environment. Everything will be digitally signed, but the installer can be recognised by IE9.0 and Norton as a stable, safe program. IE9.0 and Norton will only see the stable installer, though Norton will obviously also monitor any future updates.
Friday, July 1, 2011
Software signing and IE9.0
Internet security can be a large issue. To this end, IE9.0 has very 'tough' security measures added to try and protect users from downloading unsafe software. If a user tries to download an unsigned program, there are severe warning messages and it can be difficult to download or run the program.
To overcome the above problem the program has to be digitally signed, with a secure certificate. This can be an expensive process but fortunately there are solutions for small companies. One such company - that CnW have used - is Global Sign. This gives a digital signature which can then be traced back to Global Sign. Having this on the program now means that Internet Explorer V9 does not scream that the software could be very dangerous.
The other security measure that is causing problems is Norton Insight. This can warn that the software is new, and may be dangerous. The solution can be to get listed on their White List (is not a Black List). The problem here is that listing can take a few weeks, and with CnW this is when the next release may have been made. If a company does a single release each year this is not a problem, but for a small company doing regular updates to keep pace with new solutions, and customer requests, then Norton is not very friendly. Unfortunately, Norton does not seem to recognise the digital signature. I do not know what the ultimate solution will be.
To overcome the above problem the program has to be digitally signed, with a secure certificate. This can be an expensive process but fortunately there are solutions for small companies. One such company - that CnW have used - is Global Sign. This gives a digital signature which can then be traced back to Global Sign. Having this on the program now means that Internet Explorer V9 does not scream that the software could be very dangerous.
The other security measure that is causing problems is Norton Insight. This can warn that the software is new, and may be dangerous. The solution can be to get listed on their White List (is not a Black List). The problem here is that listing can take a few weeks, and with CnW this is when the next release may have been made. If a company does a single release each year this is not a problem, but for a small company doing regular updates to keep pace with new solutions, and customer requests, then Norton is not very friendly. Unfortunately, Norton does not seem to recognise the digital signature. I do not know what the ultimate solution will be.
Friday, June 3, 2011
Solid State devices
Many products are deemed to be dead long before they eventualy die entirely. Magnetic tape for datya storage is one, but there is a very important market for tape, though for the home user and small business, disk back up is now a much more viable proposition. The next prediction is the death of the rotating hard drive.
We are all used to solid state memory for cameras, telephones and many video devices. The capacity keeps increasing, and cost keeps coming down. Just starting to come in are solid state drives for laptops and desk top computers. As there is no head seek time, or rotational delay, reading can be very fast and many users are added them as the system drive in a PC. They report impressive performance in boot up and launching programs. It is noit all good news, as writing can be slow, and there is an issue of limited write cycles. Basically, sector will wear out if used too often. The solution to the last point is that chip controllers use a feature called wear leveling, so when a sector has been used too many times, it will be physically moved to another location, while still keeping the same logical address.
Data recovery of such SSDs has two main problems. If the controller dies, then it is necessary to work with the chips directly. These means they have to be unsoldered from the board and read. This means pointing a hot air gun at the chips and removing them, with out damage or over heating. It is possible, and not quite as bad as it sounds. The major problem though comes next. Manufacturers do not publish their wear leveling routines. As the chips are not meant to moved between devices, there is no requirement for any standards - all that matters is that when a sector number is requested, the correct data is returned. The physical sector is not relevant in any way.
CnW are now looking at such drives, and memory chips and will be developing tools to assist with recovery
We are all used to solid state memory for cameras, telephones and many video devices. The capacity keeps increasing, and cost keeps coming down. Just starting to come in are solid state drives for laptops and desk top computers. As there is no head seek time, or rotational delay, reading can be very fast and many users are added them as the system drive in a PC. They report impressive performance in boot up and launching programs. It is noit all good news, as writing can be slow, and there is an issue of limited write cycles. Basically, sector will wear out if used too often. The solution to the last point is that chip controllers use a feature called wear leveling, so when a sector has been used too many times, it will be physically moved to another location, while still keeping the same logical address.
Data recovery of such SSDs has two main problems. If the controller dies, then it is necessary to work with the chips directly. These means they have to be unsoldered from the board and read. This means pointing a hot air gun at the chips and removing them, with out damage or over heating. It is possible, and not quite as bad as it sounds. The major problem though comes next. Manufacturers do not publish their wear leveling routines. As the chips are not meant to moved between devices, there is no requirement for any standards - all that matters is that when a sector number is requested, the correct data is returned. The physical sector is not relevant in any way.
CnW are now looking at such drives, and memory chips and will be developing tools to assist with recovery
Thursday, May 26, 2011
Backup procedures
I often meet customers who say - I was about to back it up but....
Last week I suffered a serious problem with a Window 7 64 PC. It had become corrupted - probably due to device drivers - in a way that it would not respond to either mouse or keyboard. It booted up, I could see it over the network, but not control anything.
It is a Dell computer, and did not come with a Windows disk, but I had made a repair disk. I managed to boot into repair mode and first tried to go back a few restore points, however nothing worked.
I have a few backup prodcures in place, so was not too worried. The methods I have are
The first stage was to use the Microsoft image which is created evry week. The recovery mode allows for this to be restored, but it did not want to recognise the RAID box. However, the files were copied onto a USB drive, and was then visible. In the mean time, I took a complete image copy of the problem drive so that all updated files could be recovered as required.
The disk was then updaed with the microsft image, but this would not produce a bootable drive. It always came up in recovery mode, and often indicated that there were bad directories etc.
The final stage was to restore the partition, and boot sector from the Macrium backup. This was a few months old, but gave the pomise of a working system. I was very pleased that this worked, and it immediately booted and started running. The system then spent some time updating a few months of Norton and Microsoft patches. The recent file were copied back, some from disk, and some from carbonite, and all is now working correctly. No files or data lost.
My concern remains on how good the Microsoft backup system is. In the next few months I will try and receate a complete backup and see if it works.
Last week I suffered a serious problem with a Window 7 64 PC. It had become corrupted - probably due to device drivers - in a way that it would not respond to either mouse or keyboard. It booted up, I could see it over the network, but not control anything.
It is a Dell computer, and did not come with a Windows disk, but I had made a repair disk. I managed to boot into repair mode and first tried to go back a few restore points, however nothing worked.
I have a few backup prodcures in place, so was not too worried. The methods I have are
- On line carbonite
- Weekly Microsoft backup to a local RAID - in a separate box
- Periodic disk images using Macrium Reflect
- Very critical code (my source code) is backed up every 8 hours onto another PC, local but in a different room.
The first stage was to use the Microsoft image which is created evry week. The recovery mode allows for this to be restored, but it did not want to recognise the RAID box. However, the files were copied onto a USB drive, and was then visible. In the mean time, I took a complete image copy of the problem drive so that all updated files could be recovered as required.
The disk was then updaed with the microsft image, but this would not produce a bootable drive. It always came up in recovery mode, and often indicated that there were bad directories etc.
The final stage was to restore the partition, and boot sector from the Macrium backup. This was a few months old, but gave the pomise of a working system. I was very pleased that this worked, and it immediately booted and started running. The system then spent some time updating a few months of Norton and Microsoft patches. The recent file were copied back, some from disk, and some from carbonite, and all is now working correctly. No files or data lost.
My concern remains on how good the Microsoft backup system is. In the next few months I will try and receate a complete backup and see if it works.
Friday, May 6, 2011
DVD+RW apparently blank
I recently had a DVD+RW for recovery. It was from a camcorder and had probably been reformatted.
On initial examination the start of the disk was OK, but the majority was blank, ie all the sectors were filled with zeros. I could read the sectors and no error messages were displayed. Every indication was that the disk had been blanked, as if thee had been a full format.
When the disk was examined on hardware designed to read blank disks it was very interesting that the disk was not blank. There was a significant amount of video still on the disk and CnW software did a recovery, and then generated a new video disk.
The concern is that standard hardware gave every indication that the disk was blank - so do not rely on standardard hardware if there is a possibility that the data may actually still exist. CnW Recovery services will always assist anyone with such possible disks.
On initial examination the start of the disk was OK, but the majority was blank, ie all the sectors were filled with zeros. I could read the sectors and no error messages were displayed. Every indication was that the disk had been blanked, as if thee had been a full format.
When the disk was examined on hardware designed to read blank disks it was very interesting that the disk was not blank. There was a significant amount of video still on the disk and CnW software did a recovery, and then generated a new video disk.
The concern is that standard hardware gave every indication that the disk was blank - so do not rely on standardard hardware if there is a possibility that the data may actually still exist. CnW Recovery services will always assist anyone with such possible disks.
Tuesday, April 26, 2011
Finding owner of data
On a recent walk along a Californian beach I found a compact camera, covered in sand and salt water. The camera was very dead, but out of interest I eventally managed to extract the 2GB SD memory card. After drying it out, it was readable with errors. When totally dry, it read without errors.
I was keen to try and contact the owner and return the photos, over 500 of them. Unfortunately, there is very little on a memory chip to tell you about an owner, only the camera, and date and time of the photos. From this I could determine that the last photo had been taken about 10 days before I found the camera. I did try and add a 'Found camera' to a local lost and found website, but no reply.
Recently, I did a bit more investigation by looking at the photos more closely. Obviously the owner was a young person, with lots of photos in night clubs, no names identifiable, but also some at a college. One photo though I do hope will be a major clue is that it includes, possibly the owner, or close friend holding a college certificate with their unusal surname name on it. A bit of Googling took me to Facebook, and I hope a perfect match. (The name and college course both match, and the photo looks similar). I have now sent a message to the person, and am waiting for a reply.
The moral of the story is that if you want goods returned, it can be helpful to have some return details stored with it.
As a post script, the owner did contact me and a DVD of photos was sent. I have just received a very nice letter of thanks.
I was keen to try and contact the owner and return the photos, over 500 of them. Unfortunately, there is very little on a memory chip to tell you about an owner, only the camera, and date and time of the photos. From this I could determine that the last photo had been taken about 10 days before I found the camera. I did try and add a 'Found camera' to a local lost and found website, but no reply.
Recently, I did a bit more investigation by looking at the photos more closely. Obviously the owner was a young person, with lots of photos in night clubs, no names identifiable, but also some at a college. One photo though I do hope will be a major clue is that it includes, possibly the owner, or close friend holding a college certificate with their unusal surname name on it. A bit of Googling took me to Facebook, and I hope a perfect match. (The name and college course both match, and the photo looks similar). I have now sent a message to the person, and am waiting for a reply.
The moral of the story is that if you want goods returned, it can be helpful to have some return details stored with it.
As a post script, the owner did contact me and a DVD of photos was sent. I have just received a very nice letter of thanks.
Sunday, March 6, 2011
Virus issue
I don't often get involved in fixing PC problems, but recent jobs involved the same type of issue. The PC starts to report serious errors, and show problems with boot sector and other critical areas of the disk. It all looks rather real, and very worrying. However, in both cases it was caused by a similar, but different Fake Anti Virus program. I think if you continue there was an offer of the program to fix the issue - in exchange for some $$$s.
The approach I tooks was to remove the drive from the laptop and create an image for security purposes. I then ran Norton which tracked down several viruses, and removed them. This is where the two viruses behaved differently.
Virus one did not want to be removed by Norton and started each time the machine booted. The problem was that a startup function (go to msconfig) was launching the virus at startup each time. By removing this start up line - and seeing the program it was starting (it had a randomly generated name) the PC was then OK. The free AVG virus checker was added to the PC to try and prevent this happening again. A report a few weeks after this event indicated that eveything has been OK.
Virus two was removed by Norton, but left the PC in a state where no program would actually launch from explorer. Various 'Googled' ideas pointed to the registry, but this did not help. Launching the command prompt was also very difficult and the start program launcher did not work. A solution to this was rather unusual, but worked, and hence I am including it here. Do Ctrl-Alt_Del and b ring up the task manager. Under the top menu item 'File' thre is a run command. This worked and a command window was opened. It did not seem possible to change file attributes to make sure that a .exe was launched so evenually the PC was restored to a restore point from afew weeks earlier. Everything then nearly worked.
On examining the PC there were issue with McAfee antivirus which was not running, and also 18 months of Vista updates had not been loaded - 90 patches althogether. The 90 patches were installed, McAfee updated and this worked. PC now all OK, but all automatic updating was set to 4am when the PC is normally turned off. This was changed to a time when the PC was likely to be on and hopefully the problem will not arise again.
In both cases, no data was actually lost
The approach I tooks was to remove the drive from the laptop and create an image for security purposes. I then ran Norton which tracked down several viruses, and removed them. This is where the two viruses behaved differently.
Virus one did not want to be removed by Norton and started each time the machine booted. The problem was that a startup function (go to msconfig) was launching the virus at startup each time. By removing this start up line - and seeing the program it was starting (it had a randomly generated name) the PC was then OK. The free AVG virus checker was added to the PC to try and prevent this happening again. A report a few weeks after this event indicated that eveything has been OK.
Virus two was removed by Norton, but left the PC in a state where no program would actually launch from explorer. Various 'Googled' ideas pointed to the registry, but this did not help. Launching the command prompt was also very difficult and the start program launcher did not work. A solution to this was rather unusual, but worked, and hence I am including it here. Do Ctrl-Alt_Del and b ring up the task manager. Under the top menu item 'File' thre is a run command. This worked and a command window was opened. It did not seem possible to change file attributes to make sure that a .exe was launched so evenually the PC was restored to a restore point from afew weeks earlier. Everything then nearly worked.
On examining the PC there were issue with McAfee antivirus which was not running, and also 18 months of Vista updates had not been loaded - 90 patches althogether. The 90 patches were installed, McAfee updated and this worked. PC now all OK, but all automatic updating was set to 4am when the PC is normally turned off. This was changed to a time when the PC was likely to be on and hopefully the problem will not arise again.
In both cases, no data was actually lost
Thursday, March 3, 2011
Parallel Programming
Modern proessors are not getting much faster, 3GHz has been a fast processor for about 7 years. They are getting wider. It started with hyperthreading and then multiple cores. Almolst every processor is now a multiple core, and current Intel chips have 4 cores with hyperthreading, making in effect 8 cores.
Unfortunately, most programs just use a single core and so performance gains are not very significant. The solution is to use parallel programming so that different tasks are performed in different cores. This may sound simple but unfortunately many computing tasks are sequential. In data recovery it is sequence or read disk, analyse data and save data. The other problem is each time a task is split there is a processing overhead. This means that benefits may not be very significant.
An example of the limited benefit mentioned above is a simple program I wrote to experiment with parallel programming. It was purely an exercise with in memory manipulation - ie no hard disk access. The first example was single threaded and took 35 seconds to run, using a single core. The next example was using the 'parallel_invoke' function and used all possible cores. When running it looked impressive with all 8 cores running at 100%. However, the time was not reduced by a factor 8, but only roughly halved to 15 seconds. Although this would be a worth while time gain its shows how overheads of a new task eat into the gains. I am sure that a bit of tweaking could have made the improvement better, but the warning is that a PC may be running at 8 * 100% but actually alot of this may be house keeping.
In a real world example I have added some parallel processing into CnW Recovery software. The area was to do with calculating MD5 hash values while writing data to the output drive. As these processes do not depend on each other, they can run at the same time sharing the same memory buffer. The result was a reduction in time from about 3 hours 30 mins to 3 hours 10 mins. This is worth while but not very dramatic. However, it will be possible to add SHA-1 hashing with no extra time penalty and that would be a major benefit.
Unfortunately, most programs just use a single core and so performance gains are not very significant. The solution is to use parallel programming so that different tasks are performed in different cores. This may sound simple but unfortunately many computing tasks are sequential. In data recovery it is sequence or read disk, analyse data and save data. The other problem is each time a task is split there is a processing overhead. This means that benefits may not be very significant.
An example of the limited benefit mentioned above is a simple program I wrote to experiment with parallel programming. It was purely an exercise with in memory manipulation - ie no hard disk access. The first example was single threaded and took 35 seconds to run, using a single core. The next example was using the 'parallel_invoke' function and used all possible cores. When running it looked impressive with all 8 cores running at 100%. However, the time was not reduced by a factor 8, but only roughly halved to 15 seconds. Although this would be a worth while time gain its shows how overheads of a new task eat into the gains. I am sure that a bit of tweaking could have made the improvement better, but the warning is that a PC may be running at 8 * 100% but actually alot of this may be house keeping.
In a real world example I have added some parallel processing into CnW Recovery software. The area was to do with calculating MD5 hash values while writing data to the output drive. As these processes do not depend on each other, they can run at the same time sharing the same memory buffer. The result was a reduction in time from about 3 hours 30 mins to 3 hours 10 mins. This is worth while but not very dramatic. However, it will be possible to add SHA-1 hashing with no extra time penalty and that would be a major benefit.
Monday, February 7, 2011
WD disk not responding
A recnt Western Digital 500GB disk was spinning, but not recognised by a PC with USB connector. The next test I always try is with Ace Laboratories PC-3000 which is designed to test abd refirm driver firmware. At first this would not recognise the disk under general routines, but when the WD funtion tests were tried some response was found. It was not enough to recognise which family the drive belonged too, and so this was a case at looking at all the PCB layouts and matching a photo. Eventually the short list was 4 possible drive types, of which Tornado 2D was found to work.
The PC-3000 did a series of tests, and end of which it was possible to view a sector.
I did not know how long this reading would last so I started a Read to file and this moved very slowly, but positively. After a few hours the reading sped up to an acceptable rate. Over night over 200GB was imaged, but then the reading had gone painfully slow. I stopped the read and confirmed that other areas of the disk could be read without significant delays. The problem with PC-3000 is that it is not very good at reading sections of disks in an easily managed way.
The next stage was to carefully disconnect the SATA cable from PC-3000 and connect a PC (USB) SATA connector, without turning the drive power off. To my delight, the PC would read the disk anda process of incremental imaging was continued. I started with the final 200GB or so of the disk, and slowly worked back to the area that caused so many problems. When I had managed to image probably over 99% of the I decided the return rate was probably too small to consder worth continuing. The image produced a very healthy recovery of the disk, thanks both the PC-3000 and Incremental Imaging.
The PC-3000 did a series of tests, and end of which it was possible to view a sector.
I did not know how long this reading would last so I started a Read to file and this moved very slowly, but positively. After a few hours the reading sped up to an acceptable rate. Over night over 200GB was imaged, but then the reading had gone painfully slow. I stopped the read and confirmed that other areas of the disk could be read without significant delays. The problem with PC-3000 is that it is not very good at reading sections of disks in an easily managed way.
The next stage was to carefully disconnect the SATA cable from PC-3000 and connect a PC (USB) SATA connector, without turning the drive power off. To my delight, the PC would read the disk anda process of incremental imaging was continued. I started with the final 200GB or so of the disk, and slowly worked back to the area that caused so many problems. When I had managed to image probably over 99% of the I decided the return rate was probably too small to consder worth continuing. The image produced a very healthy recovery of the disk, thanks both the PC-3000 and Incremental Imaging.
Monday, January 31, 2011
Windows 7 update
I recently wrote about upgrading a PC from Vista to Windows 7. The cost was about £80 for software upgrade and £25 for extra memory.
It has been worth it. The machine is more reponsive and setting up network connections and printers much easier. The configuration is a 2.4GHz, Core 2 duo PC with 4GB RAM, 750GB hard drive, (250-500 partitions), a 2TB WD green drive, and a 2.7TB RAID. The only slow part to be investigated is the100Mb network.
The WD Green drives are not blistering fast, but they are very cool (in the sense of temperature). This should help reliablity, and mean that computer fans can run less often, giving me a quieter life.
Graphics are not important, so the PC has the standard graphics interface, but Windows 7 aero does work.
It has been worth it. The machine is more reponsive and setting up network connections and printers much easier. The configuration is a 2.4GHz, Core 2 duo PC with 4GB RAM, 750GB hard drive, (250-500 partitions), a 2TB WD green drive, and a 2.7TB RAID. The only slow part to be investigated is the100Mb network.
The WD Green drives are not blistering fast, but they are very cool (in the sense of temperature). This should help reliablity, and mean that computer fans can run less often, giving me a quieter life.
Graphics are not important, so the PC has the standard graphics interface, but Windows 7 aero does work.
Friday, January 28, 2011
Create a DVD with recovered video
Tuesday, January 25, 2011
Blank DVD-RW
I recently received 2 mini DVD-RW disks. When read with normal hardware, every sector was blank, and there were no error messages. However, when read with specialised hardware, it was possible to recovery about 20 mins of a wedding video from each disk.
The moral of this story is what looks blank, may not always be blank. Any forensic investigation should also take this into account. CnW Recovery have developed a method to examine and recover data from areas of a DVD that can not otherwise be accessed. CnW Recovery has a fixed price of £40, no fix, no fee for this type of data recovery. Contact info@cnwrecovery.co.uk for more details
The moral of this story is what looks blank, may not always be blank. Any forensic investigation should also take this into account. CnW Recovery have developed a method to examine and recover data from areas of a DVD that can not otherwise be accessed. CnW Recovery has a fixed price of £40, no fix, no fee for this type of data recovery. Contact info@cnwrecovery.co.uk for more details
Monday, January 24, 2011
Solved - problems copying files to a Mac from PC disk
Tuesday, January 18, 2011
Bye Bye Vista!
Doing software development it is essential that your product works on systems customers want to use. This means keeping upto date with standard updates and a few years ago I purchased a Vista system. It did thrown up a few compatibility problems so the purchase was worth while, but somehow the system never worked very well. The PC was a reasonable spec, Core Duo, 2.4GHz with 2GB or RAM.
However, enough has been enough, and my Windows 7 systems seem very stable, so I decided it was time to update the Vista to Windows 7 - 32bit. A search on Amazon brought up a reasonable price, and then a few more GBs or RAM to go to 4GB. The fitting of RAM was easy, though of course, 32 bit Windows only sees 3GBs.
In theory, Vista can be updated to Windows and keep the system intact. Obviously a full backup was done first, and the install was started. At this point a problem arose. The new Windows 7 package was less high end than the original Vista package, and this would not allow for a seamless upgrade. The notes implied that all data would be lost, and a clean install would be carried out. The truth was actually not quiet so bad. On starting the update, about 50GB of files were backed up to a windows.old directory, and this contained all programs and user directory. Obviously the programs are not installed, but copies are made. Unexpectedly, the rest of the hard drive was left unchanged, so all existing directories were left as original. I now just need to clear down a lot of the unwanted 50GB backup.
Overall, the upgrade was very painless and after a bit of personal tweaking, I now have a nice Windows 7 machine which seems to work.
However, enough has been enough, and my Windows 7 systems seem very stable, so I decided it was time to update the Vista to Windows 7 - 32bit. A search on Amazon brought up a reasonable price, and then a few more GBs or RAM to go to 4GB. The fitting of RAM was easy, though of course, 32 bit Windows only sees 3GBs.
In theory, Vista can be updated to Windows and keep the system intact. Obviously a full backup was done first, and the install was started. At this point a problem arose. The new Windows 7 package was less high end than the original Vista package, and this would not allow for a seamless upgrade. The notes implied that all data would be lost, and a clean install would be carried out. The truth was actually not quiet so bad. On starting the update, about 50GB of files were backed up to a windows.old directory, and this contained all programs and user directory. Obviously the programs are not installed, but copies are made. Unexpectedly, the rest of the hard drive was left unchanged, so all existing directories were left as original. I now just need to clear down a lot of the unwanted 50GB backup.
Overall, the upgrade was very painless and after a bit of personal tweaking, I now have a nice Windows 7 machine which seems to work.
Wednesday, January 5, 2011
Defragmenting videos from mobile phones
Mobile / Cell phones typically store videos as .3GP, .MP4 type files. These can be recognised easily in a hex viewer by looking at the start of the file. The second group of 4 bytes will be the string 'ftyp' followed by the exact type of file, eg '3gp5'.
If the files are to be recovered by data carving, then at times the data may be fragmented. CnW is working on a solution to this problem. Fortunately these Quick Time files have a fairly helpful data structure, and so it is possible to both verify, and hence reconstruct files from fragmented stored in different fragments. The basic file struct is 3 main data areas,
ftyp moov mdat
However, the order of moov and mdat can vary. The moov segment stores all pointers and decoding details for the data area, mdat. The mdat area basically contains frames of video and sound. As video frames normally start with the same header string, by decoding the moov it is possible to examine a possible fragment and determine if it does have the correct headers in the correct location. If a match is found then it is possible to apply this fragment to the new image.
The moov fragment is not normally very large and so the expectations are that the complete segment will be found in the first complete fragment, along with the ftyp header. This will be true for the files that store moov straight after the ftyp tag. For files which are ftype - mdat - moov sequence, it is necessary to examine the mdat for frame starts and hence calculate possible values that will be found in a moov segement. The disk then has to be searched for a suitable moov fragment.
The current status of CnW Recovery is for recovery of a ftyp - moov - mdat file. The second variation is under development. http://www.cnwrecovery.com/ for more details of software
If the files are to be recovered by data carving, then at times the data may be fragmented. CnW is working on a solution to this problem. Fortunately these Quick Time files have a fairly helpful data structure, and so it is possible to both verify, and hence reconstruct files from fragmented stored in different fragments. The basic file struct is 3 main data areas,
ftyp moov mdat
However, the order of moov and mdat can vary. The moov segment stores all pointers and decoding details for the data area, mdat. The mdat area basically contains frames of video and sound. As video frames normally start with the same header string, by decoding the moov it is possible to examine a possible fragment and determine if it does have the correct headers in the correct location. If a match is found then it is possible to apply this fragment to the new image.
The moov fragment is not normally very large and so the expectations are that the complete segment will be found in the first complete fragment, along with the ftyp header. This will be true for the files that store moov straight after the ftyp tag. For files which are ftype - mdat - moov sequence, it is necessary to examine the mdat for frame starts and hence calculate possible values that will be found in a moov segement. The disk then has to be searched for a suitable moov fragment.
The current status of CnW Recovery is for recovery of a ftyp - moov - mdat file. The second variation is under development. http://www.cnwrecovery.com/ for more details of software
Subscribe to:
Posts (Atom)