Data recovery is all about get data back when it has been lost. Often this is possible but there is one case when it just cannot be achieved. The problem is when the physical area,sector has been overwritten.
Whe a file is deleted, normally the index, or directory entry is removed. When a disk is formatted, generally only the key areas on the disk are written to. In both of these cases, old data will exist, and programs such as CnW Recovery will find the data.
When a sector is overwritten, the data is lost. If there are no backups of the data, there is nothing that can be done. There are suggestions that with a hard drive, unless data is overwritten 7 times, then it could be recovered - the author is extremely sceptical of this due to the exceptionally high density of current disks. What may have been possible 20 years ago, is no longer the case. For memory chips, it is also very definate. There is no scope of reading slightly off track and getting data back. It could be argued that routines that process wear leveling could point to old data, but this would only be a maximum of about 5% of the data, from any period of the disk. It will not contain a complete photo.
With cameras, because there is no editing, there is no requirement for any backups. The conclusion is that an overwritten photo is lost. Take care.
Monday, October 31, 2011
Wednesday, August 17, 2011
More FAT32 delete problems
I have just come across a FAT32 memory chip with deleted files. There is a well known issue that when FAT32 has deleted files, the upper 16 bits of the cluster address are blanked, though the lower 16 bits remain valid. CnW has developed routines to make use of the lower 16 bits, and with intelligence can recreate the upper 16 bits for know file types. The new chip was from a video recorder, and all 32 bits have been deleted. This means there is no information on the location of the file.
Fortunately the FAT directory entry does contain the file length. The only way that file names can be associated with files is to data carve the disk and then try and match file lengths. This is far from optimum, but does provide a partial solution to an other impossible problem.
Fortunately, FAT32 is now largely used for removeable storage and typically for one type of file, eg video, music or photos. File names are not always critical, and data carving can produce reasonable results.
Fortunately the FAT directory entry does contain the file length. The only way that file names can be associated with files is to data carve the disk and then try and match file lengths. This is far from optimum, but does provide a partial solution to an other impossible problem.
Fortunately, FAT32 is now largely used for removeable storage and typically for one type of file, eg video, music or photos. File names are not always critical, and data carving can produce reasonable results.
Monday, July 4, 2011
More Digital Signature issues with IE9.0 and Norton
My last posting talked about digitally signing software for downloads. Unfortunately this is only half of the solution. I am sure that Microsoft and Norton are working in most people's best interests, but this does not help the way that CnW operates. With CnW, the idea is to give users the latest updates (and even bug fixes) as soon as possible. This is done by updating the main software package maybe once or twice a month. The changes are small, but incrementally ensure that the produce evolves, and responds to customer feed back.
Part of IE9.0 and Norton security is how long any download has been on the internet. This period is monitored to see if there has been any negative feedback about the product. Norton suggests there is a 1 week learning time, which would man that CnW software would be viewed as dubious for maybe 50% of it's time.
The next plan to try and resolve this issue is to move to the common pattern of having a installer tool, and then in effect an update program. In this case, the installer can be written, and will not be changed for a considerable period of time, maybe a year. The installer will then download the main, and frequently updated data from it's own secure environment. Everything will be digitally signed, but the installer can be recognised by IE9.0 and Norton as a stable, safe program. IE9.0 and Norton will only see the stable installer, though Norton will obviously also monitor any future updates.
Part of IE9.0 and Norton security is how long any download has been on the internet. This period is monitored to see if there has been any negative feedback about the product. Norton suggests there is a 1 week learning time, which would man that CnW software would be viewed as dubious for maybe 50% of it's time.
The next plan to try and resolve this issue is to move to the common pattern of having a installer tool, and then in effect an update program. In this case, the installer can be written, and will not be changed for a considerable period of time, maybe a year. The installer will then download the main, and frequently updated data from it's own secure environment. Everything will be digitally signed, but the installer can be recognised by IE9.0 and Norton as a stable, safe program. IE9.0 and Norton will only see the stable installer, though Norton will obviously also monitor any future updates.
Friday, July 1, 2011
Software signing and IE9.0
Internet security can be a large issue. To this end, IE9.0 has very 'tough' security measures added to try and protect users from downloading unsafe software. If a user tries to download an unsigned program, there are severe warning messages and it can be difficult to download or run the program.
To overcome the above problem the program has to be digitally signed, with a secure certificate. This can be an expensive process but fortunately there are solutions for small companies. One such company - that CnW have used - is Global Sign. This gives a digital signature which can then be traced back to Global Sign. Having this on the program now means that Internet Explorer V9 does not scream that the software could be very dangerous.
The other security measure that is causing problems is Norton Insight. This can warn that the software is new, and may be dangerous. The solution can be to get listed on their White List (is not a Black List). The problem here is that listing can take a few weeks, and with CnW this is when the next release may have been made. If a company does a single release each year this is not a problem, but for a small company doing regular updates to keep pace with new solutions, and customer requests, then Norton is not very friendly. Unfortunately, Norton does not seem to recognise the digital signature. I do not know what the ultimate solution will be.
To overcome the above problem the program has to be digitally signed, with a secure certificate. This can be an expensive process but fortunately there are solutions for small companies. One such company - that CnW have used - is Global Sign. This gives a digital signature which can then be traced back to Global Sign. Having this on the program now means that Internet Explorer V9 does not scream that the software could be very dangerous.
The other security measure that is causing problems is Norton Insight. This can warn that the software is new, and may be dangerous. The solution can be to get listed on their White List (is not a Black List). The problem here is that listing can take a few weeks, and with CnW this is when the next release may have been made. If a company does a single release each year this is not a problem, but for a small company doing regular updates to keep pace with new solutions, and customer requests, then Norton is not very friendly. Unfortunately, Norton does not seem to recognise the digital signature. I do not know what the ultimate solution will be.
Friday, June 3, 2011
Solid State devices
Many products are deemed to be dead long before they eventualy die entirely. Magnetic tape for datya storage is one, but there is a very important market for tape, though for the home user and small business, disk back up is now a much more viable proposition. The next prediction is the death of the rotating hard drive.
We are all used to solid state memory for cameras, telephones and many video devices. The capacity keeps increasing, and cost keeps coming down. Just starting to come in are solid state drives for laptops and desk top computers. As there is no head seek time, or rotational delay, reading can be very fast and many users are added them as the system drive in a PC. They report impressive performance in boot up and launching programs. It is noit all good news, as writing can be slow, and there is an issue of limited write cycles. Basically, sector will wear out if used too often. The solution to the last point is that chip controllers use a feature called wear leveling, so when a sector has been used too many times, it will be physically moved to another location, while still keeping the same logical address.
Data recovery of such SSDs has two main problems. If the controller dies, then it is necessary to work with the chips directly. These means they have to be unsoldered from the board and read. This means pointing a hot air gun at the chips and removing them, with out damage or over heating. It is possible, and not quite as bad as it sounds. The major problem though comes next. Manufacturers do not publish their wear leveling routines. As the chips are not meant to moved between devices, there is no requirement for any standards - all that matters is that when a sector number is requested, the correct data is returned. The physical sector is not relevant in any way.
CnW are now looking at such drives, and memory chips and will be developing tools to assist with recovery
We are all used to solid state memory for cameras, telephones and many video devices. The capacity keeps increasing, and cost keeps coming down. Just starting to come in are solid state drives for laptops and desk top computers. As there is no head seek time, or rotational delay, reading can be very fast and many users are added them as the system drive in a PC. They report impressive performance in boot up and launching programs. It is noit all good news, as writing can be slow, and there is an issue of limited write cycles. Basically, sector will wear out if used too often. The solution to the last point is that chip controllers use a feature called wear leveling, so when a sector has been used too many times, it will be physically moved to another location, while still keeping the same logical address.
Data recovery of such SSDs has two main problems. If the controller dies, then it is necessary to work with the chips directly. These means they have to be unsoldered from the board and read. This means pointing a hot air gun at the chips and removing them, with out damage or over heating. It is possible, and not quite as bad as it sounds. The major problem though comes next. Manufacturers do not publish their wear leveling routines. As the chips are not meant to moved between devices, there is no requirement for any standards - all that matters is that when a sector number is requested, the correct data is returned. The physical sector is not relevant in any way.
CnW are now looking at such drives, and memory chips and will be developing tools to assist with recovery
Thursday, May 26, 2011
Backup procedures
I often meet customers who say - I was about to back it up but....
Last week I suffered a serious problem with a Window 7 64 PC. It had become corrupted - probably due to device drivers - in a way that it would not respond to either mouse or keyboard. It booted up, I could see it over the network, but not control anything.
It is a Dell computer, and did not come with a Windows disk, but I had made a repair disk. I managed to boot into repair mode and first tried to go back a few restore points, however nothing worked.
I have a few backup prodcures in place, so was not too worried. The methods I have are
The first stage was to use the Microsoft image which is created evry week. The recovery mode allows for this to be restored, but it did not want to recognise the RAID box. However, the files were copied onto a USB drive, and was then visible. In the mean time, I took a complete image copy of the problem drive so that all updated files could be recovered as required.
The disk was then updaed with the microsft image, but this would not produce a bootable drive. It always came up in recovery mode, and often indicated that there were bad directories etc.
The final stage was to restore the partition, and boot sector from the Macrium backup. This was a few months old, but gave the pomise of a working system. I was very pleased that this worked, and it immediately booted and started running. The system then spent some time updating a few months of Norton and Microsoft patches. The recent file were copied back, some from disk, and some from carbonite, and all is now working correctly. No files or data lost.
My concern remains on how good the Microsoft backup system is. In the next few months I will try and receate a complete backup and see if it works.
Last week I suffered a serious problem with a Window 7 64 PC. It had become corrupted - probably due to device drivers - in a way that it would not respond to either mouse or keyboard. It booted up, I could see it over the network, but not control anything.
It is a Dell computer, and did not come with a Windows disk, but I had made a repair disk. I managed to boot into repair mode and first tried to go back a few restore points, however nothing worked.
I have a few backup prodcures in place, so was not too worried. The methods I have are
- On line carbonite
- Weekly Microsoft backup to a local RAID - in a separate box
- Periodic disk images using Macrium Reflect
- Very critical code (my source code) is backed up every 8 hours onto another PC, local but in a different room.
The first stage was to use the Microsoft image which is created evry week. The recovery mode allows for this to be restored, but it did not want to recognise the RAID box. However, the files were copied onto a USB drive, and was then visible. In the mean time, I took a complete image copy of the problem drive so that all updated files could be recovered as required.
The disk was then updaed with the microsft image, but this would not produce a bootable drive. It always came up in recovery mode, and often indicated that there were bad directories etc.
The final stage was to restore the partition, and boot sector from the Macrium backup. This was a few months old, but gave the pomise of a working system. I was very pleased that this worked, and it immediately booted and started running. The system then spent some time updating a few months of Norton and Microsoft patches. The recent file were copied back, some from disk, and some from carbonite, and all is now working correctly. No files or data lost.
My concern remains on how good the Microsoft backup system is. In the next few months I will try and receate a complete backup and see if it works.
Friday, May 6, 2011
DVD+RW apparently blank
I recently had a DVD+RW for recovery. It was from a camcorder and had probably been reformatted.
On initial examination the start of the disk was OK, but the majority was blank, ie all the sectors were filled with zeros. I could read the sectors and no error messages were displayed. Every indication was that the disk had been blanked, as if thee had been a full format.
When the disk was examined on hardware designed to read blank disks it was very interesting that the disk was not blank. There was a significant amount of video still on the disk and CnW software did a recovery, and then generated a new video disk.
The concern is that standard hardware gave every indication that the disk was blank - so do not rely on standardard hardware if there is a possibility that the data may actually still exist. CnW Recovery services will always assist anyone with such possible disks.
On initial examination the start of the disk was OK, but the majority was blank, ie all the sectors were filled with zeros. I could read the sectors and no error messages were displayed. Every indication was that the disk had been blanked, as if thee had been a full format.
When the disk was examined on hardware designed to read blank disks it was very interesting that the disk was not blank. There was a significant amount of video still on the disk and CnW software did a recovery, and then generated a new video disk.
The concern is that standard hardware gave every indication that the disk was blank - so do not rely on standardard hardware if there is a possibility that the data may actually still exist. CnW Recovery services will always assist anyone with such possible disks.
Tuesday, April 26, 2011
Finding owner of data
On a recent walk along a Californian beach I found a compact camera, covered in sand and salt water. The camera was very dead, but out of interest I eventally managed to extract the 2GB SD memory card. After drying it out, it was readable with errors. When totally dry, it read without errors.
I was keen to try and contact the owner and return the photos, over 500 of them. Unfortunately, there is very little on a memory chip to tell you about an owner, only the camera, and date and time of the photos. From this I could determine that the last photo had been taken about 10 days before I found the camera. I did try and add a 'Found camera' to a local lost and found website, but no reply.
Recently, I did a bit more investigation by looking at the photos more closely. Obviously the owner was a young person, with lots of photos in night clubs, no names identifiable, but also some at a college. One photo though I do hope will be a major clue is that it includes, possibly the owner, or close friend holding a college certificate with their unusal surname name on it. A bit of Googling took me to Facebook, and I hope a perfect match. (The name and college course both match, and the photo looks similar). I have now sent a message to the person, and am waiting for a reply.
The moral of the story is that if you want goods returned, it can be helpful to have some return details stored with it.
As a post script, the owner did contact me and a DVD of photos was sent. I have just received a very nice letter of thanks.
I was keen to try and contact the owner and return the photos, over 500 of them. Unfortunately, there is very little on a memory chip to tell you about an owner, only the camera, and date and time of the photos. From this I could determine that the last photo had been taken about 10 days before I found the camera. I did try and add a 'Found camera' to a local lost and found website, but no reply.
Recently, I did a bit more investigation by looking at the photos more closely. Obviously the owner was a young person, with lots of photos in night clubs, no names identifiable, but also some at a college. One photo though I do hope will be a major clue is that it includes, possibly the owner, or close friend holding a college certificate with their unusal surname name on it. A bit of Googling took me to Facebook, and I hope a perfect match. (The name and college course both match, and the photo looks similar). I have now sent a message to the person, and am waiting for a reply.
The moral of the story is that if you want goods returned, it can be helpful to have some return details stored with it.
As a post script, the owner did contact me and a DVD of photos was sent. I have just received a very nice letter of thanks.
Sunday, March 6, 2011
Virus issue
I don't often get involved in fixing PC problems, but recent jobs involved the same type of issue. The PC starts to report serious errors, and show problems with boot sector and other critical areas of the disk. It all looks rather real, and very worrying. However, in both cases it was caused by a similar, but different Fake Anti Virus program. I think if you continue there was an offer of the program to fix the issue - in exchange for some $$$s.
The approach I tooks was to remove the drive from the laptop and create an image for security purposes. I then ran Norton which tracked down several viruses, and removed them. This is where the two viruses behaved differently.
Virus one did not want to be removed by Norton and started each time the machine booted. The problem was that a startup function (go to msconfig) was launching the virus at startup each time. By removing this start up line - and seeing the program it was starting (it had a randomly generated name) the PC was then OK. The free AVG virus checker was added to the PC to try and prevent this happening again. A report a few weeks after this event indicated that eveything has been OK.
Virus two was removed by Norton, but left the PC in a state where no program would actually launch from explorer. Various 'Googled' ideas pointed to the registry, but this did not help. Launching the command prompt was also very difficult and the start program launcher did not work. A solution to this was rather unusual, but worked, and hence I am including it here. Do Ctrl-Alt_Del and b ring up the task manager. Under the top menu item 'File' thre is a run command. This worked and a command window was opened. It did not seem possible to change file attributes to make sure that a .exe was launched so evenually the PC was restored to a restore point from afew weeks earlier. Everything then nearly worked.
On examining the PC there were issue with McAfee antivirus which was not running, and also 18 months of Vista updates had not been loaded - 90 patches althogether. The 90 patches were installed, McAfee updated and this worked. PC now all OK, but all automatic updating was set to 4am when the PC is normally turned off. This was changed to a time when the PC was likely to be on and hopefully the problem will not arise again.
In both cases, no data was actually lost
The approach I tooks was to remove the drive from the laptop and create an image for security purposes. I then ran Norton which tracked down several viruses, and removed them. This is where the two viruses behaved differently.
Virus one did not want to be removed by Norton and started each time the machine booted. The problem was that a startup function (go to msconfig) was launching the virus at startup each time. By removing this start up line - and seeing the program it was starting (it had a randomly generated name) the PC was then OK. The free AVG virus checker was added to the PC to try and prevent this happening again. A report a few weeks after this event indicated that eveything has been OK.
Virus two was removed by Norton, but left the PC in a state where no program would actually launch from explorer. Various 'Googled' ideas pointed to the registry, but this did not help. Launching the command prompt was also very difficult and the start program launcher did not work. A solution to this was rather unusual, but worked, and hence I am including it here. Do Ctrl-Alt_Del and b ring up the task manager. Under the top menu item 'File' thre is a run command. This worked and a command window was opened. It did not seem possible to change file attributes to make sure that a .exe was launched so evenually the PC was restored to a restore point from afew weeks earlier. Everything then nearly worked.
On examining the PC there were issue with McAfee antivirus which was not running, and also 18 months of Vista updates had not been loaded - 90 patches althogether. The 90 patches were installed, McAfee updated and this worked. PC now all OK, but all automatic updating was set to 4am when the PC is normally turned off. This was changed to a time when the PC was likely to be on and hopefully the problem will not arise again.
In both cases, no data was actually lost
Thursday, March 3, 2011
Parallel Programming
Modern proessors are not getting much faster, 3GHz has been a fast processor for about 7 years. They are getting wider. It started with hyperthreading and then multiple cores. Almolst every processor is now a multiple core, and current Intel chips have 4 cores with hyperthreading, making in effect 8 cores.
Unfortunately, most programs just use a single core and so performance gains are not very significant. The solution is to use parallel programming so that different tasks are performed in different cores. This may sound simple but unfortunately many computing tasks are sequential. In data recovery it is sequence or read disk, analyse data and save data. The other problem is each time a task is split there is a processing overhead. This means that benefits may not be very significant.
An example of the limited benefit mentioned above is a simple program I wrote to experiment with parallel programming. It was purely an exercise with in memory manipulation - ie no hard disk access. The first example was single threaded and took 35 seconds to run, using a single core. The next example was using the 'parallel_invoke' function and used all possible cores. When running it looked impressive with all 8 cores running at 100%. However, the time was not reduced by a factor 8, but only roughly halved to 15 seconds. Although this would be a worth while time gain its shows how overheads of a new task eat into the gains. I am sure that a bit of tweaking could have made the improvement better, but the warning is that a PC may be running at 8 * 100% but actually alot of this may be house keeping.
In a real world example I have added some parallel processing into CnW Recovery software. The area was to do with calculating MD5 hash values while writing data to the output drive. As these processes do not depend on each other, they can run at the same time sharing the same memory buffer. The result was a reduction in time from about 3 hours 30 mins to 3 hours 10 mins. This is worth while but not very dramatic. However, it will be possible to add SHA-1 hashing with no extra time penalty and that would be a major benefit.
Unfortunately, most programs just use a single core and so performance gains are not very significant. The solution is to use parallel programming so that different tasks are performed in different cores. This may sound simple but unfortunately many computing tasks are sequential. In data recovery it is sequence or read disk, analyse data and save data. The other problem is each time a task is split there is a processing overhead. This means that benefits may not be very significant.
An example of the limited benefit mentioned above is a simple program I wrote to experiment with parallel programming. It was purely an exercise with in memory manipulation - ie no hard disk access. The first example was single threaded and took 35 seconds to run, using a single core. The next example was using the 'parallel_invoke' function and used all possible cores. When running it looked impressive with all 8 cores running at 100%. However, the time was not reduced by a factor 8, but only roughly halved to 15 seconds. Although this would be a worth while time gain its shows how overheads of a new task eat into the gains. I am sure that a bit of tweaking could have made the improvement better, but the warning is that a PC may be running at 8 * 100% but actually alot of this may be house keeping.
In a real world example I have added some parallel processing into CnW Recovery software. The area was to do with calculating MD5 hash values while writing data to the output drive. As these processes do not depend on each other, they can run at the same time sharing the same memory buffer. The result was a reduction in time from about 3 hours 30 mins to 3 hours 10 mins. This is worth while but not very dramatic. However, it will be possible to add SHA-1 hashing with no extra time penalty and that would be a major benefit.
Subscribe to:
Posts (Atom)