www.ice-graphics.com Forum Index www.ice-graphics.com
The main forum for the ICE-Graphics software
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Immune to Shifted Data Problem

 
Post new topic   Reply to topic    www.ice-graphics.com Forum Index -> Bugs and Problems
View previous topic :: View next topic  
Author Message
NvrBst



Joined: 05 Aug 2009
Posts: 6

PostPosted: Thu Aug 06, 2009 8:46 pm    Post subject: Immune to Shifted Data Problem Reply with quote

This may just be my lack of understand on what "Immune to Shift" means, but, from my understanding I should be able to shift the data of a file left or right and ICC should be able to fix it with minimal blocks.

Quote:
8. ICE ECC uses an algorithm that will allow it to find shifted data regardless of the distance by which it has been shifted and will work with any block size. Both .ecc files and data files are immune to shift.


However, in my test case I have the following situation: A folder with 4 files totaling 414MB. 1 file is 4MB another is 165MB. I make an ECC File for the folder with 16k Block Count and 1% Recovery. It totals about 5.5MB.

If I increase and/or decrease the beginning and end of the 4MB file then Verify, it is able to fix the folder with 1 or 2 blocks.

If I increase or decrease the end of the 165MB file then Verify, it is able to fix the folder with 0 or 1 blocks.

If I increase or decrease the start of the 165MB file then Verify, it is unable to fix the folder. It states it needs 25% of recovery data.


In my tests I was increasing or decreasing by 100 bytes of "0"'s.

Is this a bug? Should it be able to repair the 165MB file in the folder if all the bytes of the file were shifted, for example, right by 100 bytes? I've tried just increasing the start by 100 bytes, and I've also tried increasing start by 100 bytes + decresing end by 100 bytes, so the filesize is the same as before. Both fail and end up needed 25% recovery files to fix the 414MB folder.

Thanks. Please help me understand the "Immune to Shift" feature. Does it only work with small files (Because it seems to work with the 4MB file fine). What is the limit of the size of the file that "Immune to Shift" works on if this is the case?
Back to top
View user's profile Send private message
ICE Graphics
Site Admin


Joined: 31 Mar 2003
Posts: 430

PostPosted: Fri Aug 07, 2009 7:39 am    Post subject: Re: Immune to Shifted Data Problem Reply with quote

If there is block filled from multiple files, that block can not be found. So, if you will shift large data file ICE ECC will find only whole blocks allocated inside this is file. First and last block of this is file can not be found.

About your case. At first look you made some mistake during testing. Probably in case then your test fail you copied wrong file. So, please repeat your test. If test will fail again, please give detailed description how to reproduce this is fail. Like this:
1. Data files: 01.txt (Size 1234567), 02.txt (Size 2345678), ...
2. Create ECC file, 1024 block, 10% redundancy
3. copy /b Dummy.txt+02.txt 02a.txt
4. del 02.txt
5. ren 02a.txt 02.txt
6. Recover data files ...

Only detailed instruction can help to resolve problem.
Back to top
View user's profile Send private message Visit poster's website
NvrBst



Joined: 05 Aug 2009
Posts: 6

PostPosted: Sat Aug 08, 2009 8:17 pm    Post subject: Reply with quote

Before I did it completely in the GUI, but, I thought I'd try command line this time. Here is what I do to reproduce the error:

Before Directory:
Code:
E:\Upload\z>dir /s /A-D
 Directory of E:\Upload\z

08/08/2009  12:24 PM               100 100byte.bin
               1 File(s)            100 bytes

 Directory of E:\Upload\z\RepairFolder

07/11/2009  01:34 AM               387 RepairFile.cue
07/08/2009  11:04 AM       257,574,576 RepairFile1.bin
07/08/2009  11:03 AM         3,175,200 RepairFile2.bin
07/08/2009  11:03 AM       173,754,000 RepairFile3.bin
               4 File(s)    434,504,163 bytes


Commands to Reproduce Bug - I Use Portable Version, not the Installer Version:
E:\Upload\z\RepairFolder>"C:\Program Files\zzzICEECC [2.7]\ICEECC.exe" c /b8192 /r1 RepairFolder.ecc
E:\Upload\z\RepairFolder>move RepairFile3.bin ../
E:\Upload\z\RepairFolder>copy /b ..\100byte.bin+..\RepairFile3.bin RepairFile3.bin
E:\Upload\z\RepairFolder>"C:\Program Files\zzzICEECC [2.7]\ICEECC.exe" vf RepairFolder.ecc


After Directory:
Code:
E:\Upload\z>dir /s /A-D
 Directory of E:\Upload\z

08/08/2009  12:24 PM               100 100byte.bin
07/08/2009  11:03 AM       173,754,000 RepairFile3.bin
               2 File(s)    173,754,100 bytes

 Directory of E:\Upload\z\RepairFolder

07/11/2009  01:34 AM               387 RepairFile.cue
07/08/2009  11:04 AM       257,574,576 RepairFile1.bin
07/08/2009  11:03 AM         3,175,200 RepairFile2.bin
08/08/2009  12:47 PM       173,754,100 RepairFile3.bin
08/08/2009  12:44 PM         5,204,420 RepairFolder.ecc
               5 File(s)    439,708,683 bytes


The Verify Log File: (25.5% Repair Needed)
Code:
ICE ECC v2.7
Physical Memory - 511 MB
Memory Usage - 306 MB

Size - 65536
Speed - 512.57 MB/Sec, 515.69 MB/Sec, 452.83 MB/Sec, 452.29 MB/Sec
Scale - 0.883, 0.877
Configuration - 9, 2, 6, 5, 4
Command line executed - "vf RepairFolder.ecc"
Measure speed finished

Verify only started: E:\Upload\z\RepairFolder; E:\Upload\z\RepairFolder\RepairFolder.ecc
Check - RepairFolder.ecc
Use ECC file - E:\Upload\z\RepairFolder\RepairFolder.ecc
Verify Data Files
Damaged block - 4923
Damaged block - 4924
Damaged block - 4925
Damaged block - 4926
Damaged block - 4927
Damaged block - 4928
Damaged block - 4929
[[SNIPED TO REDUCE SIZE]]
Damaged block - 7532
Damaged block - 7533
Damaged block - 7534
Damaged block - 7535
Damaged block - 7536
Damaged block - 7537
Damaged block - 7538
Damaged block - 7539
Wrong file size - E:\Upload\z\RepairFolder\RepairFile3.bin
Verify ECC Files
Search - RepairFile2.bin
Total bad blocks - 2089
Percent of damaged blocks - 2089 / 25.50048828125%
Not Enough Recovery Blocks - 81 / 2089
Files damaged. Recover is NOT possible
Verify only failed



Notes1: If I change the copy command so it copies at the end then it repairs fine.
Notes2: If I change the file used to the 4MB file then it works fine with 1 block.



While using the command version I found that the "/d" switch doesn't seem to be working. The following two lines don't work when I believe they should be working:
[[BUG]]
E:\Upload\z>"C:\Program Files\zzzICEECC [2.7]\ICEECC.exe" c /b8192 /r1 RepairFolder.ecc /dE:\Upload\z\RepairFolder *
ICEEEC runs with Datapath as "E:\Upload\z", No Errors Given in Log.

[[BUG]]
E:\Upload\z>"C:\Program Files\zzzICEECC [2.7]\ICEECC.exe" vf RepairFolder.ecc /dE:\Upload\z\RepairFolder
ICEEEC runs with Error "Wrong command line parameter - "/dE:\Upload\z\RepairFolder"". Datapath set to "E:\Upload\z"


Last edited by NvrBst on Sun Aug 09, 2009 6:47 pm; edited 1 time in total
Back to top
View user's profile Send private message
ICE Graphics
Site Admin


Joined: 31 Mar 2003
Posts: 430

PostPosted: Sun Aug 09, 2009 7:26 am    Post subject: Re: Immune to Shifted Data Problem Reply with quote

Thx for information. It's bug. ICE ECC skip search shifted data for some files.

Just rename file "RepairFile3.bin" to "RepairFile3a.bin" before verify and test verify again.

New version will soon be available.
Back to top
View user's profile Send private message Visit poster's website
Display posts from previous:   
Post new topic   Reply to topic    www.ice-graphics.com Forum Index -> Bugs and Problems All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group