Hi,
Recently I downloaded Quickpar and found it very interesting. Before this I never used any redundancy and recovery software. From this forum I came to know about ICE-ECC also. I downloaded that too. I tested both extensively. The problem with ICE-ECC is somewhat non-standard format, non-open software, and very much RAM dependent. It will start thrashing disk (virtual Paging file) and show infinite completion time for creating parity files for large source.
Now coming to Quickpar. There are only two shortcomings I found and there is a high need to overcome these. These are:
1. NO UNICODE FILE NAME SUPPORT.
2. NO FOLDER AND SUBFOLDER RECURSION.
I tried the batch script by Dennis Levens. But that is also not impressive as it uses source block size of 250000 which makes unnecessarily large files for small sized source files. Moreover the cmd line version 0.4 does not adjust block size. If say, block size of 2000 is given but source file is less than 2000x4 bytes, block size of 4 shall be used that is very un-economical as it creates large sized recovery files.
Now since so many years have already passed since the last release. Kindly remove these shortcomings.
Thankyou.
Offline
Use Multipar or ICEECC.
First not yet stable and under continuous development.
ICEECC is good but has their own nonPAR2 format
Google for search.
Offline
Dear Persicum,
Thanks for intimating about Multipar. After lot of Googling I had to return back to this forum.
I found the URL (http://hp.vector.co.jp/authors/VA021385) for MultiPar by Yutaka Sawada and downloaded it.
I found the program really good.
How do people of this forum came to know about such a nice program when it is not available through Google Search?
I shall post my observations on MultiPar to help other people.
Offline
My Observations on Multipar 1.1.3:
----------------------------------
Plus Points w.r.t QuickPar 0.9.1:
---------------------------------
<> Full directory depth and Unicode file name support.
<> PAR2 files created by Quickpar can be worked with Multipar and vice versa. However, this is possible if sub-folders and Unicode names are not involved.
<> Creates files "_x.ini" and "_x.bin", where x is 32 character hex number (probably md5 hash)) in the installation directory. I shall call these files as "DATE FILE". These file pairs are created when par2 files are verified first time (not when creating par2 files. Probably these files contain date-time stamps with hash of the source files associated to the PAR2 file(s). First verification takes time and if the above associated file pair is not deleted all further verification shall be done quickly.
No. of these DATE FILE pairs and verification depends upon the set options for verification.
<> Explorer Activity, Verification Window:
Supports good explorer activity - File Open, Copy, Delete and Property on right click from its Verification window. One file can be selected. Drag & Drop is not supported.
<> Explorer Activity, Creation Window:
Supports good explorer activity - Add Files, Remove Files, Paste Files and File property on right click in its Create Window. Multiple files can be selected. Ctrl+A (select all): No; Drag & Drop supported (but don't drop a folder).
Quickpar Create: Right Click = No; Multiple file select: Yes; Drag & Drop: Yes; Ctrl+A (select all): No.
<> Help file are organized in folders.
Minus Points w.r.t QuickPar 0.9.1:
----------------------------------
<> F5 (refresh) feature is not available for verification dialog. If actual verification is required delete the associated "DATE FILE" in the installation folder.
<> F1 key does not launches help. Click About > Help.
<> The verification, Creation and restoration (Recover) time are displayed respectively. But, estimated completion time is not displayed.
<> Shell integration is not available, however, "send to" feature is available for upto four items (Enable it through options). Following steps are required to use this feature appropriately:
a) Select one of the required files/folders, right click and send to Multipar.
b) Click "Reset List".
c) Click Browse (Base Directory) and select the required folder.
<> File with Unicode file name can not be "sent to" Multipar. Drag & Drop or open these.
<> PAR2 files shall not be created if a folder is also dropped (along with files or alone) in Create Window. Use Browse button to prevent this.
<> The product of Sourct Block Count (SBC) and Source Block Size (BS) is very unpredictable and varies to large extent if any one is varied. This makes calculation for my "SBC is y% of BS" method difficult.
<> Takes long time to repair. 27 missing blocks (1 file) from the set of 2615 blocks (8 files) with block size of 260876 (Total files size: 681,110,240) took 00:02:50 (RAM used Option: 1/2) whereas Quickpar took just 00:00:36. That is, Quickpar is about 5 times faster for recovering data.
Creation time is almost same. For same source data files (above 8 files, same block size etc.), same environmental conditions (no other programs running, computer not touched while creation). OS: Windows Vista.
Quickpar 0.9.1: 00:04:44 Powers of 2 = 10 files.
Multipar 1.1.3: 00:04:30 Powers of 2 = 09 files.
Thus, its OK to use Multipar, since, recovery of data shall be seldom used.
BUGS:
-----
1. Multipar 1.1.3 can not make recovery-PAR2 file if the difference of Product of Source Block Count (SBC) with Block Size (BS) and Total file size (FS) is less than or equal to 4. That is if,
SBC x BS - FS < 5
Multipar will crash.
QuickPar has no such problem.
2 Due to above reason only base-PAR2 file can be made for small file size (upto 8 bytes total). Base-PAR2 file can recover such small file(s) alone (without the help of recovery-PAR2 file) unless the file does not contain any unicode character.
Offline
Thank you for your observations!
But Multipar also supports RS_FFT mode which is much much faster than PAR2
Offline
Multipar RS_FFT probably requires just 15 sec to treat your 600M instead 4min of PAR2. Moreover, you may set SBC to 10000 or 20000 easily. Multipar supports PAR2 for legacy only = )))
Last edited by persicum (2010-02-02 06:34:42)
Offline
Dear Persicum,
I tested par3j.exe for RS_FFT. Indeed, its really fast. But, it is no use making recovery files that may not recover in disasters.
I made recovery file using very small blocks size (more the blocks more repairs the recovery files can do), to make total block count of about 65000. Although, the recovery files were made, they could not recover just 2 blocks in the original file (I had damaged two blocks intentionally using a hex editor, by replacing bytes with some other sequence). Par3j.exe kept crashing.
Moreover, as par3j.exe and PAR3 specifications are still not mature according to the programmer, I shall stick to par2j.exe and its GUI only.
Thanks, however for the info.
Last edited by Sunny Saini (2010-02-04 23:32:33)
Offline
You also can play with prog named persicum’s CRC32 ver2.21. But it is OEM-ANSI console apps. Yesterday I added dual thread here, it was really very easy so I could do that two years ago (as well as quickPAR creators = ))). Do not be hypnotized very impressive by PAR3 syllable, it may seem that a team of twenty people works hard under development, but PAR3 is really completely a one man made product. Although he is the most talented PAR3 develover, but he is indeed just an amateur of plain C and knows nothing about C++ classes = ))) So am I, an amateur of plain PASCAL.
It is too bad that RS(65537) from MultiPAR appears glitchfull = ((( But Yutaka obligatory will fix that.
Offline
From where can I get CRC32 v2.21?
Kindly mention your website or mail me the program.
Does it also works in real mode DOS?
If not, kindly make one for real mode DOS. (Note that the old pkzip stores CRC32 in its zip files made in Real mode DOS). I could not find such application (neither md5 nor crc32) for Real mode DOS on Google.
Offline
http://file.qip.ru/file/109465225/ed7e7ff/crc32.html
You can download this prog out there.
I am afraid this prog cannot run in plain DOS mode, because it is essentially 32-bit app (although some extenders exist), requires Win32 API, requires long 64-bit access files and long filenames.
Some precautions before running this prog:
1) Default memory usage is 1.2G. Normally this never causes page-swapping on 2G memory machines, but you can set desirable memory usage by –mu500m for instance
2) The prog requires 2x free HDD space for encoding and 3x while decoding.
3) Even single byte patching will cause a great recovery process as it were 100M of losses
))
4) While testing see that Transform Efficiency Indicator to be about 80-100% and not to be 50-60%, change SBC to achieve that or it will work at a half of possible speed.
Even if you feel that the prog is not for your everyday usage, I need much experienced testers like you to check wherever it stable or not. I see you understand greatly what is SBC but most people understand only what is redundancy. The irony is that a progress is endless, currently I know several algos which are even faster than default algo in my prog. If I to be alive I will try to implement some of them soon, so keep in touch!!!
Offline
OK, I will try to compile light version of my prog for 16-bit DOS. This will be CRC32, MD5 and SHA1. But not very soon...
Offline
Dear Persicum,
I have downloaded your program.
The bechmarks indicated in your benchmark file are very good.
There is one suggestion, that you change the name of your program. CRC32 indicates of some simple checksum program, whereas, your program is full suit that can create/ restore recovery files and copy portions of bad files from bad media. Some suggested names are "PersiRC", "PersiRC32", etc. Also it would be nice to change the Language of your website to English so that most people in the world can read it. (I pressed the green button to download the program without understanding what I am doing).
What really amaze me is that both of your program files have CRC32 equal to 77777777 (Although md5 is different). This is really very surprizing and can not be just coincidence. It seems you really have worked very hard to accomplish this. How did you do this? What program/ routine did you use to accomplish this? This can probably be done by iteratively changing some 4 bytes continuosly while checking CRC. This can not be done manually. I would like to have your program that did it. Kindly mail (email address sent to you) the program.
Last edited by Sunny Saini (2010-02-09 22:59:03)
Offline
Hello, Sunny Saini and persicum.
As Sunny Saini wrote here, there are some problems in MultiPar. I found some bugs and fixed, but still others left.
I came up with a new idea to improve PAR file format, and found par3j.exe has some problems also. Then sample PAR3 can not be usable by GUI temporary.
Offline
Sunny Saini
Thank you for downloading my prog.
I hope you find some courage to understand some switches and to perform some tests.
My prog is not a competitor to PAR3 rather than a friendship software, and I wish Yutaka to unsteer his bugs with multipar as soon as possible, but currently my prog is the single who overcomes 32768 SBC limit due to 32-bit Reed-Solomon codes.
Year, brute force attack to CRC is possible, but it could take you some hours or days to calculate it 4000000000 times for relatively long files. Contrary to this, instant hash adjusters also exist, but I couldn’t send you those of mine right now because malefactors could use it for their maleficent affairs.
Offline
Dear Readers,
Yutaka Sawada has removed both of the bugs mentioned above. He has also removed some other bugs and minus points.
Offline
Dear Persicum,
Your program has failed to support Unicode File names. Note that my first requisites for this subject-post was:
a) Unicode File Name support
b) Folder Recursion.
Apart from this a GUI front end would be appreciated. Full testing of your program can be done only after you add above features.
Once, on microsoft website, I had seen some file named "unicows". Probably the file is for unicode support. Please browse to microsoft website, it may help you.
Offline
Dear Yutaka Sawada,
One serious bug has been introduced in your new version 1.1.3.6b and 1.1.3.8. I am sending you the details by email about the same. Remove it as soon as possible.
Offline
Dear Peter B Clements,
IT'S TIME TO WAKE UP NOW.
Your program is still very stable and fastest when compared to the competetor PAR2 programs. The only shortcoming is no unicode file name support and no directory recursion.
So, kindly take leave for 2 to 3 days from your busy work. Go thoroughly through your source code and add unicode file name support and folder recursion.
Offline
Sunny Saini wrote:
Dear Peter B Clements,
IT'S TIME TO WAKE UP NOW.
Your program is still very stable and fastest when compared to the competetor PAR2 programs. The only shortcoming is no unicode file name support and no directory recursion.
So, kindly take leave for 2 to 3 days from your busy work. Go thoroughly through your source code and add unicode file name support and folder recursion.
No, QuickPAR is far apart to be fastest a PAR2 prog.
Yutakas par2j is MMX and multithread and 2-3 times faster then QuickPAR.
I don't know why your tests do not indicate this.
Besides, QP becomes very glitchful when blocksize becomes large enough and noone knows why...
If one want quickly treat a big set with small SBC it to be definitely failure.
Offline
Dear Sunny Saini!
In my very first post I said that my prog is ANSI only ‘cause I never use Unicode since I am not fran+ias neither espa+ol race. I like FAR 1.70 which has no Unicode…
But recursive search for subdirs is present.
So I am waiting for your reports about severe bugs excepting Unicode file failure.
Offline
Dear Sunny Saini!
I have benchmarked QuickPAR once again.
100M 3000-33%
Multipar2j took 44s
QuickPAR took 2m 20s
persicum’s crc32 took 09s (default algo)
So Multipar2j is 3 times faster than QuickPAR, please be more careful with your statements
Offline
Hello, Sunny Saini and persicum.
As speed comparison of QuickPar and MultiPar, which is faster, both you two are true on their PC. When there is enough memory for files, MultiPar runs faster than QuickPar. But, as MultiPar consumes much more memory than QuickPar, it may cause swap (serious problem which Saini might find), then it will become slower for large files.
I think that persicum has a PC with many memory, and Saini has a PC with less memory. To do benchmark and comparison, writing PC environment is important. Total speed depends on CPU, memory size, HDD speed etc.
Offline
Dear Yutaka Sawada!
In general you are right, PC specs should be specified.
But if they obsolete it means a modern PC which is most common, 2G 2core and so on.
If Shaini has PIII with 512M it is his own problem but not problem of Multipar
Offline
Dear Persicum,
I could not write in the month of february as my internet usage reached maximum limit very soon.
The problem of slow speed of Multipar, that I found on my PC, was reported with all details to Yutaka Sawada long ago via email. Thus, my statement was not false.
Secondly, Short RAM is not the end user's problem. It is of course, THE PROGRAMMER'S PROBLEM. This is due to the fact that, most of the organizations, Government and non-Government and other individuals are still using Windows XP and 512MB RAM with PIII Processor, despite the availability of Windows 7 OS, higher RAM and Multicore Processor. Use of Old PCs by them does not bar their fundamental rights of making backups and creating recovery files.
My PC
-----
Motherboard: Intel D945GCcr with 800 MHz FSB
CPU: Intel Dual Core 1.6GHz
RAM: Kingston 1GB DDR2 533MHz
OS: Windows Vista Ultimate OS (32bit) DEL OEM Release 1
HDD1: Seagate Baracuda 160GB SATA with OS
HDD2: Seagate Baracuda 160GB SATA.
I had dropped the idea of using ICE-ECC long ago despite the fact that it was faster than Quickpar. The reason for this was ICE-ECC uses "not known standards" and is very much RAM Hungry. Windows OS starts swapping data to hard disk causing too much disk thrashing for total larger input file sizes. That time I knew only these two PAR2 programs.
Most People, as stated at various nodes in this forum, generally makes iso images of their DVD's and then create PAR2 recovery files of those DVD images. When the prices of Blue Ray Drives and Disks shall drop, most people would start using Blue Ray Disks that has many a times capacity than a DVD. A RAM hungry recovery file making program then shall fail to do any good job for Blue Ray disk images. Thus, THE PROGRAMMER of such programs (like PAR2 recovery, etc) should take care of the following objectives:
1. The program should first detect available RAM, CPU and other available resources.
2. The program should send data to the CPU and RAM for processing only in parts and not fully, else, the disk swaping may occur.
3. If someone has lesser RAM on their system, the program may take longer to execute, but no frequent disk swapping should occur.
Talking about Unicode, Unicode is not meant just for various languages worldwide, but also for various symbols that are often used in file names, for example registered and trademark symbols. When you save a website as an html or mht file, sometimes unicode character appears in the filenames. One can not dispense with important web site files in one's backups. Moreover, Unicode has become a worlwide standard. Thus, it would be better to change your CRC32's code to support Unicode filenames, even if you have to restart all over again.
Also, a GUI front end is appreciated by users, especially beginners. Beginners/Novice users are often already afraid of making decision of Redundancy, block size etc. Non availability of GUI adds up their fright. In addition, a general user also likes a GUI program.
Testing by making Recovery files for just 100MB size on a system with 2GB RAM is too much discriminated. Thus, do the benchmarking again with higher sized Data files, e.g. a DVD sized total file size on your system and post the report.
Offline
Dear Readers,
Yutaka Sawada have launched his new version 1.1.4.0 of Multipar. He had removed the slow speed recovery problem and other bugs. Shell extension has also been added.
Now recovery for same set of files could be done in just 21 seconds.
Also Multipar is compatible to Quickpar if directory recursion is not used. That is, Recovery can be done by Multipar with files created by Quickpar and vice versa.
Offline