QuickPar / Problem verifying a large set

Calico · 2004-10-11 03:57:37

I'm attempting to create a par2 set for 64.5GB of files with about 20% redundancy in order to be able to recover from the complete loss of a few DVDs. I've found that a block size of approximately 180MB enables greater than 94% efficiency - which is good enough for this.

However, when I attempt to verify the set, quickpar crashes straight away.

Is there any well known flakyness around large block sizes or large sets?

I'm trying to keep it in one large set given that the data is logically a set anyway, and I'd feel like a sucker having split it only to have 1GB of extra pars in one set, and 20MB short in another.

PeterBClements · 2004-10-11 10:53:43

Yes, there is a known problem when trying to verify when using very large block sizes.

The problem is that when I wrote the verification code I made the "mistaken" assumption that no one would use such large block sizes. I therefore wrote the code to allocate a number of memory buffers large enough to hold two or three whole blocks. Obviously on a computer with relatively limited memory, the allocation will fail and QuickPar will crash (because that is also one of the very few cases where I don't actually check for a failed memory allocation).

Until I redo that bit of code, the only workarounds are to use a smaller block size, or more RAM. Obviously for such large data sets as you are using, the time taken must be very long (even though you have the source block count set down to about 367).

I'm currently working on a new algorithm that would allow you to create recovery files for a similar sized data set in the same amount of time, but using a block size of only about 16MB. Recovery files created by the new algorithm will not however be compatible with PAR2 files.

Calico · 2004-10-11 16:11:05

It's actually surprising to me that the allocations would have failed. The machine I was doing this on is reasonably well equipped.

It's an Athlon 64 3400+ (the old 2.2GHz 1MB L2 variety), with 1GB of RAM, Windows managing the swap, and a good 80GB of free space on that drive. I didn't notice any attempts to grab swap space either, it was a pretty fast crash - although I guess the allocation would be fast.

Obviously, I had the block size so huge so that it would complete in a sane amount of time (i.e. overnight rather than next week).

Thanks for the quick reply.

yesterdaysheadlines · 2010-02-20 06:24:50

Is there still no solution for using par2 on a large file? Quickpar appears to have successfully created recovery files for a 44GB source file but when I go to verify them I get the immediate crash as well. Are there any alternative programs that can handle files this large? Is Quickpar still being developed?

persicum · 2010-02-20 07:15:08

You can create 100000 blocks within one hour of time by prog persicum's CRC32.
The only problem is you need about 40G or 80G of free space for your 40G data because this prog likes to do a lot of swapping = ]]]

Last edited by persicum (2010-02-20 11:37:30)

#1 2004-10-11 03:57:37

Problem verifying a large set

#2 2004-10-11 10:53:43

Re: Problem verifying a large set

#3 2004-10-11 16:11:05

Re: Problem verifying a large set

#4 2010-02-20 06:24:50

Re: Problem verifying a large set

#5 2010-02-20 07:15:08

Re: Problem verifying a large set

Board footer