Official K-Probe Discussion (Tool for Scanning C1C2/PIPO)

[KCK]

Here is yet another possible interpretation of Karr's explanation from http://www.cdrlabs.com/phpBB/viewtopic. ... 2100#62100 .

Let's assume momentarily that no reading/slipping errors occur.

Suppose K's Probe issues a command to start reading a disc from LBA=100 onwards, and another command that triggers the chipset to start accumulating C1/C2 errors at LBA=100. Knowing the reading speed, K's Probe computes Delta_t as a (hopefully slight) overestimate of the time needed to read LBA's 100 through 174. Thus, after waiting for Delta_t seconds, K's Probe issues a command to determine the current LBA, say LBA_cur, and another command to read the accumulated C1/C2 errors.

First, suppose LBA_cur=190 (or in general LBA_cur exceeds the starting LBA + 73). If the C1/C2 errors are indeed accumulated for LBA's 100 through 174, the pictures and the csv file should refer to LBA=174 (i.e., the starting value + 74) instead of 190. Then K's Probe triggers the chipset to start accumulating C1/C2 errors at (the current) LBA=190, computes the next Delta_t, and so on. Second, for LBA_cur<174 (or the starting value + 74 in the general case), K's Probe estimates how long to wait until LBA=174 is reached, then reads LBA_cur, and so on.

If the preceding interpretation were correct, it would have several implications.

First, we should recognize that K's Probe can only probe (i.e., sample) some (hopefully most, but not necessarily all) sectors. The number of missed sectors could be lowered by employing a less conservative strategy for estimating delays (i.e., Delta_t). Of course, only sample averages and maxima should be reported (unless other estimates for the whole disc are developed).

Second, to cover all sectors, the one-pass strategy (i.e., reading the whole disc in one pass) would have to be replaced by restarts after processing every 75 sectors, i.e., demanding "fresh" reading and accumulation of C1/C2 errors from a suitable LBA (increased by 75 at each restart) after each chunk of 75 sectors is digested.

I can't claim that my interpretation is correct, but it does seem to be competitive with those given by cfitz and MediumRare, and it agrees in spirit with spath's remark on the MediaTek chipsets. Of course, it would be best for Karr to explain what "raw" numbers K's Probe gets from a drive. It's another story how such "raw" numbers are used for computing the averages, max values and pictures.

As for the latest exchanges between cfitz and spath on "raw commands", I couldn't locate any specific command for C1/C2 errors. Concerning MMC-3 and MMC-4, the classical Read CD command for reporting C2 errors (also mentioned by Eric Deppe for CD Speed) seems to be a different animal (working for arbitrarily many, and not just 75, sectors). Well, maybe Read CD (possibly in conjunction with bustrace or bushound) could be used for detecting what K's Probe does for C2 errors. If we don't get any feedback from Karr, then perhaps spath and other testers could tell us more?

[dolphinius_rex]

There's some interesting information regarding K-Probe's reactions to a CD-R burned using Yamaha's AAMQR that I think people might like to read. Here's the link: http://www.cdrinfo.com/forum/topic.asp? ... hichpage=2 look for posts made by sambs0.

[CDHero]

Here is yet another possible interpretation of Karr's explanation from http://www.cdrlabs.com/phpBB/viewtopic. ... 2100#62100 .

Let's assume momentarily that no reading/slipping errors occur.

Suppose K's Probe issues a command to start reading a disc from LBA=100 onwards, and another command that triggers the chipset to start accumulating C1/C2 errors at LBA=100. Knowing the reading speed, K's Probe computes Delta_t as a (hopefully slight) overestimate of the time needed to read LBA's 100 through 174. Thus, after waiting for Delta_t seconds, K's Probe issues a command to determine the current LBA, say LBA_cur, and another command to read the accumulated C1/C2 errors.

KProbe doesn't calculate the read speed and won't wait anyway.
The algorithm is :
For High Performance System=>
[CMD] Read Current Position (Assume LBA 100)
[CMD] Start accumulating
[CMD] Read Current Position (Assume LBA 120)
[CMD] Read Current Position (Assume LBA 138)
[CMD] Read Current Position (Assume LBA 156)
[CMD] Read Current Position (Assume LBA 174)
[CMD] Read C1C2
[CMD] Start accumulating
[CMD] Read Current Position (Assume LBA 190)
[CMD] Read Current Position (Assume LBA 207)
[CMD] Read Current Position (Assume LBA 228)
[CMD] Read Current Position (Assume LBA 248)
[CMD] Read C1C2
.......

For Poor System=>
[CMD] Read Current Position (Assume LBA 100)
[CMD] Start accumulating
[CMD] Read Current Position (Assume LBA 190)
[CMD] Read C1C2
[CMD] Start accumulating
[CMD] Read Current Position (Assume LBA 280)
[CMD] Read C1C2
.......


So , if your system performance is high , KProbe won't loss any data
but if you are using a poor system , KProbe will loss some data

[cfitz]

Thanks for the more detailed explanation Karr. I think that helps and makes it much more clear what is going on. I still have a couple of questions if you will allow me.

1. Once you have issued the 'Start Accumulating' command, at what point does the drive stop accumulating C1C2?

a. 75 blocks after the LBA at which the 'Start Accumulating' command was issued.
b. When the 'Read C1C2' command is issued
c. When the next 'Start Accumulating' command is issued

2. If the answer to question 1 is 'c', how large a count can the drives accumulate before they overflow? In other words, what is the capacity of the C1C2 accumulators? Are they 16 bits, 32 bits, or some other size?

cfitz

[KCK]

Karr Wang:

Thanks a lot for correcting my interpretation!

If I understood your examples correctly, KProbe works in two nested loops.

Initially [CMD] "Read Current Position" determines the starting LBA, say LBA_start (100 in your examples). Then an outer loop is started by [CMD] "Start accumulating". It contains an inner loop in which [CMD] "Read Current Position" is issued repeatedly until the current LBA, call it LBA_cur, is at least LBA_start+74. Then, after issueing [CMD] "Read C1C2" and resetting LBA_start to LBA_cur, KProbe starts the next outer loop from [CMD] "Start accumulating", and so on.

However, it seems that C1C2 errors are accumulated twice instead of once for the final sector of each outer loop (e.g., for LBA=174 in your "High Performance System" example), unless the current LBA is increased by 1 after [CMD] "Read C1C2".

Here I presume that if [CMD] "Start accumulating" is issued at LBA=LBA_start, then after LBA exceeds LBA_start+73, the chipset has accumulated C1C2 for sectors LBA_start through LBA_start+74.

It still seems to me that KProbe may loose data unless the system is so fast that LBA increases at most by 1 in each inner loop, i.e., after each [CMD] "Read Current Position". Just consider what would happen if "[CMD] Read Current Position (Assume LBA 174)" in your example were replaced by "[CMD] Read Current Position (Assume LBA 176)".

Again, my interpretations could be wrong, so your feedback would be highly appreciated.

[CDHero]

Thanks for the more detailed explanation Karr. I think that helps and makes it much more clear what is going on. I still have a couple of questions if you will allow me.

1. Once you have issued the 'Start Accumulating' command, at what point does the drive stop accumulating C1C2?

a. 75 blocks after the LBA at which the 'Start Accumulating' command was issued.
b. When the 'Read C1C2' command is issued
c. When the next 'Start Accumulating' command is issued

2. If the answer to question 1 is 'c', how large a count can the drives accumulate before they overflow? In other words, what is the capacity of the C1C2 accumulators? Are they 16 bits, 32 bits, or some other size?

cfitz

Dear cfitz :
The answer is "a" ,

[CDHero]

However, it seems that C1C2 errors are accumulated twice instead of once for the final sector of each outer loop (e.g., for LBA=174 in your "High Performance System" example), unless the current LBA is increased by 1 after [CMD] "Read C1C2".

It is impossible to accumulate the final sector twice , because the PUH
slips out continuously.

Here I presume that if [CMD] "Start accumulating" is issued at LBA=LBA_start, then after LBA exceeds LBA_start+73, the chipset has accumulated C1C2 for sectors LBA_start through LBA_start+74.

It still seems to me that KProbe may loose data unless the system is so fast that LBA increases at most by 1 in each inner loop, i.e., after each [CMD] "Read Current Position". Just consider what would happen if "[CMD] Read Current Position (Assume LBA 174)" in your example were replaced by "[CMD] Read Current Position (Assume LBA 176)".

KProbe will not loss data , because the data was accumulated in chip
register , program does not need to send command every sector.
When chip receives "Start Accumulating" Command , register will be
reset to 0 , and start accumulating.

[cfitz]

The answer is "a" ,

Ah, excellent! Thanks Karr. I believe I fully understand now.

It is clear that my earlier statement about the individual error counts and the averages potentially being wrong and needing to be scaled by the LBA delta was incorrect. My mistake. I misinterpreted the discrepancies between averages and totals that Medium Rare and I each noticed.

Based on your additional explanations, I now understand that each individual sample is correctly scaled to report the number of errors in the one-second interval (or 75 blocks, if you prefer) immediately following the position on the disc where accumulation of that sample was initiated. The overall averages are also correct, or at least accurately representative.

The only things that might be incorrect are the total counts of errors. If a user's system is too slow it might not be able to sample every block on the disc when testing at higher read speeds. In this situation the error totals will be reduced because only portions of the disc will be sampled. But those portions that are sampled will be accurate and, unless huge numbers of samples are missed, the averages will be representative of the true averages over the entire disc.

Thanks for clearing this up, Karr. And I apologize for creating confusion.

cfitz

[CDHero]

Dear cfitz:
It's right !!!

[cfitz]

Dear cfitz:
It's right !!!

With enough patience even slow-witted old coots like myself will understand.

cfitz

[KCK]

Karr Wang:

Going back to your "Poor System" example:

Step 1: [CMD] Read Current Position (Assume LBA 100)
Step 2: [CMD] Start accumulating
Step 3: [CMD] Read Current Position (Assume LBA 190)
Step 4: [CMD] Read C1C2
Step 5: [CMD] Start accumulating
Step 6: [CMD] Read Current Position (Assume LBA 280)
Step 7: [CMD] Read C1C2

could you tell us which of the following are true?

1. Step 4 reads C1C2 for sectors:
a. 100 through 174
b. 101 through 175
c. possibly some later sectors, depending on system delays

2. Step 7 reads C1C2 for sectors:
a. 190 through 264
b. 191 through 265
c. possibly some later sectors, depending on system delays

Here is a more precise version of my former querry:

It still seems to me that KProbe may not provide C1C2 for some sectors unless the system is so fast that LBA increases at most by 1 in each inner loop, i.e., after each [CMD] "Read Current Position".

For instance, suppose "[CMD] Read Current Position (Assume LBA 174)" in your "High Performance" example is replaced by "[CMD] Read Current Position (Assume LBA 177)". Then how will KProbe get C1C2 for LBA=176?

I may be missing something obvious, in which case I apologize in advance for bothering you with these querries.

cfitz:

I saw your latest post after finishing the above querries.

Now I'd like to take you up on your "I believe I fully understand now"! I may be dumb (and still recovering from holiday celebrations), so I would appreciate your comments on my intepretations.

[CDHero]

Karr Wang:

Going back to your "Poor System" example:

Step 1: [CMD] Read Current Position (Assume LBA 100)
Step 2: [CMD] Start accumulating
Step 3: [CMD] Read Current Position (Assume LBA 190)
Step 4: [CMD] Read C1C2
Step 5: [CMD] Start accumulating
Step 6: [CMD] Read Current Position (Assume LBA 280)
Step 7: [CMD] Read C1C2

could you tell us which of the following are true?

1. Step 4 reads C1C2 for sectors:
a. 100 through 174
b. 101 through 175
c. possibly some later sectors, depending on system delays

c. possibly some later sectors, depending on when the drive receives
STEP2 + 75 sectors.

2. Step 7 reads C1C2 for sectors:
a. 190 through 264
b. 191 through 265
c. possibly some later sectors, depending on system delays

c. possibly some later sectors, depending on when the drive receives
STEP5 + 75 sectors.

Here is a more precise version of my former querry:

It still seems to me that KProbe may not provide C1C2 for some sectors unless the system is so fast that LBA increases at most by 1 in each inner loop, i.e., after each [CMD] "Read Current Position".

For instance, suppose "[CMD] Read Current Position (Assume LBA 174)" in your "High Performance" example is replaced by "[CMD] Read Current Position (Assume LBA 177)". Then how will KProbe get C1C2 for LBA=176?

I think cfitz can answer this question correctly!!

[rdgrimes]

Hi Karr:
Have you had a chance to look at the errors I posted at CDFreaks, where Kprobe is not reporting unreadable sectors but CDSpeed does? I can reproduce this reliably.
Also, do you have any input about the cause of these errors? It occurs when SmartBurn intervenes and lowers the burn speed very late in the burn, at speeds of 45x and above, but not at lower speeds.

[cfitz]

Now I'd like to take you up on your "I believe I fully understand now"!

I think cfitz can answer this question correctly!!

Oh man! I feel like I'm back in school and have been called to blackboard to do a problem in front of the whole class. And instead of studying the night before, I was watching television.

I am hesitant to post some examples such as Karr did earlier, because to do so with sufficient detail I would have to make some assumptions about the underlying hardware. Although I don't think these assumptions would have a material effect on the final conclusion (if wrong they would likely just shift the absolute LBA's by 1 unit), I don't want to muddy the water any more than I have already done.

I think the main point to be gleaned from Karr's responses is that samples are always scaled properly to include counts of errors for a full second (75 blocks). Thus, my earlier concerns about inflated error counts and incorrect averages were unfounded. The only measurement data that might have some significant errors, depending on how fast your computer can execute KProbe, are the error totals.

KCK, as for your question about missing LBA 174 while polling via the Read Current Position commands and landing on 176 instead, I believe the answer has two parts. First, anytime you are late in polling you will end up not sampling one or more blocks. There is a continuum from Karr's bad example to his good example, with fuller sampling coverage (number of blocks in included in some sample) obtained the closer you get to perfect polling. Second, even if what you describe happens KProbe won't lose any data within any individual samples because the hardware accumulator handles collection of the 75 blocks worth of error counts per sample for the program.

Thus, KProbe might not sample block 175, but the surrounding samples will each contain the error count for a full chunk of 75 blocks. For example, KProbe might record a sample that spans blocks 100 through 174 and another sample that spans blocks 176 through 250, both of which are full 75-block samples, but no sample that contains block 175.

In practice, I don't think we will ever get full coverage. Even at 4x testing (which is the slowest I can test at, even when I specify 1x reading - I suspect this is a hardware limitation in my drive) on my 1 GHz PIII, I find an average LBA delta of 76.5, so that 2% of the sectors on my disc are not being sampled. At 48x testing while reading the end of the disc (where the actual linear speed is 48x) my LBA deltas average about 92, so that 23% of the sectors are not being sampled (the overall average at 48x CAV testing is about 16% un-sampled sectors).

This may seem like a large number, but apparently it is still not so large as to prevent KProbe from measuring a representative sample of the disc's quality and doesn't seem to affect the measurement accuracy in any significant amount. Why do I say this? Because running the same test multiple times yields virtually the same results, even though the different test runs, due to the somewhat random nature of the sample timing, omit and include different blocks from the samples.

My one concern with the un-sampled sectors issue would be in measurements of parameters that are very infrequent but important when they do occur. I am thinking particularly of C2 errors. In practice though, I haven't seen this concern borne out in real world tests, either in my KProbe testing or my CD Doctor testing (which use the same underlying mechanisms and behave in the same manner in this regard - see MediumRare's comments in an earlier post). I've not run into a disc that shows a C2 error in 75-80% of the test runs but shows up clean in the other 20-25%. Any C2 errors I find show up in every run.

I suppose it is theoretically possible I have a disc that has one sector with C2 errors and I just never happened to hit it while testing, but it doesn't seem likely. To be safe, though, in addition to KProbe we should probably also run our discs through CD Speed's Scan Disc or CD Quality tests to give a second, semi-independent check for uncorrected C2 errors.

Whew! This is a long post. I am going to end it now, and let my thoughts simmer for a while.

cfitz

[KCK]

cfitz:

First, let me mention that my initial post on Tuesday simply intended to revive the discussion of what KProbe measures. Making this post wasn't easy, but both Karr and you rose to the bait, and now we know a lot more!

Your latest post confirms my general ideas about KProbe's operation, and I agree with your statistical intepretations and conclusions.

However, one aspect is still unclear: you (and possibly Karr) assume that each sample gives C1C2 counts for 75 blocks. This, of course, is not true in general for the final sample. Unfortunately, to exhibit "too small" samples, one has to make some assumptions about the underlying hardware.

Let's go back to Karr's "Poor System" example:

Step 1: [CMD] Read Current Position (Assume LBA 100)
Step 2: [CMD] Start accumulating
Step 3: [CMD] Read Current Position (Assume LBA 190)
Step 4: [CMD] Read C1C2
Step 5: [CMD] Start accumulating
Step 6: [CMD] Read Current Position (Assume LBA 280)
Step 7: [CMD] Read C1C2

Suppose Step 2 starts C1C2 accumulation for sectors LBA_acc, LBA_acc+1, ... . According to Karr, LBA_acc can't be guaranteed to have any specific value (such as 101 or 102). Suppose LBA_acc=120 due to delays (if you deem such delays excessive, let LBA_acc=101 and assume LBA=174 at Step 3). Now, if there is no delay between Steps 3 and 4, Step 4 reads C1C2 accumulated for sectors LBA_acc=120 through LBA=190, i.e., 71 sectors in total (or 72 if there is 1 block delay, etc.).

Here my point is that apparently samples with C1C2 counts for fewer than 75 sectors can occur due to delays. At the same time, in any practical run, I would expect the number of "too small" samples to be relatively small, and the average size of "too small" samples to be close to 74. Hence "too small" samples may have negligible (if any) influence on the averages produced by KProbe.

BTW, for this example, I wonder whether KProbe outputs LBA=190 into the csv file. Or maybe Step 4 can somehow determine the final sector number for which C1C2 has been accumulated, and this number is output?

[cfitz]

Hi KCK. I was wondering if you would respond the way you did. I anticipated this line of questioning regarding under-counting, but didn't want to argue it ahead of time because, as I said, too many assumptions are involved and because, frankly, I was tired.

However, one aspect is still unclear: you (and possibly Karr) assume that each sample gives C1C2 counts for 75 blocks.

Not true. You are making assumptions about my assumptions. Since turn-about is fair play, I will now make some assumptions about your assumptions.

First, let me neglect the final sample of the disc. Depending on how it is handled (again, assumptions enter the picture) it may or may not be a problem. I certainly understand the argument that if KProbe starts accumulating errors with less than 75 blocks remaining in the disc, then it won’t be able to complete a full chunk of 75 blocks. But to say that the sample will then be reported with too few blocks assumes that it will be reported at all. Perhaps either the drive or KProbe throws out that sample since it is incomplete, and those last few blocks just end up just being a few more un-sampled blocks rather than incorrectly reported blocks.

Even if the last sample is reported with too few blocks, I wouldn't worry about it since it is a special case and would not represent a systematic problem. Personally, I am not concerned about any possibility that the very last sample may or may not under-report the error rate.

Okay, now onto your broader point. As I see it, the concern you describe rests upon some assumptions that you have not stated and may not be aware of.

1. You appear to be assuming that KProbe uses the LBA from step 1 to decide how many blocks it must wait before reading the completed 75-block C1C2 counts. I can understand why you might assume this based on the examples Karr listed, but it is possible that you are reading too much into a couple of artificial examples Karr created to illustrate a different point.

Rather than waiting 75 blocks after the LBA from step 1, KProbe may wait 75 blocks after the LBA from step 3. That would guarantee that a full 75 blocks have been accumulated before any attempt to read the C1C2 counts is made. Anyway, that is what I would do if I wanted to guarantee a full 75-block sample given the limitations of the polling as currently described. (Of course, I am making an assumption here as well - the assumption that once the Start Accumulating command returns, accumulation has actually started. As you can see, there are many potential pitfalls in the assumptions.)

2. You also appear to be assuming that the Read C1C2 command will return successfully if less than 75 blocks have been accumulated. Maybe it does, maybe it doesn't. I don't know. It might return a status value indicating that the counts aren't ready yet if one attempts to read the counts too early. Or maybe it won't return at all (within a given timeout period) until all 75 blocks are accumulated. Or maybe the Read Current Position command returns an accumulation status that KProbe uses to determine when to read the C1C2 counts, and it isn’t based on a pre-computed offset from step 1. Or maybe the Read Current Position and Read C1C2 commands are actually one and the same command that returns LBA, sample ready status and C1C2 counts simultaneously, but Karr gave them different names in his example as an expository aid designed to emphasize their main functions at each step (I doubt this, but throw it out as another example of the softness of the ground upon which we are treading).

Of course, your proposition may also be correct. I've seen some samples in my csv files that are separated by less than 75 sectors, so there may be some underreporting going on. Then again, as alluded in your last question, there is some uncertainty about the source of the reported LBA values, so the odd LBA delta that is less than 75 may have no real meaning.

In the end, I feel that there are too many unknowns and assumptions regarding the nitty details to make meaningful conclusions about those details.

The epiphany of which I spoke in an earlier post was that the drive automatically stops accumulating error counts after 75 blocks, without any external intervention by KProbe. This means that error rates are not being over-reported, which was my main concern based on the observed data MediumRare and I reported. If you look within some saved csv files, I think you will find that the vast majority of LBA deltas are greater than 75 blocks, making it unlikely that under-reporting is the problem.

I am satisfied with my understanding at the higher levels of abstraction, but don’t and won't claim to know all the fine details. I could never do so without having the actual code and chipset specifications in hand.

So with that maybe I really will let this rest for a while. I don’t think I have anything more to add unless Karr reveals more details.

cfitz

[KCK]

cfitz:

Sorry, instead of writing "you (and possibly Karr) assume that each sample gives C1C2 counts for 75 blocks", I should have quoted "samples are always scaled properly to include counts of errors for a full second (75 blocks)". What I really wanted to convey was that it seemed to me that nobody had mentioned the possibility of getting "too small" samples. Anyway, I apologize for improper wording.

Also please note that additional confusion arose because "sample" seems to be used to denote two different (albeit related) objects: what KProbe reads from the drive, and what is used for producing pictures and the csv file. Only Karr could tell us whether "too small" samples are detected and discarded from the output.

Now, concerning your guesses on my assumptions:

1. I was not "assuming that KProbe uses the LBA from step 1 to decide how many blocks it must wait before reading the completed 75-block C1C2 counts". A related assumption on waiting was explicit in my initial post of yesterday, but then Karr wrote "KProbe ... won't wait anyway", so I stopped thinking in such terms.

2. Yes, I was assuming "that the Read C1C2 command will return successfully if less than 75 blocks have been accumulated". This looked to me as the simplest assumption consistent with Karr's examples. I somehow doubt whether KProbe employs any of the more sophisticated variants which you mentioned, but again only Karr could tell us.

I agree that without access to command specifications, etc., we can only speculate, unless Karr gives us more feedback. Yet your latest post was not limited to speculations alone, since it also presented an empirical observation: "I've seen some samples in my csv files that are separated by less than 75 sectors". I hope Karr will comment on this fact.

[cfitz]

What I really wanted to convey was that it seemed to me that nobody had mentioned the possibility of getting "too small" samples.

I understood what you were saying, but didn't address it because:

1. Based on observed behavior, it is not the likely possibility. The more likely concern was including too many blocks in a sample (although that concern has since been alleviated).
2. I haven't written the software and don't have access to the code or chipset specifications that would be required to address it with any authority.

Anyway, I apologize for improper wording.

No need for any apologies. If anyone should apologize, it probably ought to be me. Re-reading my post now I could see it appearing to be curt. It wasn't intended to be, but I was up late and was quite tired. I really wasn't attacking you, if that is what you are feeling.

1. I was not "assuming that KProbe uses the LBA from step 1 to decide how many blocks it must wait before reading the completed 75-block C1C2 counts". A related assumption on waiting was explicit in my initial post of yesterday, but then Karr wrote "KProbe ... won't wait anyway", so I stopped thinking in such terms.

There might be some confusion regarding what is meant by "wait". Your earlier post and Karr's response addressed the issue of KProbe calculating a fixed time interval and "waiting" in the sense of sleeping for the duration of that interval, doing nothing until it is time to wake up and read the sample counts. "Waiting" in this sense is based solely on the passage of physical time measured in milliseconds. I believe that when Karr said KProbe does not wait, he meant that KProbe does not make an estimate of how many milliseconds it must sleep before reading the counts, but rather continuously polls the drive to determine when 75 sectors worth of error counts have been accumulated.

When I said "wait" I meant wait in the sense of KProbe polling the LBA position until it determines that 75 blocks have been read since it started accumulating error counts. Perhaps I should have said something like "You appear to be assuming that KProbe uses the LBA from step 1 to calculate the LBA beyond which KProbe can safely read the errors for a full 75-block chunk." I'm sorry I can't say this very clearly, but I'm having trouble phrasing it without the use of the word "wait" or some similar time-ordering word or phrase, because it is a fact that time does pass between starting the error count accumulation and reading the totals. I think the difference between how you were using the word "wait" and how I was using it is that you were thinking of waiting some fixed time interval measured in milliseconds while I was thinking of waiting for an LBA beyond a 75-block delta threshold to be read.

In any case, KProbe must "wait" in some form to read the C1C2 counts, because they aren't immediately available after issuing the Start Accumulating command. If you don't believe KProbe waits a fixed time interval (which, it has been explained by Karr, it does not), and you don't believe that it calculates some offset from the LBA at which started accumulating to determine the LBA beyond which accumulation is finished, and you don't believe that the Read C1C2 command returns any sort of status indicating whether or not the accumulation is finished, then how are you proposing that KProbe knows when to read the C1C2 errors? I think (but may be mistaken) that you really do have the LBA offset calculation method in mind and did make assumption 1 as I posited.

Yet your latest post was not limited to speculations alone, since it also presented an empirical observation: "I've seen some samples in my csv files that are separated by less than 75 sectors". I hope Karr will comment on this fact.

For what it is worth, only a very small percentage (e.g. 0.3%) are less than 75, and most of the examples I see are for unreadable sectors for which no error counts are reported at all. Furthermore, the samples that are recorded with deltas less than 75 almost always have deltas of 70 or greater. Thus, even if under-reporting were occurring in these limited samples, the under-reporting would likely be only on the order of 5%.

Based on all of this, I really don't think under-reporting is an issue. I haven't seen any indisputable evidence to prove that it is, what evidence I have seen to even suggest it as a possibility only supports the possibility of it occurring very infrequently, and even if it were occurring, the evidence suggests that its magnitude would be small. Combine all these, and it seems to me that the overall effect would be minimal in even the worst case.

I am willing to concede that I may be wrong. However, since I don't have the information needed to authoritatively respond, I will leave this discussion to you and Karr.

cfitz

[MediumRare]

I don't want to get involved in the sampling discussion (too much heavy thinking to do at work right now).
But here's some more info on sample length: I augmented the table for mean sample length for a good CD at various read speeds that I posted earlier with the most common sample length. Values in () denote the range where No. of samples were more that half the peak count. I added the values for the RW I've been using in the packet writing test (MRW): Verbatim Ultraspeed at max scan rate.

Code: Select all

  Rate    mean, peak sample   #<75  #>100
   8x     76.4   76 (77)        0       1
  16x     78.4   77 (77-79)     0       1
  24x     79.6   79 (78-81)     1       6
  32x     81.1   80 (79-83)     1       4
  max     84.3   84 (81-86)     0       1

  RW Max  82.8   82 (80-85)     0      19

The sample size (in sectors) is very sharply peaked around the mean value. The single value < 75 was 70. This is the most common value < 75 (none smaller!!) and showed up a great deal more in earlier versions before Karr fixed the slip-test- not significantly since then.

This agrees pretty well with your comments, cfitz. The Mexico DataLife disk I burned @48x was pretty ugly in all respects and also had the most undersize samples (not in the table).

G

[spath]

According to these figures it seems that KProbe is roughly missing from
2 to 12% of all the sectors on a disc for its C1/C2 calculations

[dolphinius_rex]

that's why proffesional testing is done on hardware made especially for the job.

[MikeTR]

According to these figures it seems that KProbe is roughly missing from
2 to 12% of all the sectors on a disc for its C1/C2 calculations

that's why proffesional testing is done on hardware made especially for the job.

We can't expect performance on par with a professional dedicated machine from a (free) tool like this. I think mr. Wang has done a great job so far and I hope he keeps going on like this.
He might,....as long as cfitz, MediumRare and KCK don't drive him from this forum with those gruelling questions .
Just kiddin' guys. Thanks for your relentless testing and for trying to understand the ins and outs of this great piece of software .

[cfitz]

According to these figures it seems that KProbe is roughly missing from
2 to 12% of all the sectors on a disc for its C1/C2 calculations

True, but sampling 88 to 98% of the sectors seems to be sufficient to give a reasonably accurate representation of a disc's quality. I say this because running multiple tests, where a different 2 to 12% of the sectors are missed each time, doesn't materially change the measured values.

In my view the one possible cause for concern would be the measurement of C2 errors. Missing a particular C1 error is not a big deal, since there are generally plenty of others and enough samples are taken to ensure that the average is representative of the true rates. But missing a C2 error is a different story. We don’t want any C2 errors on our discs, and expect them to be rare so that when one does show up it is a noteworthy event. Thus, the cost associated with missing a C2 error is higher than the cost of missing a C1 error.

Fortunately, this doesn't seem to be a significant issue in actual practice. I haven't run across any disc that when repeatedly tested sometimes shows a C2 error and sometimes does not. I have one particular disc that shows just a few C2 errors, so it would be a candidate for exhibiting this type of problem, but it shows the C2 errors consistently. Granted, it is possible that a disc may have just one or two C2 errors and that they could be missed. Therefore it would be best if KProbe guaranteed to catch all C2 errors all of the time. Lacking that guarantee, it is probably best for people with particular concerns in this area to test each disc at least twice, and also to test with a second, independent program such as CD Speed’s Scan Disc or CD Quality test. Despite having said this, it still seems to me, based on my experiences, that actual instances of completely missing all C2 errors on a disc would be rare.

By the way, WSES, CD Doctor and KProbe all exhibit this same behavior. Here is a histogram that adds to the data MediumRare posted. In this case rather than showing distributions for different testing speeds I am showing distributions for different testing programs at the same (48x CAV) testing speed:

Code: Select all

         C1 Max   C1 Avg    C1 Total  Total Samples
----------------------------------------------------
WSES       8       0.232      958         4122
CD Doc     8       0.221      942         4255
KProbe     7       0.236      981         4163

CD Doctor collected the most samples overall. WSES would have done almost as well, but for some reason it skipped the first 100 seconds of the disc even though I had it set to start at 2 seconds. I've seen it do this type of thing in the past.

All three have pretty tightly grouped sample spacing, with WSES and CD Doctor being a little better than KProbe. However, the difference isn't enough to make a significant affect in the overall measurement outcome, as can be seen by comparing the maximum and average values. Keep in mind, while viewing the histogram chart, that I have zoomed in on the area around the 75-block sample spacing to show greater detail. Thus, differences are artificially magnified.

Note that none of sample spacings (with the exception of CD Doctor which I will explain in a moment) are less than 75 blocks. In fact, the smallest sample spacing is 78 blocks. This offers additional evidence to support the contention that KCK's concern about under-sampling, while potentially theoretically valid, is not warranted in practice.

The exception of which I spoke regarding the CD Doctor results is that 28 samples were reported with a spacing of 0 blocks, and 28 with an average spacing of 162 blocks. It was suspicious that the same numbers of samples were spaced by 0 as by 162 blocks, and that 162 blocks is half the average spacing. Further investigation revealed that these 56 incidents were the result of CD Doctor reporting two samples in a row with the same LBA 28 times. I think this is a bug or anomaly with how CD Doctor reports the LBA rather than with how it measures and reports the error count data, since the C1 counts varied even though the LBA remained the same. Here are a few examples:

Code: Select all

LBA      C1
------------
3185     4
3185     1

12281    0
12281    2

21990    2
21990    2

Based on this, I don't believe that the sample size was actually 0 blocks. I think there were two full size 75-block samples separated by an average of 81 blocks that were incorrectly reported as having the same LBA. It does make me wonder, though, if CD Doctor is using a slightly different polling technique. To me it smacks of polling on a Read C1C2 command that returns complete/incomplete status, and reading the LBA only after the Read C1C2 command returns with status "complete". This is, however, pure speculation.

In the end, all three programs show very similar behavior and results. I believe that the limiting factor is likely the characteristics of the underlying hardware in the drives. So, if they all show essentially the same behavior and results, which should be used? That obviously is a question that everyone has to answer for himself or herself, but I will state my personal views.

WSES was the only game in town when it first came out, and it was great to have such a tool, but it is very inconvenient to use. CD Doctor made a big advancement in convenience since it can run under Windows and allows results to be easily saved in annotated chart format for display or csv file for additional analysis. KProbe improves on CD Doctor by including more control over formatting and scaling and by providing additional tests and information. Also, KProbe is still under development and Karr has shown a very generous spirit in working with us and incorporating our suggestions. Based on all of these considerations, I personally intend to use KProbe for most of my future testing.

We can't expect performance on par with a professional dedicated machine from a (free) tool like this. I think mr. Wang has done a great job so far and I hope he keeps going on like this.

I agree 110%!

He might,....as long as cfitz and KCK don't drive him from this forum with those gruelling questions

Hopefully we aren't driving him away. I've tried to convey to him that I support his efforts and appreciate that he has given us such a nice test program. I wouldn't want inquisitiveness and constructive criticism to be confused with mean-spirited bashing. I'm definitely not against KProbe.

And by the way, why don't you include MediumRare in the list of grueling inquisitors? I think he deserves credit (or blame ) as well.

cfitz

[MikeTR]

[And by the way, why don't you include MediumRare in the list of grueling inquisitors? I think he deserves credit (or blame ) as well.

cfitz

Done!

Trying to spread the blame, aren't you? (make that credit)

All kidding aside, thanks for digging into Kprobe for all of us cfitz (as well as MediumRare and KCK of course).
Like I said before: this is the ultimate Beta-testing going on.

[spath]

> that's why proffesional testing is done on hardware made especially for the job.

Really ? Like what ?