Official K-Probe Discussion (Tool for Scanning C1C2/PIPO)

[MikeTR]

> that's why proffesional testing is done on hardware made especially for the job.

Really ? Like what ?

Take a look here: http://www.audiodev.com for a start. I remember reading an article about a specialized testing machine recently. If I come across the link, I'll be sure to post it for ya.

[dolphinius_rex]

I know of AudioDev and Quantized for makers of CD-R/W testing hardware. I don't know much about Quantized, but here's their website: http://www.quantized.com/

Generally the hardware is REALLY expensive though, yes even more expensive then Plextor drives!!

[MediumRare]

Done!

Trying to spread the blame, aren't you? (make that credit)

All kidding aside, thanks for digging into Kprobe for all of us cfitz (as well as MediumRare and KCK of course).
Like I said before: this is the ultimate Beta-testing going on.

I'll gladly shoulder part of the blame as long as Karr doesn't get fed up.

I haven't been part of Beta-testing before, but this is sure an interesting experience (takes up gobs of time though). I signed up to this board mainly because CD Doctor became available. It was too late to participate much in exploring its possibilities, but I use(d) it extensively and wanted to share my results. So when K's Probe showed up, I just had to be part of it.

Incidentally, Audiodev is the lab used by c't to test their media (there was a flap about this last week- I won't link to the thread, let it sleep).

G

Mike: I've meaning to throw this footer at you and Dodecahedron for about 2 months:

Ash nazg durbatulûk, ash nazg gimbatul, ash nazg thrakatulûk agh burzum-ishi krimpatul.

[dodecahedron]

A Elbereth Gilthoniel

[MikeTR]

Mike: I've meaning to throw this footer at you and Dodecahedron for about 2 months:

Ash nazg durbatulûk, ash nazg gimbatul, ash nazg thrakatulûk agh burzum-ishi krimpatul.

"Never before has any voice dared to utter the words of that tongue in Imladris"
"And let us hope that none will ever speak it here again"

All I can say is:

A Elbereth Gilthoniel o menel palan-diriel, le nallon si di'nguruthos! A tiro nin, Fanuilos!

[MediumRare]

A Elbereth Gilthoniel

"Never before has any voice dared to utter the words of that tongue in Imladris"
"And let us hope that none will ever speak it here again"

(I won't use that other language again.) )

G

[spath]

> I know of AudioDev and Quantized for makers of CD-R/W testing hardware.

I know them, I just wanted to be sure you were thinking of such tools.
These are a completely different thing from the error reporting feature
of a drive, they are just showing different informations so you should not
compare the two. On the other hand professionals who have designed their
chipsets with method A for error reporting will not miss any sector on a disc,
at any speed and without any special hardware.

[cfitz]

On the other hand professionals who have designed their chipsets with method A for error reporting will not miss any sector on a disc, at any speed and without any special hardware.

Do you mean method 1 that I postulated earlier and that you said was the most common method you have seen?

Right now I envision four possibilities for what the program reads from the drive every time it samples:

1. number of errors since last sample
2. number of errors since test started
3. number of errors in the last 75 sectors immediately preceding the instant at which the sample was taken
4. number of errors in the last full "chunk" of 75 sectors, where these "chunks" are aligned on integral 75-sector boundaries

If so, then I have to agree with you. That would be the superior method, and it would guarantee not to miss any sector regardless of testing speed. It also seems to me that it would be the simplest method to implement and use, both from the chipset designer's perspective and the testing software developer's perspective. But, if the chipset doesn't support it (which apparently the MediaTek chipsets used in the LiteOn drives don't), then what can be done about it? I guess we just have to live with the limitations of the polling and sampling method.

cfitz

[spath]

> Do you mean method 1 that I postulated earlier and that you said was the
> most common method you have seen?

yes, that one

[dodecahedron]

Ash nazg durbatulûk, ash nazg gimbatul, ash nazg thrakatulûk agh burzum-ishi krimpatul.

you have spoken words in the Black Tongue, so i had to answer.

[KCK]

Apparently a substantial portion of my recent exchanges with cfitz was caused by fairly simple misunderstandings due to lack of precision. Instead of decyphering who meant what, I'll try to summarize my view of KProbe in a simplified model that runs as follows.

Algorithm KProbe1 (simplified version of KProbe)
Step 1: Read Current Position into LBA_ini.
Step 2: Start accumulating from sector LBA_acc.
Step 3: Read Current Position into LBA_cur.
Step 4: If LBA_cur < LBA_ini + Delta_sam, go back to Step 3.
Step 5: Read C1C2.
Step 6: Set LBA_ini := LBA_cur.
Step 7: Go back to Step 2.

My assumptions are as follows. At Step 1, LBA_ini is set to the number of the latest sector read by the drive. Step 2 starts accumulating C1C2 errors from sector LBA_acc onwards, where LBA_acc > LBA_ini needn't be known to KProbe1. At Step 3, LBA_cur >= LBA_ini. At Step 4, Delta_sam >= 75 is fixed. Letting LBA_fin >= LBA_cur stand for the latest sector processed when Step 5 is reached, Step 5 outputs C1C2 errors accumulated for sectors LBA_acc through min{LBA_acc+74,LBA_fin}.

Introduce an outer loop counter k by setting k:=1 at Step 1, k:=k+1 at Step 7. Let LBA_ini(k), LBA_acc(k), LBA_cur(k), LBA_fin(k) denote the current values of LBA_ini, LBA_acc, LBA_cur, LBA_fin at Step 5. Then Steps 6 and 4 yield the basic relation

LBA_cur(k+1) >= LBA_cur(k) + Delta_sam.

Hence the estimated sample size Delta_est(k):=LBA_cur(k)-LBA_cur(k-1) satisfies Delta_est(k) >= Delta_sam for k>1. It should be distinguished from the accumulated sample size Delta_acc:=min{75,LBA_fin-LBA_acc+1}, which (using LBA_cur(k-1)=LBA_ini(k)) may be expressed as

Delta_acc = min{ 75, Delta_est + Delta_r - Delta_a}

in terms of the accumulation delay Delta_a:=LBA_acc-LBA_ini-1 of Step 3 and the read delay Delta_r:=LBA_fin-LBA_cur of Step 5. If both delays are absent (or Delta_r=Delta_a), Delta_acc=min{75,Delta_est} as expected. However, Delta_acc<75 is possible, as shown below.

Example 1: LBA_ini=200, LBA_acc=203, LBA_cur=276, LBA_fin=276 for k=2; then Delta_est=76 and Delta_acc=74.

Assume Step 5 outputs LBA_cur followed by the C1C2 counts into a csv file. Then, by inspecting the csv file, we can recover each Delta_est(k), but not Delta_acc(k). Thus we can't be sure whether the C1C2 counts were accumulated over 75 or less than 75 sectors!

Next, the basic property Delta_est(k) >= Delta_sam combined with Delta_sam >= 75 implies that the LBA values in the csv file should be separated by at least 75.

In contrast, both cfitz and MediumRare have mentioned KProbe's csv files with some samples separated by less than 75 sectors, and apparently such samples may be produced even when no reading errors occur. Does anybody have any idea why KProbe can produce such output?

Maybe KProbe employs a smaller Delta_sam? Here consider

Example 2 (Karr's "High Performance System"): LBA_ini(1)=100, LBA_cur(1)=174, LBA_ini(2)=174, LBA_cur(2)=248; such values could be produced by KProbe1 for Delta_sam=74 (not 75!).

BTW, assuming momentarily that the outputs of KProbe and KProbe1 are similar, "under-sampling" may refer to two different cases:
(a) the estimated sample size Delta_est(k) is less than 75;
(b) the accumulated sample size Delta_acc(k) is less than 75.
My former concern has concentrated on case (b), which apparently can't be detected in csv files. This post has originated from my inability to understand case (a) reported by cfitz and MediumRare.

Consider now another variant called KProbe2, assuming that the "Read C1C2" command of Step 5 additionally delivers the number LBA_fin of the latest processed sector.

KProbe2 is derived from KProbe1 by outputting LBA_fin (instead of LBA_cur) at Step 5, and setting LBA_ini := LBA_fin (instead of LBA_ini := LBA_cur) at Step 6. This boils down to resetting LBA_cur to LBA_fin at Step 5. Hence all the preceding relations hold for KProbe2 as well (with a null read delay Delta_r).

Of course, neither KProbe1 nor KProbe2 can model other important features of KProbe (e.g., handling of the beginning/end of the disc, or reading/slipping errors). Still, I hope this post will make it easier to formulate more specific questions on KProbe, which hopefully Karr will be able to answer.

[cfitz]

Yow! Lots and lots of variables... Well, even if it doesn't make for easy reading it does make things more explicit, which I believe was your goal.

My assumptions are as follows. At Step 1, LBA_ini is set to the number of the latest sector read by the drive. Step 2 starts accumulating C1C2 errors from sector LBA_acc onwards, where LBA_acc > LBA_ini needn't be known to KProbe1.

That is assumption 1 that I was insisting all along that you were making:

1. You appear to be assuming that KProbe uses the LBA from step 1 to decide how many blocks it must wait ...

This assumption may be correct, but it may not be. For instance, KProbe may actually reverse steps 1 and 2 from how you have listed them. Then, even though the exact LBA_acc value isn't known, it can at least be guaranteed that LBA_ini >= LBA_acc, the opposite of the scenario incorporated in your assumption. This would eliminate the possibility of what you refer to as under-sampling type (b):

(b) the accumulated sample size Delta_acc(k) is less than 75.

Of course, it wouldn't explain under-sampling type (a) where Delta_est < 75, but as I indicated in my earlier post, the vast majority of this type of under-sampling is for unreadable sectors where the entire issue of sampling is moot and the 70-block offset is simply the offset Karr selected to skip over the bad sectors. The remaining ones I can't explain. They could be glitches where the PUH slipped and reset itself or something like that. I do not know.

the estimated sample size Delta_est ... should be distinguished from the accumulated sample size Delta_acc

Delta_acc<75 is possible

I have understood these points all along (the second one being contingent on your assumption 1). That is why I said "support" rather than "prove" when I discussed my additional test results today:

This offers additional evidence to support the contention that KCK's concern about under-sampling, while potentially theoretically valid, is not warranted in practice.

If your assumption 1 is correct, then you are correct that the fact that Delta_est >= 75 does not prove that Delta_acc >= 75. However, when all of the Delta_est values exceed 75, and the vast majority exceed it by 5 blocks or more, it does support the notion that under-sampling of either type you have defined is not an issue.

You've obviously put a lot of thought and effort into this. Hopefully it will prove useful in the manner you intended.

cfitz

[KCK]

cfitz:

Referring to our prior exchanges in my previous post, I hinted that I saw little merit in arguing about "who meant what". Hence I'm quite suprized that you are still discussing your assumption 1 on the assumptions which, according to you, I might have employed in my posts prior to the previous one. Let's just say that your assumption 1 would need additional assumptions in order to be discussed in more precise terms; recall your three-paragraph explanation from http://www.cdrlabs.com/phpBB/viewtopic. ... 2808#62808.

Anyway, let's get back to my previous post.

Concerning my assumptions:

"At Step 1, LBA_ini is set to the number of the latest sector read by the drive. Step 2 starts accumulating C1C2 errors from sector LBA_acc onwards, where LBA_acc > LBA_ini needn't be known to KProbe1."

are you questioning their validity? If yes, please tell my why.

If you wish to discuss another variant of KProbe1 that somehow "reverses" Steps 1 and 2, please list it (copy, paste, modify). (Here I don't want to make assumptions on your assumptions!) I doubt whether it will be close to Karr's examples, but let's wait and see.

Concerning under-sampling of type (a) where Delta_est < 75, it might help if MediumRare could inspect his csv files to tell us whether, in his opinion, samples separated by less than 75 sectors could be attributed to reading/slipping errors. Maybe separation by precisely 70 sectors indicates some special cases, but this would have to be confirmed by Karr.

Finally, my doubts about technical details don't invalidate your general conclusions. Specifically, even if KProbe1 described KProbe exactly, I wouldn't worry about under-sampling in practice.

[cfitz]

Referring to our prior exchanges in my previous post, I hinted that I saw little merit in arguing about "who meant what". Hence I'm quite suprized that you are still discussing your assumption 1 on the assumptions which, according to you, I might have employed in my posts prior to the previous one.

KCK, in all earnestness, I am not fighting with you, attacking you, or casting aspersions on you. I am sorry if I have offended you, and apologize if there are hurt feelings. It was not my intention to offend. I mentioned "assumption 1" simply because your new, detailed framework gave me the opportunity to clarify what I had been trying to say earlier but was unable to convey clearly.

If you wish to discuss another variant of KProbe1 that somehow "reverses" Steps 1 and 2, please list it (copy, paste, modify). (Here I don't want to make assumptions on your assumptions!)

Actually, I don't want to discuss another variant, since it is just speculation about what Karr may or may not be doing, and I feel that I have already spent too much time on this one issue, which I still don't see any evidence to be a serious one. But, since I brought it up, I will try to explain the variant:

Step 1: Start accumulating from sector LBA_acc.
Step 2: Read Current Position into LBA_ini.
Step 3: Read Current Position into LBA_cur.
Step 4: If LBA_cur < LBA_ini + Delta_sam, go back to Step 3.
Step 5: Read C1C2.
Step 6: Go back to Step 1.

Note that I have neglected start-up and shut-down conditions at the beginning and end of the disc. These steps are intended only to represent the steady state while the test is ongoing. Also, steps 1 and 2 incorporate an assumption that the Start Accumulating command does not return until the drive actually starts accumulating data. I think I mentioned this in an earlier post. Thus, although the algorithm does not know the exact value of LBA_acc, it does know that LBA_ini >= LBA_acc.

I doubt whether it will be close to Karr's examples, but let's wait and see.

I don't think it does match Karr's example as closely as your KProbe1 algorithm. But that is why I earlier said "it is possible that you are reading too much into a couple of artificial examples Karr created to illustrate a different point." I'm not sure whether Karr intended his examples to have sufficient detail to discuss all the finer points we have been discussing. I think he was just trying to illustrate that an excessively slow system will end up skipping blocks between the chunks of 75 blocks that it does sample. He may have provided enough detail in that example to draw more detailed conclusions about his sampling algorithm, and you may be interpreting that detail correctly, but it is also possible that he did not anticipate the direction of the resulting discussion and didn't provide a sufficiently accurate and detailed example to permit the exploration of finer points of his collection algorithm. I just don't know, and can not know without additional input from Karr.

So, I will return to one of my earlier comments, and this time I will try to stick my convictions with a little more fortitude:

In the end, I feel that there are too many unknowns and assumptions regarding the nitty details to make meaningful conclusions about those details.

There are no hard feelings at this end, and I certainly hope there aren't any at your end.

Peace.

Over and out.

cfitz

[MikeTR]

Ash nazg durbatulûk, ash nazg gimbatul, ash nazg thrakatulûk agh burzum-ishi krimpatul.

you have spoken words in the Black Tongue, so i had to answer.

So had I:

A Elbereth Gilthoniel,
silivren penna mnriel
o menel aglar elenath!
Na-chaered palan-dnriel
o galadhremmin ennorath,
Fanuilos, le linnathon
nef aear, sn nef aearon!


I think we'd better get back on topic now. Wouldn't want to disturb the fascinating exchange between these techies .

[MediumRare]

Just a quick note on 2 subtopics (I'm really tired ):
MikeTR and Dodecahedron- I'd throw some more poems at you, but the copy of LOTR I have right now is German and my memory is too fragmentary to reach back to much from the last time I read it in English (about 20 years ago). So back to the main topic.

KCK and cfitz- I won't engage in models of KProbe but have some additional information on the "sample length" = Delta-LBA:
- the short deltas (=70) I saw are not pathological in any way- the C1 count is similar to the neighbours, the neighbours have normal lengths, no skipping.
- I can confirm cfitz' observation re. CD Doctor- no short deltas, those of length 0 were followed by double-length deltas.
- I'm curious as to which time is reported for the samples: normally the first value is around 150 LBA, but the last value is often greater than the length reported by K's Probe. For the series of MRW tests I did (read this as 12 scans of the same RW), the Delta (length - last LBA) varied from +8 to -66, most were negative. On the "good CD" I tabulated, the last LBA was greater than the length in 1 of 5 scans, and <20 less in the other cases.

G

[MediumRare]

I was wondering how strong the influence of the "Real-Time chart" was on the performance of K's Probe- Karr Wang recommends turning it off for better accuracy.
The results are shown in the following chart. I added a couple of sanning rate variations, just to make it more interesting. All scans are for a Verbatim Ultraspeed, Mount Rainier formatted, full (11 files).



Note how the peak of the distribution shifts to higher Delta-LBA values as the scan spped goes up. When scanned @4x, the majority of the Deltas are 75, @8x it's 76 and @max, the value is 81.

Turning on the realtime chart flattens out the distribution considerably and the peak value goes up to 91. So now you can see why Karr Wang recommends turning it off.

Let me explain in a bit more detail. Since each sample counts 75 sectors, the Delta-LBA value shows how much of the disk is not explicitly registered in the C1/C2 counts. As cfitz already noted, this should not really affect our conclusions regarding the burn quality, since the sample is representative for the disk as a whole.

The diagram shows that by scanning at lower speeds, more of the disk is explicitly examined:at 4x, only 0.5% is missed (75.4/75 -1), at 8x ca. 1.2% and at maximum speed (here) ca. 9-10%. However, the C1 (and normally C2) counts are not just a function of the disk, but also of the drive and speed used to read it. The average C1 values for this RW were 1.8 @4x, 1.3 @8x, and 4.7 @max- so reducing the scan speed also affects the "quality", i.e. C1 count. Normally we do want to test under the same conditions as when we read a CD, i.e. at maximum read speed. So that's why deactivating the realtime chart is important if we also want to reach the greatest possible sampling fraction. When it is turned on, the average C1 is essentially unchanged, but the portion of the disk not sampled increases from 10% to 22%,

G

[KCK]

cfitz:

I don't know why you felt the need for apologies. I just thought it would be better to advance the current framework instead of trying to clarify the meaning of our earlier posts, but if you felt different, it is I who should apologize! Still, I do appreciate your concern.

Let's call your variant KProbe3. As for technical details, apparently it's enough to assume that LBA_ini >= LBA_acc, without making more stringent demands on the "Start accumulating" command. Relative to KProbe1, KProbe3 has the advantage of ensuring that C1C2 errors are accumulated over 75 sectors (if Delta_sam >= 74 for KProbe3, whereas Delta_sam >=75 for KProbe1), at the possible expense of making more inner loops between Step 3 and 4, which might in relatively infrequent cases result in skipping more sectors for the C1C2 counts.

In general, it's impossible to get a more precise idea about the dynamics of KProbe1 and KProb3 without having at least rough estimates for the delays associated with each command (of course, delays expressed in LBA values may grow with reading speeds). Apparently Karr's distinction between high and low performance systems boils down to whether the inner polling loop is nonvacuous (i.e., returns from Step 4 to 3 occur).

Of course, it would be very interesting to hear from Karr whether KProbe1 or KProbe3 is closer to KProbe.

MediumRare:

I also have csv files with the last LBA value greater than the length of the disc reported by KProbe. On the other hand, the fist LBA value is usually around 150 (156 seems to be most common), but in some cases it can be as low as 69 (even for discs which don't exhibit reading or slipping errors, as confirmed by CD Speed).

Although initial and final sample counts needn't invalidate the overall statistics of KProbe, I would advise Karr to check the first and last printouts.

As for your example of "Distribution of Delta-LBA", let me add a couple of remarks.

The fact that turning the realtime chart on may result in significant decreases of sampling fractions becomes even more important on slower systems. For instance, on one of my discs (read at max speed) the sampling fraction decreases from 79.9% to 63.6% (the fractions are computed by dividing the number of lines in the csv file multiplied by 75 by the final LBA reported by KProbe). This was obtained on my P4-M 2GHz box with LTR-48125W connected via FireWire.

Of course, sampling fractions for the whole disc are just rough indicators. For instance, if the "local-mean" Delta-LBA values grow linearly with LBA (as discussed by cfitz), being about 77/84/91 for the beginning/whole/end of the disc, the sampling coverage is about 97/89/82%. This is simply due to missing relatively more sectors at higher reading speeds.

Yet another point is that Karr's improvement for reading/slipping errors might have lead to deterioration of sampling fractions. More specifically, although I have no idea how detection of such errors could be handled in models such as KProbe1, it seems that KProbe might be highly sensitive to any changes in its basic command loop. Hence maybe earlier versions (e.g., 1.5) are better in terms of sampling fractions.

[MediumRare]

KCK:
just a bit more information in response to your last post:

Of course, sampling fractions for the whole disc are just rough indicators. For instance, if the "local-mean" Delta-LBA values grow linearly with LBA (as discussed by cfitz)...

I think that the average fractions are fine for practical purposes- the variation is connected to the transition from reading @CLV to CAV when scanning the disk. The Deltas-LBA values are pretty well constant throughout the disk for scans @4x and 8x. The scan @max has essentially the linear increse that cfitz noted. I made additional scans @16x (CLV) and 24x (CAV) and they confirmed this.

... even more important on slower systems

I don't think my system is faster than yours. I have an Athlon XP1700+ and a LTR48246S. Please note that the values I reported are for a RW, where the max. read speed is 40x.

Hence maybe earlier versions (e.g., 1.5) are better in terms of sampling fractions.

I did a scan of the Ultraspeed MRW with Version 1.1.4 and the sampling fraction is does not differ from the newer version. In fact, there are some stray values of 300-700 blocks. The newer version is definitely better

G

[CDHero]

I add a feature in v1.1.9.
Please visit the KProbe website.
This feature is showing the disc manufacturer name.
If Kprobe cannot recognize your disc , please send the
LeadIn and LeadOut information to me , thanks!!!

[MediumRare]

I add a feature in v1.1.9.
Please visit the KProbe website.
This feature is showing the disc manufacturer name.
If Kprobe cannot recognize your disc , please send the
LeadIn and LeadOut information to me , thanks!!!

Hi Karr Wang
Thank you for the new version. I just tried it out on a Verbatim DataLife Pastel disk, and there was a problem (more tests will follow when I have time- I'll send them per e-mail). Here's the ATIP:

Code: Select all

Verbatim DL Pastel 32x
ATIP:              97m 24s 01f
Disc Manufacturer: Taiyo Yuden Company Ltd.
Reflective layer:  Dye (Long strategy; e.g. Cyanine, Azo etc.)
Media type:        CD-Recordable
Recording Speeds:  min. unknown - max. unknown
nominal Capacity:  702.83MB (79m 59s 73f / LBA: 359848)

Your Probe identifies this as Sony, also available in Disc Info.

I think it's a good thing to have this information- Maybe you could add the ATIP and nominal capacity to the display too ?

G

[cfitz]

Thanks Karr! That is a very convenient feature. I know a lot of people have been requesting it.

cfitz

[rdgrimes]

Another nice feature! Still not reporting unreadable sectors though..

[CDHero]

Another nice feature! Still not reporting unreadable sectors though..

I also want to add the feature you mentioned.
But a unreadable sector does not mean it is
really unreadable.Sometimes it is caused by PUH slipping.
So , I still think how to cover this situation in KProbe.
I think whether the KProbe should retry n times if PUH slips.
Welcome to give me a suggestion.

[KCK]

MediumRare:

My box (P4-M 2GHz, 768 MB, XP Pro) is quite fast, but my LTR-48125W is probably slower than your LTR-48246S, and it's hard to guess whether my FireWire connection introduces significant delays for KProbe's commands relative to standard IDE connections.

It would help if KProbe gave one or more indicators for sampling coverage. Here is a specific proposal.

Let LBA_end be the final LBA reported by KProbe. Referring to the csv file with records of the form "LBA,C1,C2", let N_sam = # of records with C1 >= 0, N_beg = # of records with C1 >= 0 and LBA <= 0.1*LBA_end, N_end = # of records with C1 >= 0 and LBA >= 0.9*LBA_end. (Here we ignore records with C1 < 0 due to reading/slipping errors.) Let Sample_avg = 75 * N_sam/LBA_end, Sample_beg = 75 * N_beg/(0.1*LBA_end), Sample_end = 75 * N_end/(0.9*LBA_end). KProbe could output Sample_avg, Sample_beg, Sample_end as rounded percent values, e.g., in the form

Sampling [%]: Average = 84, Beginning = 91, End = 77

The total average sampling fraction Sample_avg is, of course, the most important indicator that should be output. It is enough for scans made at constant speeds. For scans made at max speed, Sample_avg alone may suffice if one keeps in mind cfitz's observations, but it wouldn't hurt to get some idea about the sampling coverages of the initial and final 10% fractions of the disc in the form of Sample_beg and Sample_end.

I do feel that at least Sample_avg should be output (the usefullness of the other indicators being open for debate). To save space, the output of Sample_avg could be controlled via the setup options.