What does a CAC sticker really mean? -- A proposal

[brg5658]

Please humor me for a few minutes. I invite you to read the below proposal with an open mind. Try not to dismiss it immediately as a futile academic exercise. I am a scientist, and such data as requested below would be highly useful to me (and others) to draw conclusions about the CAC service.

 

In many areas of our everyday lives, we take for granted the reliability and validity of things such as medical diagnostic tests. For simple analogy, I will use a pregnancy test throughout to help understand my arguments. I design studies for a living, so the methodology proposed here is the standard in such fields as medicine, IT, aerospace, etc.

 

The feasibility of such an experiment in the context of a major coin show would not be difficult. Though, it would require approximately 2-3 hours of time from the CAC finalizer (John Albanese) and 4 (or more) other very well respected numismatic leaders in the field. In addition, it would require preparation time on the part of facilitators, and the cooperation of CAC in the reapplication of stickers to appropriate coins after their independent review by other numismatic experts (see design methods below).

 

---------------------------------

RELIABILITY is the ability of a test by the same means to give repeated and similar results. For example, if one were to take a pregnancy test that was positive, and then take the same pregnancy test a 2nd time, the test would be considered reliable if it were to give the same result each time. In terms of CAC, we can assess the reliability by taking a group of coins with known first submission CAC results, resubmit them to the same CAC service, and determine if the same outcome came about. This type of assessment would actually not be that difficult. For example, take 100 coins that have previously been submitted to CAC, remove all stickers, and resubmit the coins for a 2nd time to the CAC service for opinion. If all coins came back with the same CAC decision, then the service would be considered highly reliable (reliability is also sometimes called the ability to repeat or replicate results). For this measure of reliability to be meaningful, CAC personnel (namely John Albanese) would not be allowed to consult the CAC databases of previous submissions in the assessment of the coins the 2nd time (which is supposedly already what happens).

 

---------------------------------

VALIDITY is the ability of a test to actually measure the quality it claims to be measuring. In the pregnancy test analogy, the test would only be valid if when it gave a positive result the woman is actually pregnant. And, conversely, it is valid if when it gave a negative result the woman is actually not pregnant. In the case of CAC, validity can only be measured by involving other relevant outside opinions. For example, if for the same 100 coins that were submitted to CAC for re-assessment for the bean, another set of independent market "big hitters" were to assess the same coins and give their opinions of the "C" status (fail or no CAC), "A/B" status (pass or green CAC), and "A+" status (exceed or gold CAC).

 

I actually wrote up (for my own humor) an entire experimental design proposal for this about 6 months ago, as I was curious who these outside opinions could be. I proposed in my (hypothetical experiment) that they would be Q. David Bowers (Stacks), David Hall (PCGS), Mark Salzberg (NGC), and Mark Feld (Heritage). I thought this was a good list of broad representation and with name recognition in the field. Each of these 4 persons would evaluate the 100 coins blinded to the previous "CAC bean" status, and mark each as either fail, pass, or exceed the grade on the holder in their professional opinion. What this assessment would tell persons is whether the opinion of CAC with respect to "high end for the grade" jibed with another set of external numismatists. In other words, is the CAC sticker really measuring what we think it is in terms of the broader numismatic opinion. Or put in lay-mans terms -- does the CAC bean jibe with a likely true "high end" coin in the eyes of multiple independent reviewers.

 

---------------------------------

BOTH reliability and validity are required for a test to be considered useful. For example, if one developed a pregnancy test that always returned a positive result, the test would be highly reliable. In other words, it would be repeatable or replicable always giving a positive result. However, it would not be valid as it is not actually measuring the state of pregnancy as advertised.

 

With regard to CAC, reliability is the ability of CAC to consistently give the same opinion of a coin when submitted multiple times. That process in and of itself does not guarantee that the opinion offered by CAC jibes with the opinion of the larger numismatic community. However, if a panel of other respected numismatists consistently agreed with the assessment of CAC, the service would also be considered valid to the market. Both reliability and validity of the CAC service would support (with data!) the current market premium paid for CAC coins.

 

---------------------------------

OTHER METHOD DETAILS:

• The 100 coins would include 10 coins from each of 10 different classic series (pre 1930s).
  
• The 10 coins from each series would consist of 5 NGC graded coins and 5 PCGS graded coins (a total of 50 from each service). Such a design would allow the assessment of any conscious or subconscious bias toward "high end for the grade" status given to one grading company or another. If the validity and/or reliability differed across the two brands, this would support a bias in the way the CAC status is applied.
  
• TPG assigned grades within each series would range from XF through MS. This allows to assess reliability and validity separate for circulated and uncirculated coins.
  
• All of the 100 coins will have been previously submitted and reviewed by CAC; however, only approximately 40-60% of the coins chosen for review will have received the green or gold CAC sticker. The others will have been rejected by CAC for a sticker at first review. This design guarantees that both the rejection agreement and acceptance agreement can be assessed.
  
• All CAC stickers would be removed from approved coins before the review process to ensure blinding of the reviewers. The re-application of CAC stickers based on their database records after the completion of the study would need to be approved and okayed by the CAC service.
  
• The 100 coins will be reviewed in a stratified random order such that no two coins of the same series can be compared "side by side" during review (independent review in random blocks of 10 coins -- one coin from each series).
  
• Each reviewer will assess each coin by marking one of three boxes. 1) "C" -- CAC fail equivalent, 2) "A/B" -- CAC pass equivalent, 3) "A+" -- CAC exceed equivalent. Only one definitive response per coin is allowed.
  

---------------------------------

CONCLUSIONS: The reliability and validity of the CAC service can be assessed across many different coin series, and many different grade levels. These data would reassure collectors that the CAC service is consistent in its application of the approval sticker on the same coin submitted multiple times, and that the sticker is in actuality measuring a numismatic quality that extends beyond the personal opinion of just CAC. The independent publication of such results could significantly impact the market acceptance of the CAC program. However, results will not and can not in any way reflect on the quality of a coin in a PCGS or NGC holder that does not display the CAC approval sticker.

[mkman123]

I like it! Lets increase it to 1000 coins though

[physics-fan3.14]

I think this is a fantastic idea, and would be willing to send some coins to be a part of this test. 100 is a good number (although detractors will say the sample size is too small). 1000 would be better of course, but the time and expense would be impractical.

 

I do not have any contacts to help make this happen, but I know there are people on the boards who do.

 

Actually, thinking about it, this seems like something that a respected, independent third-party would be best to conduct. I know a while ago CoinWorld did something similar with TPGs. Might it be a good idea to propose the idea to them, and have them conduct it (to remove all possible perceived bias - a true piece of investigative journalism)? That would also help to get the publicity that something like this seems like it would deserve.

[stldanceartist]

I would be willing to contribute some coins to this test as well.

 

I'm just curious how you would take into account the variable of human opinion into a scientific experiment, or if you are just applying some principles from the scientific method to reach a conclusion, and if the results could be considered scientific and conclusive based on what parts of the scientific method are present or not present.

[AnkurJ]

You have three experienced graders at CAC looking at the coins. It baffles me as to why that not enough. If you show a coin to 20 different people, you may get 10 or more different opinions. Frankly I think the experiment is a waste of time.

[Hard Times]

Fantastic idea - if the CAC graders are as good as some think, they have nothing to worry about and would only gain even more credibility.

 

Best, HT

[mumu]

You have three experienced graders at CAC looking at the coins. It baffles me as to why that not enough. If you show a coin to 20 different people, you may get 10 or more different opinions. Frankly I think the experiment is a waste of time.

 

You have three experienced graders at PCGS/NGC looking at the coins. It baffles me as to why that not enough. If you show a coin to 20 different people, you may get 10 or more different opinions. Frankly I think the experiment is a waste of time.

[mumu]

If CAC passes the test, then they as a service should be awarded a sticker that they place next to their own stickers, assuring anyone buying the stickered coin that this independent test took place and validated the CAC service as accurate.

[AnkurJ]

You have three experienced graders at CAC looking at the coins. It baffles me as to why that not enough. If you show a coin to 20 different people, you may get 10 or more different opinions. Frankly I think the experiment is a waste of time.

 

You have three experienced graders at PCGS/NGC looking at the coins. It baffles me as to why that not enough. If you show a coin to 20 different people, you may get 10 or more different opinions. Frankly I think the experiment is a waste of time.

 

The reason CAC was created is to fight over graded and problem coins being put into holders! While CAC is a business, their intent is to make the hobby better by weeding out overgraded and problem coins.

[Mark T]

You have three experienced graders at CAC looking at the coins. It baffles me as to why that not enough. If you show a coin to 20 different people, you may get 10 or more different opinions. Frankly I think the experiment is a waste of time.

 

+1

 

Every CAC discussion I have seen here morphs into a multiple page mess.

 

I think it is rather unlikely that CAC would have anything to do with such an “experiment” .

[Hard Times]

You have three experienced graders at CAC looking at the coins. It baffles me as to why that not enough. If you show a coin to 20 different people, you may get 10 or more different opinions. Frankly I think the experiment is a waste of time.

 

You have three experienced graders at PCGS/NGC looking at the coins. It baffles me as to why that not enough. If you show a coin to 20 different people, you may get 10 or more different opinions. Frankly I think the experiment is a waste of time.

 

The reason CAC was created is to fight over graded and problem coins being put into holders! While CAC is a business, their intent is to make the hobby better by weeding out overgraded and problem coins.

 

Sorry, can't help it, so they green bean a 63 and then bean it again green as a 64, how is that helping weed out overgraded coins? I am lost pure and simple in how that logic works.

 

Best, HT

[dimefreak]

You have three experienced graders at CAC looking at the coins. It baffles me as to why that not enough. If you show a coin to 20 different people, you may get 10 or more different opinions. Frankly I think the experiment is a waste of time.

 

You have three experienced graders at PCGS/NGC looking at the coins. It baffles me as to why that not enough. If you show a coin to 20 different people, you may get 10 or more different opinions. Frankly I think the experiment is a waste of time.

 

The reason CAC was created is to fight over graded and problem coins being put into holders! While CAC is a business, their intent is to make the hobby better by weeding out overgraded and problem coins.

 

The reason PCGS/NGC was created is to fight over graded and problem coins from being put in holders! While PCGS/NGC is a business, their intent is to make the hobby better by weeding out overgraded and problem coins.

[FishyOne]

You have three experienced graders at CAC looking at the coins. It baffles me as to why that not enough. If you show a coin to 20 different people, you may get 10 or more different opinions. Frankly I think the experiment is a waste of time.

 

You have three experienced graders at PCGS/NGC looking at the coins. It baffles me as to why that not enough. If you show a coin to 20 different people, you may get 10 or more different opinions. Frankly I think the experiment is a waste of time.

 

The reason CAC was created is to fight over graded and problem coins being put into holders! While CAC is a business, their intent is to make the hobby better by weeding out overgraded and problem coins.

 

Sorry, can't help it, so they green bean a 63 and then bean it again green as a 64, how is that helping weed out overgraded coins? I am lost pure and simple in how that logic works.

 

Best, HT

 

Huh? Are you still stuck on that? I thought the dozen posts in the other thread pointing out the flawed logic here would be enough but apparently not.

 

That coin is a solid 64 and worth 64 to CAC, simple as that. As a 63+ they felt the same. What in the heck does that have to do with the question? This has gotten to the point that it's like talking to a teenage kid who knows all the answers but refuses to learn anything from anybody else. Grow up.

[brg5658]

You have three experienced graders at CAC looking at the coins. It baffles me as to why that not enough. If you show a coin to 20 different people, you may get 10 or more different opinions. Frankly I think the experiment is a waste of time.

 

I have to call a spade a spade here -- Ankur, you are quite possibly the biggest CAC fan-boy on the planet. You felt the need to create a completely closed and private forum for the topic, and you apparently live very close to their shop so you get a lot of insider elbow rubbing. That being said, given your posts on these boards as long as I have been here (3+ years), the superficiality of your understanding of coin grading and numismatics in general is out in the open for all to see. Even the way you describe coins and their "surfaces" is presumptuous and makes me almost feel sorry for you. Am I perfect, no -- I never said I was. But I think one of the signs of intelligence is a recognition of your limitations. The Dunning-Kruger effect is alive and well in the Kool-aid brigade. An opinion that you think such an experiment is a waste of time is probably more of a shout for its need than you realize.

 

Do I expect the proposal to ever fly -- no, not really. And, why you may ask? Because the last thing CAC would want people to know would be exactly what they are getting. If such a company would NOT want such a validation to occur tells you something about their "faith" in their product. God forbid that someone try to actually provide data to test the claims of the big players in the industry.

 

Alas, ignorance is bliss. Hand waving and cronyism are the norms in the TPG and "stickering" worlds of numismatics. In the end, it's all about money.

[Hard Times]

You have three experienced graders at CAC looking at the coins. It baffles me as to why that not enough. If you show a coin to 20 different people, you may get 10 or more different opinions. Frankly I think the experiment is a waste of time.

 

You have three experienced graders at PCGS/NGC looking at the coins. It baffles me as to why that not enough. If you show a coin to 20 different people, you may get 10 or more different opinions. Frankly I think the experiment is a waste of time.

 

The reason CAC was created is to fight over graded and problem coins being put into holders! While CAC is a business, their intent is to make the hobby better by weeding out overgraded and problem coins.

 

Sorry, can't help it, so they green bean a 63 and then bean it again green as a 64, how is that helping weed out overgraded coins? I am lost pure and simple in how that logic works.

 

Best, HT

 

Huh? Are you still stuck on that? I thought the dozen posts in the other thread pointing out the flawed logic here would be enough but apparently not.

 

That coin is a solid 64 and worth 64 to CAC, simple as that. As a 63+ they felt the same. What in the heck does that have to do with the question? This has gotten to the point that it's like talking to a teenage kid who knows all the answers but refuses to learn anything from anybody else. Grow up.

 

I am not going to stoop to your level and begin name calling. I obviously don't agree with your logic above as to how this might happen and I obviously believe I have to repeat myself because of the justifications some are presenting that I find to be incorrect.

 

Best, HT

[brg5658]

I would be willing to contribute some coins to this test as well.

 

I'm just curious how you would take into account the variable of human opinion into a scientific experiment, or if you are just applying some principles from the scientific method to reach a conclusion, and if the results could be considered scientific and conclusive based on what parts of the scientific method are present or not present.

 

The methods for assessing reliability and validity have existed for 60+ years. They are quite basic concepts in fields like medicine, epidemiology, and engineering. Many of the methods were developed in the field of medicine for such subjective things as interpreting X-rays or clinical tests. Human opinion (namely physician opinion) is at the core of those issues, not unlike the visual interpretation of a coin's grade. The only difference here is that a "gold standard" is not a pathologic finding, but it would be the consensus (or lack thereof) of the external 4 or more reviewers.

 

Reliability is agreeing with yourself on two separate occasions -- there is no "standard" of comparison.

 

Validity is agreeing with an external standard. In the case of the proposed study, this would be descriptive in nature. For example, on the coins for which CAC awarded the "pass green bean", what proportion of those coins were considered "pass green A/B" coins by 1 external reviewer, 2 external reviewers, 3 external reviewers, or all 4 external reviewers. If CAC is validly measuring some "gestalt" about the numismatic quality of the coin, then they would hope that for their "green bean" coins, almost all would have 3-4 external reviewers who agreed. Similarly but of less importance, for the coins that CAC did not give a "green" bean to, one would expect that few if any of the 4 external reviewers would give them "A/B" pass grades.

 

A total of 100 coins is a reasonable size sample that would not be over-burdensome to the reviewers, but at the same time would allow blocking and stratification on factors like the TPG company, the grade levels, and 10 different types of coins. I would foresee the experiment being most feasible in the context of a large national coin show, probably in an evening when all 4 external reviewers and a CAC reviewer would be convened in a single room, under the same lighting conditions, and away from distractions. Reviewers would be assigned a box of 10 coins at a time to assign fail, pass, exceed opinions -- being required to review a total of 10 boxes of 10 coins. Having 5 reviewers and 10 boxes of 10 coins would allow for reviewers to review the coins at differing speeds (If reviewer A finishes their first box quickly, they can be given one of the "standby" 5 boxes for subsequent review), but I would expect the total process to take 2-3 hours, and would likely require 3-4 organizers/facilitators to keep track of the scoring sheets and to assign the boxes to reviewers so they can remain outside-affiliatelinksnotallowed It would be similar to lot viewing standards and set-up.

[Tradedollarnut]

Sigh. Here we go again.

[brg5658]

Sigh. Here we go again.

 

I know Bruce, you don't like science or data. Just keep waving your hands over there...

[AnkurJ]

Brg: no need to feel sorry for me. I've learned to identify properly graded coins much better than I was able to before and also can identify problem coins much easier due to what I learned from CAC.

 

This is my last post in this thread as it's completely useless and a waste of my time to try to reason with some of you.

 

[Tradedollarnut]

Sigh. Here we go again.

 

I know Bruce, you don't like science or data. Just keep waving your hands over there...

 

You know nothing about my like or dislike for science or data. I find it interesting how you can state the TPG's are raft with cronyism and then disparage an independent company designed to keep such in check.

[AnkurJ]

Sigh. Here we go again.

 

I know Bruce, you don't like science or data. Just keep waving your hands over there...

 

You know nothing about my like or dislike for science or data. I find it interesting how you can state the TPG's are raft with cronyism and then disparage an independent company designed to keep such in check.

 

Bruce, I will suggest you save your time and energy.

[jom]

Sigh. Here we go again.

 

I know Bruce, you don't like science or data. Just keep waving your hands over there...

 

 

jom

[dimefreak]

He said fanboy

[brg5658]

Sigh. Here we go again.

 

I know Bruce, you don't like science or data. Just keep waving your hands over there...

 

You know nothing about my like or dislike for science or data. I find it interesting how you can state the TPG's are raft with cronyism and then disparage an independent company designed to keep such in check.

 

I guess your definition of "independent" differs from mine. Because, as far as I can tell, CAC is not anywhere close to being an independent entity from the TPGs. If anything they are parasitic or maybe symbiotic.

 

People would not accept such gray definitions in any other realms of their lives like their health care, their car mechanic, their appliances, even their toothpaste, etc. I just find it interesting that the "norm" in the coin grading (or verification of grading) business is that data is evil -- and if we don't all blindly follow the "know-it-alls" we are somehow the bad guys.

 

I am a consumer, and demanding to have data is the norm. This is the information age -- not the pontification age. What are people so afraid of if they think they are really offering a useful service? CAC has nothing to lose and everything to gain if the study supports what they are doing. But, keep us in the dark -- as if we were all strapped in place like in Plato's Allegory of the Cave and watching the shadows dance by on the walls.

 

Really, TDN, you think I expect an impartial assessment from you -- a principal in a huge coin dealership who deals in almost exclusively PCGS / CAC blessed coins?

[Tradedollarnut]

yawn

[brg5658]

yawn

 

Brilliant rebuttal.

[Tradedollarnut]

yawn

 

Brilliant rebuttal.

 

You've brought me down to your level

 

I'm done with these silly little threads...

[orifdoc]

Well, this is an interesting idea, but if you're really serious, a better study would be to take 500 coins that you are 100% positive had never been submitted, send them to CAC, record the result, crack them out, re-submit to PCGS or NGC with new cert numbers, wait 6 months or so, and then send them back through again. I'd repeat this until each coin had been through each TPG twice and CAC four times. It has to be blinded. If the people being studied know they're being studied, their behavior will change.

 

Of course, doing this will require 2000 grading fees at the TPGs and another 2000 grading fees at CAC. Expensive endeavor.

 

However, if I owned/operated a large coin wholesale operation (I'm talking of the really big boys here) I would have already completed a similar analysis on my own submissions & resubmissions. This information would be enormously valuable to those playing the crackout game and wouldn't be publicized.

 

The idea of including other industry leaders isn't terrible, but it introduces a variable that may or may not be relevant to the question you're trying to answer. My study would yield a huge amount of information that a statistician could use to evaluate the validity, accuracy, and reproducibility of the entire market grading process.

 

You could stratify the data by rarity, key dates, series, composition and learn a whole ton of information. This would be akin to the sabermetric process gaining traction in baseball. Whoever mastered this information and could use it to make a killing, even without knowing a single thing about numismatics.

 

Two things:

 

1) An Internet forum is a terrible place to discuss this. Any useful conversation will be endlessly and predictably sidetracked by people unfamiliar with the scientific process or who have a particular axe to grind.

 

2) Such analysis may be offensive to those who rely on imprecision to make their living. This includes crackout artists, the TPGs themselves, and a large number of dealers. TPG grading reassures the market but absolute precision would remove the speculative component of the market and would likely result in a lowering of value & liquidity.

[dimefreak]

 

Brandon- the guy has no worries... He employs a PCGS employee and is a business partner with CAC. Ontop of that His PCGS employee is quite the lady and thus his company has quite the following. I had to take a business ethics class to obtain my degree and my Professor quit his job at Delphi because they were going bankrupt at that time and he was being asked to do some really shady things. He couldn't do it so he quit and decided at the ripe age of 53 he would just teach and operate his own business. He taught us a lot of the "no no's" of business. With that said the people willing to do the "no no's" are going to be the most successful. Its just a matter of what you can sleep with at night!

[Tradedollarnut]

 

Brandon- the guy has no worries... He employs a PCGS employee and is a business partner with CAC. Ontop of that His PCGS employee is quite the lady and thus his company has quite the following. I had to take a business ethics class to obtain my degree and my Professor quit his job at Delphi because they were going bankrupt at that time and he was being asked to do some really shady things. He couldn't do it so he quit and decided at the ripe age of 53 he would just teach and operate his own business. He taught us a lot of the "no no's" of business. With that said the people willing to do the "no no's" are going to be the most successful. Its just a matter of what you can sleep with at night!

 

You are bordering on a very slippery slope