Analysis of the accuracy of the recruiting services

[JasonB]

Out of curiosity, I did an analysis of the SMU roster to see what was the most accurate predictor of whether or not a player would become a starter.

My scoring: A player was a "win" if they have been considered a starter, "loss" if they are an upperclassmen and are not considered a starter, and a "unknown" if they are an underclassman and aren't starting. I wasn't making judgements of how good the players actually are, just whether or not they have landed a starting spot. *33 players on the roster are considered "unknowns", so there is a pretty good degree of play in these numbers.

My thought was that if the services can accurately predict who will be starting for us, then they probably do a pretty good job of actually determining who the best players are.

Rivals high ranking: 3 star, 5.6 and above. Medium is 3 star below 5.6, and low is 2 star.
Scout: High is 3 star and position ranking in top 100. Medium 3 star below 100. Low is 2 star.
ESPN: High is 74 and up, Medium is 70 - 73, Low is 69 and lower or unranked.

Best Predictor:
The analysis shows that the best predictor is actually the ratings of the services, and NOT the number of P5 offers a player has.

The highest predictors:
Rivals "High" rating - 83.33%
All three services rate "high" - 81%
More than one "high" - 75%
Scout "High" - 68%
ESPN "High" - 64%
Rivals 3 star (High or Low) - 63%
Multiple P5 offers - 60%
ESPN over 70 - 59%
P5 Offer - 57%
Scout 3 star (H or L) - 56%

Looking at the other side of the coin, predicting what players wouldn't start, was a little more challenging, because of the large number of low rated underclassmen on the roster. I counted those guys as ties, which made the percentages a lot higher.

Best predictor of players who won't ever start:
All three services rate low - 45% of the players end up starting.
Rivals or ESPN rate as low - 51% of the players end up starting.

Scout rating as low, or multiple sites rating as low wasn't a very good predictor at all.

Conclusions:
1) Rivals is definitely the more accurate service.
2) If you want to know who will push for starting spot, look at the Rivals rating or agreed high ratings by all three services
3) Players who are rated low by all three services actually are Diamonds in the rough and have a lower chance of becoming starters.
4) Offers, according to this research, might not be as big of an indicator as we thought.

Just thought I would share for you guys ahead of signing day...

[ftworthmustang]

Is the sample of players with multiple P5 offers large enough to even consider as statistically significant?

[Stallion]

You aren't measuring apples to apples because in some years Rivals 3+ stars was Texas Top 162 while Scout 3+ stars was Texas Top 223 (ex. 2015). In other years like in 2011 Rivals 3+ stars was (from memory) Texas Top 272 while Scout was something like Texas Top 200. Rivals used to be more stringent then it flipped and now has flipped again. Therefore your calculations have a huge statistical margin of error probably greater than the percentage differences you found. You simply can't compare Rivals and Scout star ratings through different years because the definition has repeatedly changed. You can't even compare Rivals to Rivals through the years based on star ratings Now you might be able to compare state rank or Texas Top 100 lists but not enough SMU recruits have probably been state ranked or in the Texas Top 100

[JasonB]

You aren't measuring apples to apples because in some years Rivals 3+ stars was Texas Top 162 while Scout 3+ stars was Texas Top 223 (ex. 2015). In other years like in 2011 Rivals 3+ stars was (from memory) Texas Top 272 while Scout was something like Texas Top 200. Rivals used to be more stringent then it flipped and now has flipped again. Therefore your calculations have a huge statistical margin of error probably greater than the percentage differences you found. You simply can't compare Rivals and Scout star ratings through different years because the definition has repeatedly changed. You can't even compare Rivals to Rivals through the years based on star ratings Now you might be able to compare state rank or Texas Top 100 lists but not enough SMU recruits have probably been state ranked or in the Texas Top 100

I know you just like to [deleted] about everything I post, but if you ha actually read you would have seen that is exactly WHY I categorized recruits as high, medium, and low. 5.6 rating for Rivals, 3 star top 100 position for Scout, and 74+ on ESPN are about the levels where a player would actually be rated as a top 100 prospect at their position. That is why I chose those comparison points - to make it apples to apples.

The second level - the OK level - that is a little more apples and oranges, because it is comparing the remaining 3 star rivals with 3 star scout and 70+ on ESPN.

As far as the other question about the P5 offers:
Kids with multiple BCS offers: 6 starters, 3 non, 6 unknown.
Kids with at least 1 BCS offer: 9-6-6

You can determine for yourself if that is statistically relevant, but good question....

Some other interesting info:
Kids with a "major" p5 offer - 5-3-6
Kids with only "minor" p5 offers (ex. Vandy) - 4-3-0
Rivals rates as "high": 7-1-1 (very telling the low number of rivals high rated recruits)
All 3 services are high: 6-1-1 (when rivals rates them as high, they pretty much all do)
All 3 services rate as low: 4-6-10. So, we have way more recruits where all 3 services rate as low than have where all think highly. And a lot of them are young (10 unknowns), which points out the recruiting rankings of the past two classes).

[Stallion]

You haven't proved jack because you haven't shown one defined category is similar statistically to another including as rating services have changed their own definitions over several years

[couch 'em]

We have so few quality recruits, and so many that P5 schools avoided because of issues like academics, and such a horrible and bizarre June coaching staff, that I don't think we have a good sample size. It would be much more representative to look at a sane, upper tier G5 to analyze.

[hubberlang]

Agree, such a small sample of kids had p5 offers I am not sure this tells you much. Look At this in a couple years when we have 30 kids on roster with p5 and then we will see. You have to like what ccm is doing regardless

[ponyboy]

You haven't proved jack because you haven't shown one defined category is similar statistically to another including as rating services have changed their own definitions over several years

Can't understand why you always act in such bad faith when he comes to JasonB. I don't believe he was trying to [b]prove[/] anything in the mathematical or statistical sense. The way I read his post was he was just trying to provide some information. If not dispositive, pretty interesting in my opinion.

[JasonB]

Because I have been one who has traditionally expressed my hesitancy with the recruiting service's evaluation practices, I thought people would be interested in the results showing how accurate Rivals has actually been over the past several classes.

[Puckhead48E]

You haven't proved jack because you haven't shown one defined category is similar statistically to another including as rating services have changed their own definitions over several years

Honestly, not quite sure what you are trying to say...I think you have been reading too many SOC decommit tweets.

The first thing Jason is proving is that, well, Jason has much too much time on his hands. The second thing he is showing is, with a limited sampling size, the services do not all rate all the same and whatever their respective levels are each year, they are not all exactly equal in efficacy. Oh well, nothing to yell about.

[ponyboy]

I think everyone agrees, in the vital background context of fit to system, that offers are the single best predictors of recruit quality. But there is a big problem with the quality of the offer data itself. Was someone offered early to solidify a slot, but had the offer withdrawn when something better came along? Would a recruit have been offered by multiple schools, but was solid early with a school? Etc, etc. It's the best we have, but far from perfect.

[The PonyGrad]

Hey, I'm sure it was a fun exercise and interesting FWIW. Thanks for sharing that.