fbpx

Ranking the Rankers

An overly-quantified mock drafter performance review
By | 26 Comments | Jun 20, 2022

Jun 20, 2019; Brooklyn, NY, USA; A general view of the draft board following the first round of the 2019 NBA Draft at Barclays Center. Mandatory Credit: Brad Penner-USA TODAY Sports

The following is a reader submission from OmriOhMy!

For Kings fans, basketball seasons follow a well-worn rhythm: as our playoff hopes dwindle from naïve aspiration to pipe dream, we gradually hedge against the disappointment of yet another lottery season by getting acquainted with the new crop of draft prospects. 

While Kings fans are an informed bunch, by and large these draft crushes aren’t born out of disciplined research, but rather from the work or a few specific basketball writers. And while I read their work religiously each year, I always struggle with a burning question: Which writer should I trust? Do any of them consistently predict the draft order or player outcomes better than the rest? Or is the cottage industry of couch scouts more of an echo chamber with random results?

I decided to conduct an analysis to answer this questions for myself and the TKH community. I evaluated five prominent mock drafters’ performance over the past six drafts (2016-2021), attempting to determine which writer is the best at predicting player outcomes (as measured by win shares).

For the data-curious, feel free to download the analysis, draft results, and big board compilations from this link.

I used three quantitative frameworks to answer each of this question:

  • How well does a writer’s big board correlate with the draft class’s future win shares?
  • For each writer, what’s the average difference between a player’s big board rank and the player’s career win share rank (compared to the rest of his draft class)?
  • What percentage of this writer’s big board rankings land within 1, 2, or 2 spots of the player’s actual class win share rank?

Writers considered:

  • Sam Vecenie (The Athletic)
  • Bryant West (The Kings Herald)
  • Kevin O’Connor (The Ringer)
  • Jonathan Tjarks (The Ringer)
  • John Hollinger (The Athletic)

Before we get into my results, I want to lay out a few major, gaping caveats to my findings:

  • Pseudo-statistics disclaimer: What follows is my attempt at taking a quantitative approach to answering a very fuzzy, subjective question. My primary aim is to generate discussion, and I don’t presume that the spreadsheets I used are a crystal ball. It’s largely an exercise in pseudo-statistics and simple excel formulas and there are many aspects with which one could quibble (feel free to quibble away in the comments section – again, this is meant to generate discussion!). 
  • Synthesizing my findings into a digestible story: I ran a few different types of analyses, and they don’t always yield 100% consistent results – so what follows is a mix of objective and subjective findings, impacted in no small part by my own personal biases. I synthesized them into a narrative that makes sense to me.
  • Imperfect metric for player outcomes: For this analysis, I used win shares as a proxy for player success – it’s by no means perfect (especially for more recent drafts, whose draftees have skimpier statistical resumes) but for the sake of having a succinct output, I took it at face value.
  • Limitations in data collection: Not all writers have rankings released for every year, introducing substantial disparities in sample sizes across writers. Additionally, some writers only rank players 1-20 while others go past 100. So for the sake of this analysis (and since the Kings always draft in the lottery anyway), I limited players considered to the top 20 selected.
  • Purely numbers-based: I would have liked to supplement this with real examples from each writer’s rankings that drove the results, but that got messy very quickly and felt more like calling out writers’ mistakes than praising their hits – which doesn’t reflect my intention with writing this piece. There simply weren’t enough writers to establish a “consensus” and identify specific players each writer was above or below the consensus on.

Now, on to the results…

High level findings:

  • In general, the actual draft order turned out to be a better predictor of win shares than any individual writer’s projections. It seems like NBA executives are, in aggregate, better at talent evaluation than journalist drafnicks.
  • In two of the six years considered (2016, 2018), the draft order correlated well with player win shares. The other four were much more chaotic (2017, 2019, 2020, 2021).
  • There is not a lot of separation between mock drafter’s projections, especially at the top of the draft. When a writer did go out on a limb and stray from the pack, more often than not their projection did not pan out.

Quantitative results:

Correlation between big board rank and actual class win share rank

Tier 1: NBA Executives (27% correlation)

Tier 2: Tjarks (25%), Vecenie (23%)

Tier 3: West (19%), Hollinger (18%), KOC (15%) 

Average difference between big board ranking and class win share rank:

When all picks are considered:

Tier 1: NBA Executives (avg distance of 5.4 slots)

Tier 2: West (6.3), Vecenie (6.4), Hollinger (6.5), Tjarks (6.5), KOC (6.7)

When only top 10 picks are considered:

Tier 1: NBA Executives (5.5)

Tier 2: West (5.8), Hollinger (5.9)

Tier 3: Vecenie (6.1), Tjarks (6.1), KOC (6.5)

Of the players drafted in the top 10, what percentage of these players were ranked within 3 spots of the actual class win share ranking:

Tier 1: NBA Executives (48%), Hollinger (45%),

Tier 2: West (38%), Tjarks (37%), Vecenie (33%), KOC (32%)

Commentary:

  • Overall, we don’t see a ton of difference between the projection accuracies: especially in the top 5, writers seem very reluctant to go out on a limb. Anecdotally, I’ve always surmised that the draft prognosticators operate in an echo chamber and the data bears that out: mock drafters rarely have someone ranked more than a couple of slots away from their cohort and when they do, it generally doesn’t play out according to those projections. 
  • It seems that the people paid to be in front offices are indeed better at talent evaluation than desk jockeys and amateur scouts. I had always assumed that the opportunities to lead an NBA Front Office are driven more by nepotism and name recognition than by genuine talent. But whether it’s their own discerning eye or the boundless resources they have access to, the data indicate that GM’s produce more accurate talent evaluations. 
  • Our perception of how good NBA execs are at their jobs is strongly subjective and based largely on the immediate media response after a pick is made. They lose their jobs when public sentiment or the whims of the owner dictate that it’s time for a change, and there are rarely objective looks at historical picks and performance. It’s nice to see that despite this lack of objective accountability, the people with our dream jobs, by and large, do a better job than us lesser-pedigreed fans not steeped in professional basketball circles. 
  • When forced to split hairs among writers, the actual NBA Draft order and John Hollinger’s big board turn out to be the best predictor of actual win shares. Perhaps the fact that current and former NBA executives are the best at predicting player outcomes shouldn’t be surprising, but it’s a good sanity check. After the current/former NBA executives, the middle class of mock drafters is comprised of THK’s Bryant West, The Athletic’s Sam Vecenie, and The Ringer’s Jonathan Tjarks. Finally, we see The Ringer’s KOC consistently ranking last. I find it ironic that in its first three years of existence, the Ringer published big boards from three writers: KOC, Tjarks, and Danny Chao. Starting in 2020, they selected KOC as their sole draft prognosticator and have been publishing only his rankings ever since. I’ve always chafed at Bill Simmons and his lackey’s tendency to make fun of the Kings relentlessly, so I won’t deny that I find some spiteful joy in the data’s conclusion that their ordained draft guru is ranked lowest.

Writer profiles / cherrypicked stats:

  • John Hollinger: With only two big boards published (in 2020 and 2021), Hollinger’s projections were the closest to the actual WS rankings (after the draft order itself). He distinguished  himself in the top 10, and I’ll give him extra credit for not having a big board published in 2018, which was by far the most “predictable” draft and inflated the stats of many of his comrades.
  • Sam Vecenie: Vecenie had the most robust catalog to analyze, having published a big board for all six of the seasons I considered. His overall win share correlation of 23% ranked 3 of 5, with 2016 and 2018 as his standout seasons (45% and 62%, respectively). His results in projecting the top 5 and top 10 prospects were middling, but he and Hollinger were the most accurate predictors (aside from the actual draft order) in the 11 through 20 range.
  • Jonathan Tjarks: Tjarks published big boards in 2017, 2018, and 2019. While he had a lot of exact matches between his big board and WS rankings (especially in the top 5), he benefited greatly from only publishing three big boards in the time span, since 2018 was the easiest draft to predict and dragged everyone’s scores up.
  • Bryant West: Bryant was solid in the top 5 and was carried by his results in the 6-10 range. He had the second-smallest aggregate difference when considering only picks in the top 10, and was tied for the most exact hits in the top 10: 10% of his projections landed at the exact player win share ranking. He also had the most players from picks 10-20 land within 3 picks of their win share ranking.
  • KOC: There’s a lot not to like on KOC’s resume. His big boards’ correlation to win shares were the lowest of the bunch at just 15%. And with rankings published every year since 2017, his sample size is sufficient for me to discount his evaluations compared to his peers (I’ll come forward now and say that I’m not the biggest fan of his podcast persona as Bill Simmons’ number one Celtics footman, either).

So what? Applying these findings to this year’s draft, I decided to aggregate the mock drafters’ current rankings, weighting each based on the above results (I only considered the 3 writers who currently have big boards or mock drafts published: Vecenie, KOC, and Hollinger):

  • 40% Hollinger
  • 40% Vecenie
  • 20% KOC

Final projected win share ranking, lottery only:

  1. Jabari Smith (weighted average win share ranking = 0.9)
  2. Chet Holmgren (1.8)
  3. Paolo Banchero (2.7)
  4. Jaden Ivey (3.6)
  5. Shaedon Sharpe (4.5)
  6. Keegan Murray (5.4)
  7. Jeremy Sochan (6.3)
  8. Benedict Mathurin (7.2)
  9. AJ Griffin (8.1)
  10. Dyson Daniels (9.0)
  11. Jalen Duren (9.9)
  12. Ousman Dieng (10.8)
  13. Mark Williams (11.7)
  14. Johnny Davis (12.6)

 

These results may mean something or nothing – as I hope I’ve hammered home, there really isn’t a lot of difference between these writers’ rankings at the top of the draft. But my hope is that they can generate thoughtful discussion among Kings fans. After all, connecting with others over our shared (misplaced?) passion for our ragtag franchise is what being a member of the Kings community is all about. Rest assured that whatever we think, the GM’s are smarter than us, in aggregate. So let’s see how the draft shakes out, make our critiques, and enjoy our lack of accountability.

Patreon Membership
* indicates required


To prevent spam, our system flags comments that include too many hyperlinks. If you would like to share a comment with multiple links, make sure you email [email protected] for it to be approved.

26 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
SPTSJUNKIE
Patreon Supporter
Comments
Vote Up
Original Member
Author
Patreon Supporter
Comments
Vote Up
Original Member
Author
June 20, 2022 4:25 pm

Fantastic work.

And fantastic work Bryant. One more push to take down Hollinger 🙂

Kingsguru21
Comments
Vote Up
Original Member
Patreon Supporter
Comments
Vote Up
Original Member
Patreon Supporter
June 20, 2022 7:59 pm
Reply to  SPTSJUNKIE

I don’t think Hollinger is the guy that Bryant should be aiming to take down…..

(It’s Givony.)

Last edited 2 years ago by Kingsguru21
Bryant
Member
Original Member
Author
Comments
Vote Up
Original Member
Author
Comments
Vote Up
June 20, 2022 8:05 pm
Reply to  Kingsguru21

👀

Kingsguru21
Comments
Vote Up
Original Member
Patreon Supporter
Comments
Vote Up
Original Member
Patreon Supporter
June 20, 2022 8:09 pm
Reply to  Bryant

Never been a big fan of Givony. He’s not useless in all ways, given he’s Israeli and understands Europe, but he’s mostly…eh, whatever at evaluating US prospects. And he lives in the US, so it shouldn’t be that hard.

I don’t know what it is. I think he’s largely overrated in talent evaluation. Yes, I thought this before ESPN. Do I care? Not really? But it’s not insignificant either.

Last edited 2 years ago by Kingsguru21
Kingsguru21
Comments
Vote Up
Original Member
Patreon Supporter
Comments
Vote Up
Original Member
Patreon Supporter
June 20, 2022 8:11 pm
Reply to  Kingsguru21

I think it’s revealing that Schmitz got hired into a NBA FO before Givony.

Mateen Cleaves
June 21, 2022 10:26 am
Reply to  Kingsguru21

Now Givony is suggesting Ivey wouldn’t agree to play for the Kings lol

“Will the Kings, who must consider the likelihood of their pick signing with the struggling franchise, strike a draft-night deal?”

andy_sims
Vote Up
Comments
Original Member
Patreon Supporter
Vote Up
Comments
Original Member
Patreon Supporter
June 21, 2022 2:46 pm
Reply to  Mateen Cleaves

What a load of shit. I honestly can’t remember the last time an NBA draftee refused to report.

This is what’s wrong with the NBA being a twelve-month league, there always has to be some bullshit drama pushed to wrap a story around.

No drama here. We’ll either trade the pick to get better sooner, or we’ll take whoever the fuck we want, because this is a business.

King4life
Original Member
Vote Up
Comments
Original Member
Vote Up
Comments
June 21, 2022 3:50 pm
Reply to  Kingsguru21

Totally agree. Schmitz always stood out in terms of his media presence so I wasn’t surprised that he got hired before Givony. Schmitz has a lot of natural charisma and I always found his videos with draft prospects much more engaging.

murraytant
Comments
Vote Up
Comments
Vote Up
June 20, 2022 4:39 pm

good job
Interesting subject and great analysis. The front offices are privy to more direct information and offer a collective opinion for each team/office. Strength in numbers.

In the lead up to the actual draft it is helpful to get a roadmap- even if the map is deficient. I have liked Vacenie and Hollinger in the past..

Mike Shmitz has left the draft analysis game and joined a front office- Memphis?

eddie41
Comments
Vote Up
Patreon Supporter
Comments
Vote Up
Patreon Supporter
June 20, 2022 5:04 pm
Reply to  murraytant

Portland – assistant GM

Greg
Admin
Vote Up
Comments
Original Member
Author
Vote Up
Comments
Original Member
Author
June 20, 2022 5:15 pm
Reply to  murraytant

I had a similar thought. It’s not surprising that front offices are better strictly because they have more information available. Still takes skill to sort through it all, but the info is there.

I also thought it was kinda funny that nobody, front offices or draft predictors, are super accurate.

Bbmuteman
Original Member
Comments
Patreon Supporter
Vote Up
Original Member
Comments
Patreon Supporter
Vote Up
June 20, 2022 5:05 pm

Koc always falls in love with one guy that puts him out from the mainstream, which probably pushes his numbers down. One year, it was Killian Hayes as a future harden.

Very interesting data you’ve shifted through for us. Thanks!

Dub_TC
Original Member
Comments
Vote Up
Original Member
Comments
Vote Up
June 20, 2022 6:37 pm
Reply to  Bbmuteman

I was gonna say … I looked into his draft boards the last 2 years (just to see where people had guys ranked so I have a more open mind for this draft) and he had Hayes #1 on his board in 2020. Think Hali was 10th?

andy_sims
Vote Up
Comments
Original Member
Patreon Supporter
Vote Up
Comments
Original Member
Patreon Supporter
June 21, 2022 2:49 pm
Reply to  Dub_TC

I had doubts about Hayes from jump, but also about Suggs. Good college player, but never jumped off the screen in the way the press covered him so breathlessly.

Adamsite
Patreon Supporter
Comments
Vote Up
Original Member
Nostradumbass 14
Patreon Supporter
Comments
Vote Up
Original Member
Nostradumbass 14
June 20, 2022 5:56 pm

We have some amazing people on this site who come up with excellent content, and you are one of the Matt. Great work! I love this community.

BasketballHella
Comments
Vote Up
Comments
Vote Up
June 20, 2022 6:18 pm

Great job awesome read. Why I love this site and sincerely think if things don’t change they should branch out on all things NBA not just Kangz.

Also love Vecenie and his pod, great analysis and doesn’t just shit on teams to get some clicks or a sound bite.

Last edited 2 years ago by BasketballHella
jwalker1395
Comments
Vote Up
Comments
Vote Up
June 20, 2022 6:57 pm

Really interesting. It would seem that KOC likely ranks more for narratives rather than purely as talent evaluation. It doesn’t mean anything, so why not say that Killian is #1 or that Poku is the second coming of Wilt Chamberlain or whatever?

As for the draft ratings, I suspect Sharpe is gonna be the big whiff, Mathurin will be underrated as a fringe all-star wing, and Daniels is the two-way role player that every team wants on their title contender ala Shaun Livingston. Otherwise, everyone’s position makes sense to me.

UpgradedToQuestionable
Comments
Vote Up
Patreon Supporter
Comments
Vote Up
Patreon Supporter
June 20, 2022 8:36 pm

And how do the draft prospects agents influence who is drafted where?

Those selected do not grow in the dark – some bloom because of how they are cultivated.

I read this with interest:
https://www.welcometoloudcity.com/platform/amp/2022/6/18/23173363/the-figures-behind-the-curtain

It made me think, Bill Duffy also represents Luka Donçīc and there was no way in basketball hell that he was going to let King Vivek and GM Divac chose his client. He represents Chet Holmgren in this draft. Maybe GM McNair makes a difference, but I doubt it. No way Chet’s a King (my hot take)

It makes Banchero and Murray more likely than anyone else of those top group.

Mintz has a vested interest in ensuring that Ivey ends up in a good situation. At present, Jaden Ivey is mocked to end up with the Kings. The Kings are a dysfunctional franchise with two other lead ball-handlers in De’Aaron Fox and Davion Mitchell. It is the sort of place where growth is stunted.

Last edited 2 years ago by UpgradedToQuestionable
andy_sims
Vote Up
Comments
Original Member
Patreon Supporter
Vote Up
Comments
Original Member
Patreon Supporter
June 21, 2022 2:57 pm

That’s an interesting story, thanks for the link.

Shit, me though, if I’m a lottery pick, I wouldn’t even hire an agent. There isn’t a ton of wiggle room in rookie deals, so I’ll just keep 100% of my money instead of 85% or 90%.

Get you a business manager for endorsements and financial stuff, and look into an agent when your rookie deal starts to run out. And I if have a reasonable expectation of getting a max RFA deal or extension, I’ll get in touch with an agency in year seven.

Lotta crooks in that line of work.

Tunel_21
Comments
Vote Up
Patreon Supporter
Comments
Vote Up
Patreon Supporter
June 21, 2022 7:51 am

I love KOC, and the Mismatch is my favorite NBA podcast, but he is all over the place with his draft evaluations. Nevertheless, he completely owns his mistakes and makes fun of himself for the misses like Bamba and Hayes. One thing he got incredibly right was that he wasn’t high on Bagley at all.

Either way, we all know the draft is a crapshoot. The only real evaluator of talent is West. All hail.

Bryant
Member
Original Member
Author
Comments
Vote Up
Original Member
Author
Comments
Vote Up
June 21, 2022 9:28 am

Truly appreciate that you included me in this list, especially since everyone else here does this for a living. Really cool to see my success rate compared to some other evaluators I truly respect and learn from. I’m glad my 6-10 rankings have carried me all these years!

Awesome content, thanks for putting this out there. Now I’m off to yell at Hollinger that I’m coming for him.

andy_sims
Vote Up
Comments
Original Member
Patreon Supporter
Vote Up
Comments
Original Member
Patreon Supporter
June 21, 2022 2:58 pm
Reply to  Bryant

Tell him I miss our quiet dinners at home.

andy_sims
Vote Up
Comments
Original Member
Patreon Supporter
Vote Up
Comments
Original Member
Patreon Supporter
June 21, 2022 2:41 pm

Really, really, really amazing work, and even the caveats are great!

Kudos to you and SPTSJUNKIE, both for the time and know-how to even put analyses of such incredible depth together for those of us who refer to Google whenever using Excel.

You should both be very proud, and we are fortunate to have you as contributors.

Ellis5
Comments
Vote Up
Comments
Vote Up
June 21, 2022 3:03 pm

Bob Odenkirk GIF by The Office - Find & Share on GIPHY

Badge Legend

Patreon Supporter Patreon Supporter   Registered On Day 1 Registered On Day 1   Published Post Published Post  Published Post Nostradumbass
Comment Up Votes 200 Up Votes   Comment Up Votes 500 Up Votes    1,000 Up Votes    3,000+ Up Votes

Comments 50 Comments   Comments 100 Comments    250 Comments    500 Comments    1000+ Comments