Tuesday, October 28, 2008

Analyzing Our Opinions: State & the DtB Plans

We at Down the Backstretch have been eager to offer our opinion on how MSHSL State Meet qualifying could be improved. We see inherent unfairness in the current two-teams-from-each-section qualifying system. Last year, we offered a plan to reward sections that performed better at the previous year’s State Meet that we thought would improve the quality and fairness of selection to the meet.

Now that the 2008 State Meet field is set, we can actually observe how the DtB plan would have changed the composition of the State Meet in 2008 in the often-problematic boys Class AA division.

One way to frame the discussion about the MSHSL State Meet qualifying system is to think of it in terms of the “other eight” teams. Nothing that DtB has proposed and very few of the other solutions we’ve heard around the “virtual water-cooler” have advocated ditching the sectional qualifying framework. Section qualifying is an MSHSL staple – across its sports’ landscape – and a big part of Minnesota’s high school sport's tradition.

That being the case, if we consider sectional qualifying sacred and assume that section champs will and continue to earn the right to compete at State, then eight teams – the “first eight” – are essentially not part of the discussion. Win your section – whether you’re the #1-ranked team in the state or a team that’s never gotten a coaches’ poll vote – and you’re in the meet, no questions asked. You’re the section champ, after all!

That means, in 2008, #6 Owatonna, #10 Chaska, Burnsville, #4 White Bear Lake, Coon Rapids, #1 Wayzata, #5 Moorhead, and #7 Forest Lake, the section champs of 2008, are not part of the analysis and discussions below. What we’re really discussing, when we talk about changes to the system – or the need to keep things as they stand – is how to choose the “other eight” teams at State. That is, the State participants that aren’t section champs.

Currently, of course, the “second eight” are selected based on whoever finishes second in their section. Simple enough, except that not all sections are equally strong and history has shown that more than a few deserving teams – teams ranked in the 12-deep Coaches Association rankings after sections – have not qualified for State.

To get specific, the “other eight” teams in this year’s meet are:

#11 Lakeville South
Willmar
St. Thomas Academy
#9 Roseville
Osseo
#2 Eden Prairie
Duluth East
#8 Alexandria

Four of those teams, you’ll notice, are not ranked in the top-12 of the state rankings.

The following state-ranked teams are not in the State Meet because they finished worse than runner-up in their section:

#3 Edina
#12 Minneapolis South

Last year DtB proposed a system, whereby a section’s teams’ finish at the State Meet would determine whether more teams than just the section’s champion would qualify to State the following year. The post where we described the system -- and some additional tweaks we thought might be reasonable -- is HERE.

In a nutshell, the system works such that, if none of a section’s teams finished in the top eight in a previous year, that section would only qualify its champion to State in the current year. If a section had one team in the top eight, it would qualify an additional team the next year. If a section had two teams in the top eight, it would qualify two additional teams the next year.

Now that we know the make-up of the 2008 State Meet field, we can actually observe how the proposed system would have worked this year in boys’ Class AA. Granted, a single season isn’t the end-all for analyzing competing systems, but it can be instructive all the same.

If such the system had been in place this year based on last year’s State results, the “other eight” teams at State in Class AA on Saturday would be:

Willmar … 2AA runner-up
St. Thomas Academy .. 3AA runner-up
#9 Roseville … 4AA runner-up
#2 Eden Prairie … 6AA runner-up
#8 Alexandria … 8AA runner-up
Rosemount … 3AA 3rd place
#3 Edina … 6AA 3rd place
Bemidji … 8AA 3rd place

The following ranked teams would be out the meet:

#11 Lakeville South
#12 Minneapolis South

The following teams that are in the actual 2008 State Meet but not in State via the DtB plan are:

#11 Lakeville South
Osseo
Duluth East

Another way to put it is that you’re trading those three teams listed above for:

Rosemount
#3 Edina
Bemidji


As you can see, using the previous year’s top eight to choose the second tier of State teams this year would have done a slightly better job of strengthening the field at State than the current system. While the same number of ranked teams still on the outside looking in at State – two – is the same in both plans, the top ten teams in the rankings all make the meet with the DtB plan because #3 Edina gets in the meet based on the strength of Section 6AA at State last year. The system didn’t actually add to the number of ranked teams, because another ranked team, albeit a lower-ranked one, #11 Lakeville South, got bumped by the DtB system.

DtB also suggested last year that using a mix of qualifiers earned the previous year and teams selected by an objective committee might be a more precise way to get the most deserving teams on the starting line at State. We suggested that six teams could be earned at the previous year’s State Meet and another two could be selected by the committee of coaches and officials. The two “selected teams” could work as a fail-safe for outstanding teams that rise from sections that haven’t earned enough extra qualifying slots to advance them to State.

If such a system was in force for 2008, the field would, we think, look like this:

“Earned Teams”
Willmar … 2AA runner-up
St. Thomas Academy … 3AA runner-up
#9 Roseville … 4AA runner-up
#2 Eden Prairie … 6AA runner-up
#8 Alexandria … 8AA runner-up
Rosemount … 3AA 3rd place

“Selected Teams”
#3 Edina
#11 Lakeville South

We’re assuming, of course, an objective committee would select #3 Edina and #11 Lakeville South, the top-ranked but-as-yet-not-qualified teams in Class AA.

In that case, only a single team ranked in the top-12 is left at home in 2008:

#12 Minneapolis South.

The following teams who are in the actual 2008 State Meet but not in via this second DtB method would be:

Osseo
Duluth East

Or, put another way, you’re trading those two schools – neither of which are ranked – for:

#3 Edina
Rosemount

Sorting it All Out … I think the DtB systems fared pretty well in 2008. Ideally, we would like to have seen one of our plans put all 12 ranked team in the meet. That was impossibility from the start for our “eight earned teams” plan, because in the first year of its existence no more than three schools could qualify from a single section. For a Minneapolis South – 4th in their section -- to make it to State solely via the eight earned teams plan, Section 6AA would need to have had three top-8 finishers at the previous year’s State meet, and you don’t get a third team, of course, until you've earned the third team.

The six-earned, two-selected plan didn’t pick up South either. After the earned slots were awarded, there were still three ranked teams awaiting two selected slots, and #12 Minneapolis South was behind #11 Lakeville South in the pecking order. If you’re going to leave a ranked team behind, it is best that it be the lowest-ranked one, but ideally the top-12 would all be in the meet.

That neither plan brought Minneapolis South into the meet prompts one to look at the strength of teams that got to State via “earned” slots. If only eleven ranked teams made the meet, then, obviously, five un-ranked teams advanced. Only half of the teams qualifying in the “other eight” with “earned” slots were ranked – 4 of 8 and 3 of 6 in the two plans. We anticipated the earned slots would bring more ranked teams than un-ranked ones into the meet. Perhaps it was an odd year that way.

Or, maybe sectional strength moves around between sections in a way that makes using one year’s State Meet results to earn the next year’s qualifiers a less than ideal way to select the State Meet field. It improved the quality of the field relative to the current system, it should be stressed, but did so less accurately than we expected or hoped. We liked the “all-earned” system because of its objectivity, but acknowledge that the system that mixed earned and selected teams chose a stronger State Meet field – assuming of course the selectors would identify and choose the most deserving teams.

Based on 2008 – which is only a single year’s worth of data, we feel compelled to note -- employing three “selected” teams and only 5 “earned” teams would have struck a better balance and gathered a stronger field. Such a plan would have put #12 Minneapolis South in the meet at the expense of un-ranked Rosemount, a qualifier via our other plans.

Getting the #3 team in the State meet, though, was the critical problem of 2008, we think. In a 16-team State Meet, it’s an egregious shortcoming for a qualifying system not to advance the #3-ranked team. Both of our systems solved that problem.

In sum, we still think the MSHSL should modify its current manner of advancing teams to State. We acknowledge that neither of the DtB plans would have worked “perfectly” in 2008, but both improved the quality of the field – getting #3 on the starting line.

If the strength of the State Meet field is an important value – and we feel strongly that it should be – a system that combines earned and selected teams to fill the “other eight” slots of the State Meet field is still the course to follow.

7 comments:

Chad said...

Charlie,

I was thinking more about this the other day. You said, "If the strength of the State Meet field is an important value..." Has the MSHSL ever said that that's their aim? Maybe they're more conerned with spreading the wealth of teams qualifying for state throughout the state.

Also, have you proposed your systems to MSHSL?

Finally, how much faith do we have in the ranking system? Thinking of college football, the experts had the Badgers ranked #9 then they proceeded to lose 4 straight. Are we certain that a #12 team is better than all the unranked teams?

Charlie said...

Chad,

By their actions in XC, spreading the wealth evenly seems to be the MSHSL goal. In track, though, they do make provision for quality -- letting additional individuals qualify, if they meet a time standard ... so?

Good question about the ranking system. Right now, because it's not used for anything official, I don't think the rankings are exactly put together by a team of voters representing all the sections and govened by any special rules. I think they're pretty good for what they are, though ... and worth using for the arguments I made in the post.

(As an aside, I will say that I'm surprised Stillwater fell out of the rankings after a 2-point loss to the team now ranked #9, Roseville.)

If you're going to use the polls or some other sort of power rating to choose teams, though, you'd really want to make sure you were doing things right, i.e., wieghing teams objectively. I know D1 has all sorts of rules and regs that govern how they choose their at-large selections.

As for me bringing plans to the MSHSL ... I don't think that's my role, nor do I think the MSHSL would think I have any standing to do so. I figure -- as a guy with a publication -- I should get some ideas out there and have them batted around. The coaches -- who do have standing through their Coaches' Association, I beleive -- would need to bring any new plans forward.


-- Charlie

greghoffman said...

Charlie,

Speaking purely from a mathematical/statistical standpoint, you seem to be basing how successful your system is solely on a single year's worth of data. I'd like to see this analysis done for, say, the last 5 years, and see what the results are.

Now, I know it's not going to _quite_ be a complete analysis, since you won't have the advantage of the system actually being in place, and the cumulative effects of the five years, but I'd like to see the same single-year analysis - I'd be happy with just the purely subjective version - done for the 5 previous years and see how well your system worked. I'd give it a try myself, but I don't have the historical data.

I'm not knocking your system - seems, at first glance, like it might be pretty good method - but it won't stand up to scrutiny being based solely on one year's results.

One other possible problem - how would you deal with re-sectioning every few years?

Charlie said...

Greg,

Thanks for the comments ...

I certainly didn't mean to oversell the single year's worth of "data" I was working with. I tried to say as much in the piece itself. I guess I saw the exercise as an instructive way of looking how the systems would have worked this one time.

As for a study with more breadth, State Meet results since 1991 are archived on RaceberryJam at:

http://www.raceberryjam.com/archives.html

I'd love to see someone mine that for information about how to make the State Meet better. (I'd happily publish the results!)

Realignment is indeed a tricky matter -- and it's something I must admit I didn't realize was done as often as it is. Should earned teams move with re-aligned schools or stay with the section they were earned in? Hmmm ...

You might think that with re-alignment every few years the MSHSL could have avoided the problems we've seen in boys' Class AA -- #3 & #12 out in 2008, #7, #8, & #9 out last year, 6 of the top-12 teams, including #4, out in 2006 -- but either realignment isn't aggressive enough or isn't the proper tool.


-- Charlie

greghoffman said...

I looked through the archive, but soon realized that I probably couldn't get too far without knowing the historical data on section alignment as well as state poll rankings for the previous years.

Wait - just found the hutchcrosscountry.com page - looks like I should be able to get some data from there. I'll work on it over the weekend...

greghoffman said...

Okay - initial results show that the 6AA section probably does have something to complain about. I looked up polls vs. state meet participants, and from 2001-2008, there were 27 poll-ranked teams that did not make the state meet. 24 of these were from 6AA.

Taking Eastview, Rosemount, Burnsville, and Holy Angels (all now in 3AA) out of the historical 6AA mix reduces that number to 17, but it still seems to be coming up as a very deep section, at least in the past 8 years.

More to come...

greghoffman said...

Here are the results of applying the purely subjective - only based on the previous year's top 8 - method for the past 8 years. For this first round, the results were not cumulative, i.e. there's only a maximum of three entries from each section each year. I'm working on some method to approximate cumulative effects.

For each year listed, there are three groups:

Added: ranked teams that weren't in the state meet that would have been under this new system

Still Out: Ranked teams that weren't in the state meet, and still wouldn't be under the new system

Dropped: Ranked teams that were in the meet under the old system, but would be dropped under the DTB system

Here are the results - the dash-number after the school name is the section number, and the number in parentheses after the dropped schools is their actual place in the state meet:

2001:
Added:
*Eastview-6 Poll-4

Still Out:
Eden Prairie-6 Poll-5
Holy Angels-6 Poll-7
Burnsville-6 Poll-12

Dropped:

2002:
Added:
*Wayzata-6 Poll-6

Still Out:

Dropped:
Minneapolis Edison-5 Poll-12 (8)

2003:
Added:
*Edina-6 Poll-6

Still Out:
Wayzata-6 Poll-7

Dropped:
Brainerd-8 Poll-11 (10)

2004:
Added:
*Edina 6 Poll-6
*Mankato West-2 Poll-10

Still Out:
Rosemount-6 Poll-9
Wayzata-6 Poll-11

2005:
Added:
*Rosemount-6 Poll-6

Still Out:
Eastview-6 Poll-8
Wayzata-6 Poll-9
Mounds View-4 Poll-11

2006:
Added:
*Wayzata-6 Poll-4
*Roseville-4 Poll-10

Still Out:
Hopkins-6 Poll-5
Edina-6 Poll-8
Burnsville-6 Poll-11

2007:
Added:
*Edina-6 Poll-7

Still Out:
Robbinsdale Armstrong-6 Poll-8

Dropped:
Eastview-6 Poll-11 (5)

2008:
Added:
*Edina-6 Poll-3

Still Out:
Minneapolis South-6 Poll-12

Dropped:
Lakeville South-1 Poll-11

As you can see, the fact that the vast majority of ranked, non-competing teams were 6AA, combined with the one-year-only application of the system means that there were still several ranked section 6 teams that were left out most years.

The most glaring dropped team was the Eastview 2007 team, which ended up placing 5th. But, this was the year the sections were re-aligned, and if the Rosemount top-8 finish from the year before would have moved with them, Eastview would have remained in the state meet.

More (probably) later. I'm still trying to figure out how to accumulate the effects - probably have to use year-end rankings instead of race placement for non-competing teams who "might have placed in the top eight if they were there". I'm thinking that by 2008 we might just end up with an all 6AA state meet :).