The "Long Walk" to Learning Outcomes Standard: Discussion Draft of ABA's Assessment of Learning Outcomes Standard Fundamentally Flawed in External Assessment Metrics

The Discussion Draft of the 300 Standards prepared by the Student Learning Outcomes Subcommittee of the ABA Section of Legal Education and Admissions to the Bar's Standard Review Committee was made available at the Legal Education at the Crossroads Version 3.0: A Conference on Assessment, held September 11-13, 2009 at the University of Denver’s Sturm College of Law.

The 300 series of Accreditation Standards includes substantial changes, most notably the addition of Standard 303, Assessment of Learning Outcomes (reprinted below; the current Standard 303 would be renumbered as Standard 304). The Assessment of Learning Outcomes Standard features adaptations from Accreditation Standards and Guidelines for the Professional Program in Pharmacy Leading to the Doctor of Pharmacy Degree and Commission on Dental Accreditation's Outcomes Assessment.

Do note Interpretation 303-1. "Assessment activities and tools are likely to be different from school to school and law schools are not required by Standard 303 to use any particular tools." With respect to external assessment of learning outcomes tools, bar exam passage rates, placement rates, surveys of attorneys, judges, and alumni, and assessment of student performance by judges, attorneys or law professors from other schools "when properly applied and given proper weight, are among the tools generally regarded to be valid and reliable."

External Assessment Metrics for Measuring Learning Outcomes. While internal assessment tools may need to be law school specific, there is no need for external metrics to be so. The crux of the matter is the "when properly applied and given proper weight" provision. This is a huge gaping hole that allows plenty of room for law school gaming. Bar passage rates may not be subject to manipulation but unaudited placement rates are already gamed. Survey and performance assessment methodologies are sufficiently complex that one has to wonder whether the legal academy will turn to their on-campus counterparts in the education and opinion research fields for advice. Doubtful.

To make external survey metrics work, standardized tools need to be developed by the ABA-AALS cartel, required to be used by law schools, reported annually directly to the ABA, audited by the ABA, and publicly disclosed annually otherwise we might as well just reply on the Assessment Score by Lawyers/Judges found in the US News Law School Rankings.

Excepted from Discussion Draft:

Standard 303. ASSESSMENT OF LEARNING OUTCOMES
(a) A law school shall develop and carry out assessment activities to measure achievement of the identified learning and other outcomes and shall gather data demonstrating that its students have, by the time of graduation, achieved those outcomes. Consistent with sound pedagogy, the assessment activities must employ a variety of valid and reliable measures systematically and sequentially throughout the course of the students’ studies. A law school shall provide feedback to students periodically and throughout their studies as to their progress in achieving learning outcomes with a view towards encouraging proficiency in each student. There shall be broad-based involvement of the faculty of the law school in developing and carrying out assessment activities.

(b) A law school shall periodically and systematically evaluate its curricular structure, content, organizations and outcomes. As part of the review, a law school shall review whether the outcomes it has selected and the assessment tools it has selected are sufficient to ensure that its students are prepared to participate effectively, ethically and responsibly in the legal profession. The law school shall use the analysis of outcome measures and results for
systematic improvement of the curriculum and its delivery.

Interpretation 303-1
Assessment activities and tools are likely to be different from school to school and law schools are not required by Standard 303 to use any particular tools. Learning and other outcomes should be assessed using tools both internal to the law school and external to the law school. The following internal tools, when properly applied and given proper weight, are among the tools generally regarded to be valid and reliable to assess student performance: completion of courses with appropriate assessment mechanisms, performance in clinical programs, performance in simulations, preparation of in-depth research papers, preparations of pleading and briefs, performance in internships, peer (student to student) assessment, compliance with an honor code, achievement in co-curricular programming, evaluation of student learning portfolios, student evaluation of the sufficiency of their education and performance in capstone courses or other courses that appropriately assess a variety of skill and knowledge. The following external tools, when properly applied and given proper weight, are among the tools generally regarded to be valid and reliable: bar exam passage rates, placement rates, surveys of attorneys, judges, and alumni, and assessment of student performance by judges, attorneys or law professors from other schools.

See the complete text of the Discussion Draft in Steve Bahls' (Chair of the Student Learning Outcomes Subcommittee of the American Bar Association Section of Legal Education and Admission to the Bar’s Standards Review Committee) conference presentation entitled Shifting to an Outcomes Measure Approach for Accreditation Standards for Law Schools. Link to a video of his presentation here.

The ABA is only "beginning this long walk..." (quoting from the video).

Hat tip to Robert Richard's Legal Informatics Blog post, which includes links to many additional resources related to the Conference. [JH]