Technology Committee/Project requests/WikiRate - rating Wikimedia

This page in in the process of being written. Please do not comment yet. This message will be removed when the text is complete

Objective

There is an increasing desire, and need, for Wikimedia editors, chapters and other groups to be able to measure quantitatively the impact of the work they are doing. There is a growing understanding and agreement that SMART metrics are in many cases the best and most reliable approach, but such metrics are very difficult to come by when we try to measure impact as 'improvements in the quality of Wiki articles or uploaded media files'. Improvements in quantity can easily be measured; improvements in quality are much harder.

To date, most attempts to measure article quality have relied on manual evaluation by experts. Such approaches will always be needed, and will always represent the 'gold standard' of quality evaluation, but they suffer from two very significant drawbacks: expense and lack of scalability. We suggest that there is a need for a tool or series of tools that can measure article and media file quality in an automated way. The output of such an automated tool will always be far inferior to manual evaluation in any particular case, but when applied to a large corpus may nevertheless provide very illuminating data on overall quality. By measuring how comparable outputs vary over time we can get at least some measure of the impact that a chapter or other group is having as it tries to improve overall article and media file quality. Measuring quality via a series of numerical metrics would make this extremely important facet of Wikimedia amenable to SMART analysis for the first time.