Talk:Technology Committee/Project requests/WikiRate - rating Wikimedia

SMART

I think this is now being used so much as a jargon word that the meaning was lost a while back. Metrics are not SMART, could this please be corrected as this just promulgates further misunderstanding? --Fæ (talk) 20:11, 12 April 2014 (BST)

'SMART metrics' -> 'SMART targets'. Thanks for pointing that out. --MichaelMaggs (talk) 20:57, 12 April 2014 (BST)

Thoughts on task breakdown

Thanks Michael this looks like a great start. I'm just having a bit of a brain dump here. There are some challenges on which input, or/and experimentation will be required. These include:

• Defining the classes/outcomes
• Defining the relevant input feature set (and their detection) which will be used as predictors
• Segmentation - both with regard to the level at which outcomes are assigned (e.g. edit, article) and with respect to

Some of this matters more for machine learning techniques, while we might also use some simple measures (e.g. looking at T1-T2 difference on article specific Wiki ToDo list) which depend on features as proxies for outcomes. On that Re: outcomes, one simple thing for machine learning particularly as a first step might just be: 1) was the edit positive, 2) negative, 3) neutral with respect to improving the article. We could then provide a breakdown of such edits. . We should also be cautious that our feature selection doesn't exclude some widely missed but important features (e.g. alt-text). The rubric below might be a good way to 'present back' and assess improvements (with some aggregation method probably for overall improvement).

Assessment area	Scoring methods	Score
Comprehensiveness	Score based on how fully the article covers significant aspects of the topic.	1-10
Sourcing	Score based on adequacy of inline citations and quality of sources relative to what is available.	0-6
Neutrality	Score based on adherence to the Neutral Point of View policy. Scores decline rapidly with any problems with neutrality.	0-3
Readability	Score based on how readable and well-written the article is.	0-3
Formatting	Score based on quality of the article's layout and basic adherence to the Wikipedia Manual of Style	0-2
Illustrations	Score based on how adequately the article is illustrated, within the constraints of acceptable copyright status.	0-2
Total		1-26

from https://en.wikipedia.org/wiki/Wikipedia:WikiProject_United_States_Public_Policy/Assessment#Quantitative_Article_Quality_Assessment_Metric

It's also the case that the salient features may vary with a combination of temporal and editor-interaction factors. Early stage articles benefit greatly from addition of different features to later stage ones (e.g. amongst others http://scholar.google.co.uk/scholar?cluster=2386972856451571180&hl=en&as_sdt=0,5).

There's also an interesting point re: namespace contributions on talk and article pages, presumably in the first instance we're looking only at article contributions.

It is also worth noting that whatever we do, we shoud where possible consider implications for non-English Wikipedias, in particular the ways in which references are used are (I believe) different in different Wikipedias. This may well also be true of different projects.

Finally, we should also note if we did anything sucessfully, a number of benefits might also be gained including automation of quality (or semi-automation) within projects, etc., and potential for new editor engagement experiments e.g. sending editors to articles which we think might be easily improved (a more sophisticated 'wiki to do' tool). Sjgknight (talk) 10:34, 13 April 2014 (BST)

Just crossed my mind it's also worth noting the potential benefits for e.g. education being able to take the contributions of a particular editor, and look for particular features (e.g. use of 'cite' templates) across the contributions. That would have benefits outside of Wikipedia (on Mediawiki) where analytics on writing style and content could be conducted. Sjgknight (talk) 10:45, 13 April 2014 (BST)

: Regarding assessment areas, take a look at Table 3 of Stvilia, Besiki, et al. "Information quality work organization in Wikipedia." Journal of the American society for information science and technology 59.6 (2008): 983-1001.. It gives a good list of criteria. For example, "Accessibility" has "caused by (1) Language barrier (2) Poor organization (3) Policy restrictions imposed by copyrights, Wikipedia internal policies, and automation scripts" and suggests actions such as "Reorganize, duplicate, remove, translate, split, join, rearrange"." See the whole list, I think it might be helpful for your chart, Simon. Jodi.a.schneider (talk) 10:48, 13 April 2014 (BST)