Talk:Strategy monitoring plan 2014-15

skilled & capable

"G2a.3 WMUK volunteers are skilled and capable. Volunteer Activity Units [4] in training sessions and editathons" - do we want to distinguish (measure both?) between units of people delivering training and receiving? Sjgknight (talk) 11:57, 6 February 2014 (UTC)

I suppose it depends whether the aim is more on increasing the skill of volunteers (in which case I think we should capture how much training volunteers receive) or assessing the skill (in which case we could focus on the feedback received by e.g. volunteer trainers after they deliver an event). It does need distinguishing. Daria Cybulska (WMUK) (talk) 10:55, 7 February 2014 (UTC)

We of course care about both, so I guess one answer is jut to measure both. No. of sessions run by trained volunteers is captured elsewhere Sjgknight (talk) 11:06, 7 February 2014 (UTC)

'access to Wikimedia projects is increasingly available'

Enabling new readers to access the projects is a relatively new type of activity for Wikimedia UK. Perhaps we could set a target as running several pilot projects to assess how this area could be approached in the future. Daria Cybulska (WMUK) (talk) 10:55, 7 February 2014 (UTC)

Yes, broadly that's what the output measure indicates (as it's about estimating our increased reach). The projects we see relating to this outcome are things like: accessibility increase; translation work; digital literacy work (which enables people to engage with content); Kiwix deployment and similar projects. QRPedia scans help with bits of that (and of course, are in part about directing people to content in any easy access fashion in places they would ordinarily have not been directed to Wikimedia sites), but clearly we will need to think about other measures as our work continues. Sjgknight (talk) 11:10, 7 February 2014 (UTC)

'supporting Technical Communities' and 'increased awareness of the benefits of Open Content'

Some of the external communities we work with aren't strictly technical (e.g. Open Rights Group) and it's still an important area of our open knowledge work. I suppose a clarification is needed where in G4.1 we are focusing on technology development, and there is another area where we link with other groups to further open knowledge aims. Daria Cybulska (WMUK) (talk) 10:55, 7 February 2014 (UTC)

I have copied this to Talk:Strategic goals. Suggest it is best discussed there. --MichaelMaggs (talk) 08:53, 15 February 2014 (UTC)

Improving on positive edit size

(Copying relevant point from above the TOC to under this heading) Just spotted "G1.2 The quality of educational Open Content continues to improve" should include something re: article quality incrementing, think this must've been lost in a rewording. Sjgknight (talk) 11:51, 6 February 2014 (UTC)

Good/Featured article or Did You Know are a possible although not ideal approaches. Daria Cybulska (WMUK) (talk) 10:55, 7 February 2014 (UTC)

I note that you are looking for something better than positive edit size to measure the contribution that our activities lead to. Has anyone come across WikiTrust? I think the algorithms could be adapted to our requirements. Yaris678 (talk) 23:04, 15 February 2014 (UTC)

Thanks Yaris678, I'm a big fan of WikiTrust and it's on our list of things to look at for the future, at the moment we hadn't thought much about exactly how we'd use such a tool. I guess we'd need to extract the data that the tool uses to do its colour coding, such that for each article edited we had an indicator of the stability of wikitext contributed through wmuk activities v. the rest? It's something we should put some more thought into (and that'd need technical exploration) so any ideas there would be much appreciated too! cheers Sjgknight (talk) 09:25, 16 February 2014 (UTC)

Agreed. I would love to see us working on this. --MichaelMaggs (talk) 09:42, 16 February 2014 (UTC)

By the way, I don't know how stable WikiTrust will be, I think it's going to move to Toolserver, but will also need a maintainer (I don't know if wmuk can support in any way around this). One of the issues re: looking at quality metrics is that we don't just want the rev summary (which tell us +/-) we actually need to do stuff: for each user, for each rev. The WikiTrust api looks interesting http://wikitrust.soe.ucsc.edu/vandalism-api but we'd need to think about how best to implement it. Perhaps we could run some manual testing to try and prototype something useful in the near future.

By the way, I was also chatting to Magnus yesterday about the WikiToDo tool, because another thing we could be interested in is, for particular articles that've received attention (e.g. in an editathon) what was the pre/post event state of the page on some simple metrics (e.g. does it have an image, etc.). Again we'd need to think about how to store it, when to run it and over what span, etc. but it would be fairly easy to implement if we wanted to. Sjgknight (talk) 10:18, 16 February 2014 (UTC)

At the moment WikiTrust doesn't display the reputation of individual contributors (although this is what it bases the text 'trust' metric on). We may be able to gain access to these, although I agree with them that just using aggregate data on cohort reputation (or alike) would be the most we'd want to do. Another tact, which would be closer to our other ideas re: improving quality, would be to look at the 'trust' of the text inserted by wmuk associated editors/activities. Thinking aloud here! Sjgknight (talk) 11:35, 16 February 2014 (UTC)

Yes. That's what I was thinking of using the WikiTrust algorithms for. We want a measure of the "trust" of text added because of our activities. ie. How much does the text stick around? I would have thought the adaptation would be fairly simple. We would effectively treat all such additions as being by one virtual editor and then calculate the "trust" of that editor. It would need someone capable of modifying the code though. Yaris678 (talk) 18:51, 16 February 2014 (UTC)

As I understand, the problem is that the trust of the text is based on the reputation of the contributor in part, so we can't just amalgamate, and from new editors text will never be (immediately) of high trust. That is, the stability of the text isn't the only metric used to calculate text-trust in WikiTrust. A comparative metric might work (between what's there and what's added), or looking at added text after a set period (or number of edits) had passed.

This is actually a broader issue too, one of the things we'll need to think about is tracking period. How long after attendance at a WMUK event do we collect metrics on contributors and their contributions? Should each new event attended "reset" the clock? Do we collect in perpetuity? Do we start to 'discount' our own impact with regard to editors who came to an event, are still editing, but never come to another event? This is an interesting problem and definitely something we (as a community) should spend some time thinking on. Sjgknight (talk) 22:37, 17 February 2014 (UTC)

Yes. The question of which edits should get counted as due to our activities is a tricky one. This would apply if we were using "positive edit size" too. Obviously edits at our events count, whether by a newbie or otherwise. If a newbie starts at one of our events, do all their edits ever count? First six months? And then you have Wikimedians in residence, and people talked to by Wikimedians in residence...

Actually, I know I suggested it, but I am starting to think that WikiTrust is taking us in the wrong direction... I guess it might make sure we aren't responsible for adding a load of guff that gets removed... but that isn't going to happen much in practice.

I think the aspect of WikiTrust that we can draw inspiration from is the more intelligent analysis of a diff, as opposed to the simple change in the number of characters. The improved measure would look at how many characters were added and how many removed. This should be extractable from a diff by looking at how many characters are highlighted blue on the right and how many highlighted orange on the left. For example, this diff would be -17 and +10, rather than -7. This would count as 27 in our improved measure.

Yaris678 (talk) 18:52, 19 February 2014 (UTC)