Talk:Strategy monitoring plan 2014-15

skilled & capable

"G2a.3 WMUK volunteers are skilled and capable. Volunteer Activity Units [4] in training sessions and editathons" - do we want to distinguish (measure both?) between units of people delivering training and receiving? Sjgknight (talk) 11:57, 6 February 2014 (UTC)

I suppose it depends whether the aim is more on increasing the skill of volunteers (in which case I think we should capture how much training volunteers receive) or assessing the skill (in which case we could focus on the feedback received by e.g. volunteer trainers after they deliver an event). It does need distinguishing. Daria Cybulska (WMUK) (talk) 10:55, 7 February 2014 (UTC)

We of course care about both, so I guess one answer is jut to measure both. No. of sessions run by trained volunteers is captured elsewhere Sjgknight (talk) 11:06, 7 February 2014 (UTC)

'access to Wikimedia projects is increasingly available'

Enabling new readers to access the projects is a relatively new type of activity for Wikimedia UK. Perhaps we could set a target as running several pilot projects to assess how this area could be approached in the future. Daria Cybulska (WMUK) (talk) 10:55, 7 February 2014 (UTC)

Yes, broadly that's what the output measure indicates (as it's about estimating our increased reach). The projects we see relating to this outcome are things like: accessibility increase; translation work; digital literacy work (which enables people to engage with content); Kiwix deployment and similar projects. QRPedia scans help with bits of that (and of course, are in part about directing people to content in any easy access fashion in places they would ordinarily have not been directed to Wikimedia sites), but clearly we will need to think about other measures as our work continues. Sjgknight (talk) 11:10, 7 February 2014 (UTC)

'supporting Technical Communities' and 'increased awareness of the benefits of Open Content'

Some of the external communities we work with aren't strictly technical (e.g. Open Rights Group) and it's still an important area of our open knowledge work. I suppose a clarification is needed where in G4.1 we are focusing on technology development, and there is another area where we link with other groups to further open knowledge aims. Daria Cybulska (WMUK) (talk) 10:55, 7 February 2014 (UTC)

I have copied this to Talk:Strategic goals. Suggest it is best discussed there. --MichaelMaggs (talk) 08:53, 15 February 2014 (UTC)

Improving on positive edit size

(Copying relevant point from above the TOC to under this heading) Just spotted "G1.2 The quality of educational Open Content continues to improve" should include something re: article quality incrementing, think this must've been lost in a rewording. Sjgknight (talk) 11:51, 6 February 2014 (UTC)

Good/Featured article or Did You Know are a possible although not ideal approaches. Daria Cybulska (WMUK) (talk) 10:55, 7 February 2014 (UTC)

I note that you are looking for something better than positive edit size to measure the contribution that our activities lead to. Has anyone come across WikiTrust? I think the algorithms could be adapted to our requirements. Yaris678 (talk) 23:04, 15 February 2014 (UTC)

Thanks Yaris678, I'm a big fan of WikiTrust and it's on our list of things to look at for the future, at the moment we hadn't thought much about exactly how we'd use such a tool. I guess we'd need to extract the data that the tool uses to do its colour coding, such that for each article edited we had an indicator of the stability of wikitext contributed through wmuk activities v. the rest? It's something we should put some more thought into (and that'd need technical exploration) so any ideas there would be much appreciated too! cheers Sjgknight (talk) 09:25, 16 February 2014 (UTC)

Agreed. I would love to see us working on this. --MichaelMaggs (talk) 09:42, 16 February 2014 (UTC)

By the way, I don't know how stable WikiTrust will be, I think it's going to move to Toolserver, but will also need a maintainer (I don't know if wmuk can support in any way around this). One of the issues re: looking at quality metrics is that we don't just want the rev summary (which tell us +/-) we actually need to do stuff: for each user, for each rev. The WikiTrust api looks interesting http://wikitrust.soe.ucsc.edu/vandalism-api but we'd need to think about how best to implement it. Perhaps we could run some manual testing to try and prototype something useful in the near future.

By the way, I was also chatting to Magnus yesterday about the WikiToDo tool, because another thing we could be interested in is, for particular articles that've received attention (e.g. in an editathon) what was the pre/post event state of the page on some simple metrics (e.g. does it have an image, etc.). Again we'd need to think about how to store it, when to run it and over what span, etc. but it would be fairly easy to implement if we wanted to. Sjgknight (talk) 10:18, 16 February 2014 (UTC)

At the moment WikiTrust doesn't display the reputation of individual contributors (although this is what it bases the text 'trust' metric on). We may be able to gain access to these, although I agree with them that just using aggregate data on cohort reputation (or alike) would be the most we'd want to do. Another tact, which would be closer to our other ideas re: improving quality, would be to look at the 'trust' of the text inserted by wmuk associated editors/activities. Thinking aloud here! Sjgknight (talk) 11:35, 16 February 2014 (UTC)

Yes. That's what I was thinking of using the WikiTrust algorithms for. We want a measure of the "trust" of text added because of our activities. ie. How much does the text stick around? I would have thought the adaptation would be fairly simple. We would effectively treat all such additions as being by one virtual editor and then calculate the "trust" of that editor. It would need someone capable of modifying the code though. Yaris678 (talk) 18:51, 16 February 2014 (UTC)

As I understand, the problem is that the trust of the text is based on the reputation of the contributor in part, so we can't just amalgamate, and from new editors text will never be (immediately) of high trust. That is, the stability of the text isn't the only metric used to calculate text-trust in WikiTrust. A comparative metric might work (between what's there and what's added), or looking at added text after a set period (or number of edits) had passed.

This is actually a broader issue too, one of the things we'll need to think about is tracking period. How long after attendance at a WMUK event do we collect metrics on contributors and their contributions? Should each new event attended "reset" the clock? Do we collect in perpetuity? Do we start to 'discount' our own impact with regard to editors who came to an event, are still editing, but never come to another event? This is an interesting problem and definitely something we (as a community) should spend some time thinking on. Sjgknight (talk) 22:37, 17 February 2014 (UTC)

Yes. The question of which edits should get counted as due to our activities is a tricky one. This would apply if we were using "positive edit size" too. Obviously edits at our events count, whether by a newbie or otherwise. If a newbie starts at one of our events, do all their edits ever count? First six months? And then you have Wikimedians in residence, and people talked to by Wikimedians in residence...

Actually, I know I suggested it, but I am starting to think that WikiTrust is taking us in the wrong direction... I guess it might make sure we aren't responsible for adding a load of guff that gets removed... but that isn't going to happen much in practice.

I think the aspect of WikiTrust that we can draw inspiration from is the more intelligent analysis of a diff, as opposed to the simple change in the number of characters. The improved measure would look at how many characters were added and how many removed. This should be extractable from a diff by looking at how many characters are highlighted blue on the right and how many highlighted orange on the left. For example, this diff would be -17 and +10, rather than -7. This would count as 27 in our improved measure.

Yaris678 (talk) 18:52, 19 February 2014 (UTC)

Overall comments and query about analytic scope for impact assessment

I think this document is a great demonstration of all the thoughtful work WMUK has been engaged in through strategic planning. Kudos on making good progress in mapping your objectives to performance targets and other measures! I have one caution or inquiry at this time. It seems that some of the metrics you propose are focused on the broad counts on wikipedia or other projects rather than identified cohorts of users, is this something that just has not been articulated in the table? Sometimes looking for change in the big picture does not allow you to see progress along the immediate and shorter term outcomes along the pathway to change on the projects overall. It would be nice to see a clearer mapping of the activities you plan do implement in order to target these changes along with the participants/population which would be effected by such activities, but I recognized that you may also be mapping that separately from what is presented here and just not wanting to present a more detailed model. For instance, I know that Daria is leading program work that will provide "interventions" to groups of new or existing editors in various ways and would be appropriate for cohort analysis, so I think this is just an omission from the actual plan/mapping presented here. Importantly, you will likely want to examine most of these indicators at both the program cohort and project level. Lastly, I see you have a lot of plans for developing proxy and outcome measures =) I will look forward to providing any additional consulting on those and/or helping you to route successful measures back to portal documentation and resource supports so that others may also benefit from some of that cross-cutting work. Thank you for sharing your plan openly as an example of the strategic alignment of your objectives and measurement targets. I am sure this is helpful to many chapters and program leaders engaged in similar processes. JAnstee (WMF) (talk) 18:26, 26 February 2014 (UTC)

Other communities in goal 5 and the national level

Just had a chat with User:Seddon who has usefully pointed out that the 5.2/5.3 goals may miss something of the community between WMUK and 'international'. This might mean we're missing metrics in that area (obvious e.g. is meetups and attendance) and that we could track in some more detail e.g. around diversity where on an international level that would be far more challenging.

Reach in G3 and pre/post intervention surveys

User:Seddon also raised interesting possibility around measurement under 3.1 around increasing reach, e.g. running a pre/post intervention use-survey (or similar) before/after running an event or developing some tech for a particular target group. I think this is something we should look at in the long term, and we'll need to think about whether it's for individual activity evaluation or for these strategic measures and targets. Sjgknight (talk) 14:23, 28 February 2014 (UTC)