Talk:Strategy monitoring plan 2014-15

Redirect to:

• Talk:Strategy consultation

skilled & capable

"G2a.3 WMUK volunteers are skilled and capable. Volunteer Activity Units [4] in training sessions and editathons" - do we want to distinguish (measure both?) between units of people delivering training and receiving? Sjgknight (talk) 11:57, 6 February 2014 (UTC)

I suppose it depends whether the aim is more on increasing the skill of volunteers (in which case I think we should capture how much training volunteers receive) or assessing the skill (in which case we could focus on the feedback received by e.g. volunteer trainers after they deliver an event). It does need distinguishing. Daria Cybulska (WMUK) (talk) 10:55, 7 February 2014 (UTC)

We of course care about both, so I guess one answer is jut to measure both. No. of sessions run by trained volunteers is captured elsewhere Sjgknight (talk) 11:06, 7 February 2014 (UTC)

'access to Wikimedia projects is increasingly available'

Enabling new readers to access the projects is a relatively new type of activity for Wikimedia UK. Perhaps we could set a target as running several pilot projects to assess how this area could be approached in the future. Daria Cybulska (WMUK) (talk) 10:55, 7 February 2014 (UTC)

Yes, broadly that's what the output measure indicates (as it's about estimating our increased reach). The projects we see relating to this outcome are things like: accessibility increase; translation work; digital literacy work (which enables people to engage with content); Kiwix deployment and similar projects. QRPedia scans help with bits of that (and of course, are in part about directing people to content in any easy access fashion in places they would ordinarily have not been directed to Wikimedia sites), but clearly we will need to think about other measures as our work continues. Sjgknight (talk) 11:10, 7 February 2014 (UTC)

'supporting Technical Communities' and 'increased awareness of the benefits of Open Content'

Some of the external communities we work with aren't strictly technical (e.g. Open Rights Group) and it's still an important area of our open knowledge work. I suppose a clarification is needed where in G4.1 we are focusing on technology development, and there is another area where we link with other groups to further open knowledge aims. Daria Cybulska (WMUK) (talk) 10:55, 7 February 2014 (UTC)

I have copied this to Talk:Strategic goals. Suggest it is best discussed there. --MichaelMaggs (talk) 08:53, 15 February 2014 (UTC)

Improving on positive edit size

(Copying relevant point from above the TOC to under this heading) Just spotted "G1.2 The quality of educational Open Content continues to improve" should include something re: article quality incrementing, think this must've been lost in a rewording. Sjgknight (talk) 11:51, 6 February 2014 (UTC)

Good/Featured article or Did You Know are a possible although not ideal approaches. Daria Cybulska (WMUK) (talk) 10:55, 7 February 2014 (UTC)

I note that you are looking for something better than positive edit size to measure the contribution that our activities lead to. Has anyone come across WikiTrust? I think the algorithms could be adapted to our requirements. Yaris678 (talk) 23:04, 15 February 2014 (UTC)

Thanks Yaris678, I'm a big fan of WikiTrust and it's on our list of things to look at for the future, at the moment we hadn't thought much about exactly how we'd use such a tool. I guess we'd need to extract the data that the tool uses to do its colour coding, such that for each article edited we had an indicator of the stability of wikitext contributed through wmuk activities v. the rest? It's something we should put some more thought into (and that'd need technical exploration) so any ideas there would be much appreciated too! cheers Sjgknight (talk) 09:25, 16 February 2014 (UTC)

Agreed. I would love to see us working on this. --MichaelMaggs (talk) 09:42, 16 February 2014 (UTC)

By the way, I don't know how stable WikiTrust will be, I think it's going to move to Toolserver, but will also need a maintainer (I don't know if wmuk can support in any way around this). One of the issues re: looking at quality metrics is that we don't just want the rev summary (which tell us +/-) we actually need to do stuff: for each user, for each rev. The WikiTrust api looks interesting http://wikitrust.soe.ucsc.edu/vandalism-api but we'd need to think about how best to implement it. Perhaps we could run some manual testing to try and prototype something useful in the near future.

By the way, I was also chatting to Magnus yesterday about the WikiToDo tool, because another thing we could be interested in is, for particular articles that've received attention (e.g. in an editathon) what was the pre/post event state of the page on some simple metrics (e.g. does it have an image, etc.). Again we'd need to think about how to store it, when to run it and over what span, etc. but it would be fairly easy to implement if we wanted to. Sjgknight (talk) 10:18, 16 February 2014 (UTC)

At the moment WikiTrust doesn't display the reputation of individual contributors (although this is what it bases the text 'trust' metric on). We may be able to gain access to these, although I agree with them that just using aggregate data on cohort reputation (or alike) would be the most we'd want to do. Another tact, which would be closer to our other ideas re: improving quality, would be to look at the 'trust' of the text inserted by wmuk associated editors/activities. Thinking aloud here! Sjgknight (talk) 11:35, 16 February 2014 (UTC)

Yes. That's what I was thinking of using the WikiTrust algorithms for. We want a measure of the "trust" of text added because of our activities. ie. How much does the text stick around? I would have thought the adaptation would be fairly simple. We would effectively treat all such additions as being by one virtual editor and then calculate the "trust" of that editor. It would need someone capable of modifying the code though. Yaris678 (talk) 18:51, 16 February 2014 (UTC)

As I understand, the problem is that the trust of the text is based on the reputation of the contributor in part, so we can't just amalgamate, and from new editors text will never be (immediately) of high trust. That is, the stability of the text isn't the only metric used to calculate text-trust in WikiTrust. A comparative metric might work (between what's there and what's added), or looking at added text after a set period (or number of edits) had passed.

This is actually a broader issue too, one of the things we'll need to think about is tracking period. How long after attendance at a WMUK event do we collect metrics on contributors and their contributions? Should each new event attended "reset" the clock? Do we collect in perpetuity? Do we start to 'discount' our own impact with regard to editors who came to an event, are still editing, but never come to another event? This is an interesting problem and definitely something we (as a community) should spend some time thinking on. Sjgknight (talk) 22:37, 17 February 2014 (UTC)

Yes. The question of which edits should get counted as due to our activities is a tricky one. This would apply if we were using "positive edit size" too. Obviously edits at our events count, whether by a newbie or otherwise. If a newbie starts at one of our events, do all their edits ever count? First six months? And then you have Wikimedians in residence, and people talked to by Wikimedians in residence...

Actually, I know I suggested it, but I am starting to think that WikiTrust is taking us in the wrong direction... I guess it might make sure we aren't responsible for adding a load of guff that gets removed... but that isn't going to happen much in practice.

I think the aspect of WikiTrust that we can draw inspiration from is the more intelligent analysis of a diff, as opposed to the simple change in the number of characters. The improved measure would look at how many characters were added and how many removed. This should be extractable from a diff by looking at how many characters are highlighted blue on the right and how many highlighted orange on the left. For example, this diff would be -17 and +10, rather than -7. This would count as 27 in our improved measure.

Yaris678 (talk) 18:52, 19 February 2014 (UTC)

Overall comments and query about analytic scope for impact assessment

I think this document is a great demonstration of all the thoughtful work WMUK has been engaged in through strategic planning. Kudos on making good progress in mapping your objectives to performance targets and other measures! I have one caution or inquiry at this time. It seems that some of the metrics you propose are focused on the broad counts on wikipedia or other projects rather than identified cohorts of users, is this something that just has not been articulated in the table? Sometimes looking for change in the big picture does not allow you to see progress along the immediate and shorter term outcomes along the pathway to change on the projects overall. It would be nice to see a clearer mapping of the activities you plan do implement in order to target these changes along with the participants/population which would be effected by such activities, but I recognized that you may also be mapping that separately from what is presented here and just not wanting to present a more detailed model. For instance, I know that Daria is leading program work that will provide "interventions" to groups of new or existing editors in various ways and would be appropriate for cohort analysis, so I think this is just an omission from the actual plan/mapping presented here. Importantly, you will likely want to examine most of these indicators at both the program cohort and project level. Lastly, I see you have a lot of plans for developing proxy and outcome measures =) I will look forward to providing any additional consulting on those and/or helping you to route successful measures back to portal documentation and resource supports so that others may also benefit from some of that cross-cutting work. Thank you for sharing your plan openly as an example of the strategic alignment of your objectives and measurement targets. I am sure this is helpful to many chapters and program leaders engaged in similar processes. JAnstee (WMF) (talk) 18:26, 26 February 2014 (UTC)

Other communities in goal 5 and the national level

Just had a chat with User:Seddon who has usefully pointed out that the 5.2/5.3 goals may miss something of the community between WMUK and 'international'. This might mean we're missing metrics in that area (obvious e.g. is meetups and attendance) and that we could track in some more detail e.g. around diversity where on an international level that would be far more challenging. Sjgknight (talk) 14:23, 28 February 2014 (UTC)

Reach in G3 and pre/post intervention surveys

User:Seddon also raised interesting possibility around measurement under 3.1 around increasing reach, e.g. running a pre/post intervention use-survey (or similar) before/after running an event or developing some tech for a particular target group. I think this is something we should look at in the long term, and we'll need to think about whether it's for individual activity evaluation or for these strategic measures and targets. Sjgknight (talk) 14:23, 28 February 2014 (UTC)

Belated comments from Mike

1. "Percentage of WMUK-related files (e.g. images) in mainspace use on a Wikimedia project (excluding Commons)" - wouldn't that be better as an absolute number rather than a percentage?
   1. In this case we're using a percentage because under 1.2 we're considering quality rather than quantity. The indicator for 1.1 is an absolute number, while a percentage is a better indicator of quality. We could feasibly set a target for 1.2 against a percentage but not an absolute number. Given we're providing both absolute (total) figures and the percentage used in mainspace working out the absolute figure wouldn't be hard. Sjgknight (talk) 17:51, 1 March 2014 (UTC)
      1. I'm not so sure about that. You can upload high-quality images to Commons that aren't directly useful on the projects but are still very much worthwhile having on Commons, and further the movement's mission. E.g. uploading different angles of an object of which only one are used on Wikipedia. Maybe look at both the absolute number and percentage? Thanks. Mike Peel (talk) 18:19, 1 March 2014 (UTC)
         1. I think there's a misunderstanding here, we'll track the number of uploads under G1.1, and the percent of those used on Wikipedia, as an indicator of quality, under G1.2. Sjgknight (talk) 09:08, 2 March 2014 (UTC)
            • Doh - sorry, you're right! Apologies. Mike Peel (talk) 18:58, 2 March 2014 (UTC)
2. "Number of files having valued image status on Commons" - you don't seem to be expecting any growth in this, with 21 in both columns?
   1. We'll take another look at this and see if it can be revised Sjgknight (talk) 17:51, 1 March 2014 (UTC)
3. "Institution reputation rating" - that's an odd concept, isn't there an external standard that can be used here? Also, rating 1-12 rather than 1-10 seems a bit odd.
   1. Because we're ranking reputations across multiple of organisation we cannot use a single external standard but insofar as possible we will look to external sources here. Sjgknight (talk) 17:51, 1 March 2014 (UTC)
      1. Using several external standards would be better than trying to define a new (and very arbitrary) standard. Thanks. Mike Peel (talk) 18:19, 1 March 2014 (UTC)
         1. That is indeed what we are doing, but we will not be publishing information that will allow identification of scores for individual partner organizations. That could easily alienate some of our partners or drag the charity into time-consuming and ultimately fruitless arguments about why we have given GLAM organization X a score of, say, 6 rather than 7. --MichaelMaggs (talk) 03:35, 2 March 2014 (UTC)
4. "Number of Friends" - needs a definition of what a 'Friend' is. Why not use membership here?
   1. We have a volunteers working group in existence who will be publishing ideas for increasing the number of volunteers and friends. We consider that increasing our active volunteer base and our number of friends to be more important than increasing the numbers that are formal members of the charity as a legal entity. Sjgknight (talk) 17:51, 1 March 2014 (UTC)
      1. That's the first I've heard of a volunteers working group about this - is there documentation anywhere on this wiki about it? Or is this a cabal? ;-) Thanks. Mike Peel (talk) 18:19, 1 March 2014 (UTC)
         1. Govcom_Minutes_23Jan14#Non-board committees. --MichaelMaggs (talk) 03:35, 2 March 2014 (UTC)
5. "QRPedia codes scanned" - how about also considering the number of qrpedia codes in existence?
   1. While accessing the number of distinct codes created is possible, we're unclear how much information it gives because it doesn't necessarily indicate the codes are used. Sjgknight (talk) 17:51, 1 March 2014 (UTC)
      1. Thinking again about this, there's two different meanings of 'QRPedia codes scanned'. Is this the number of scans, or the number of unique codes scanned? My argument here was about tracking both of these, making the assumption that QRpedia codes that exist are used. I guess you could also ask organisations that use qrpedia codes how many codes they have (or ask volunteers working with those organisations to count them), which would then be another interesting metric when combined with usage statistics. Thanks. Mike Peel (talk) 18:19, 1 March 2014 (UTC)
         1. The KPI counts the number of actual scans made. The number of unique codes scanned is available via the Piwik stats page, but we don't think that is of sufficient important to be accorded KPI status in its on right. Regularly asking other organizations what they do would be possible, but the effort would be disproportionate to the information value that we could obtain. --MichaelMaggs (talk) 03:35, 2 March 2014 (UTC)
6. "Funding to support other chapters and Wikimedia groups" - I would avoid having this as a monetary value as spending money should never be a goal in itself, why not have it as a number of initiatives supported/run?
   1. We agree that it is undesirable in general to count inputs - monetary, activities, etc. - as indicators. However, in this case we feel that money spent directly on things like travel grants, scholarships, etc. offers more information than the number of initiatives supported. Sjgknight (talk) 17:51, 1 March 2014 (UTC)
      1. I disagree. Why do you feel that a monetary value is more relevant here? Thanks. Mike Peel (talk) 18:19, 1 March 2014 (UTC)
         1. Counting 'initiatives' is much less well-defined. Also, because this is not a core activity we might well result in a score of 1 or 2, which has less information-content than "£10000". --MichaelMaggs (talk) 03:35, 2 March 2014 (UTC)
7. I might be missing it, but I can't spot anything about tracking the number of events that WMUK runs/supports? You also don't seem to be including any measures for educational work that WMUK does, and not many on GLAM either?
   1. We have decided not to count events as an indicator. Events form one of our inputs, we do not wish to be in a situation in which events are counted as outputs and as such there is an incentive for the charity to run many events which may not be impactful. Instead we're counting volunteer activity units as a better indicator of our impact. Sjgknight (talk) 17:51, 1 March 2014 (UTC)
      1. Yes indeed. The omission of an 'event count' is intentional for the reasons Simon gives. --MichaelMaggs (talk) 03:35, 2 March 2014 (UTC)

breaking the below into two questions, hope that's ok

1. In general, it would be worth thinking about: if you achieve all of the goals you've set out here, would they have been worth the money that WMUK has spent in that year?
   1. We have tried to write the strategic plan such that we can be relatively clear that the activities on which we spend our funds are targeted at our strategic goals. We will of course continue to evaluate our success against our goals, and capacity build to develop the targets towards our charitable impact over time. Sjgknight (talk) 17:51, 1 March 2014 (UTC)
2. Also, what is WMUK doing that isn't directly impacting one of these goals - and should those goals be added to this list, or the activities ceased? Rather than finalising this part of the plan now, it might be worth putting out a specific appeal for more comments/suggestions for additions and reviewing it again at a future board meeting. Thanks. Mike Peel (talk) 11:26, 1 March 2014 (UTC)
   1. The strategic plan is intended to focus our attentions on our impact as a charity. If we're doing things that are not impacting on the goals in the strategic plan, then that implies those activities should be ceased. However, we're of course not doing that blindly and are open to the possibility that there might be other goals towards our impact. This applies to new activities, and existing ones. We are now in the process of finalising the 5 year plan which the board will vote on next weekend. While the measures and targets form part of the set of the documents the board will vote on, it's important to note that these are annual targets, and we would anticipate both the targets themselves, and potentially the ways we measure towards them changing over the course of the plan. Any changes in measures, will be aligned with the goals as they stand. Although the annual targets will change, they are cumulative and will build impact over time. The cumulative impact in terms of distance travelled will be continually reported on over the 5 years. We don't anticipate the goals changing over the course of the 5 years, however the board would not rule this out should be there be major changes in the context of the charity. Sjgknight (talk) 17:51, 1 March 2014 (UTC)