Talk:WMUK membership survey 2013/Survey draft

Demographics

Ok so in 2012 we asked questions about income, marital status, and children. What we were trying to do was get a sense of barriers to attend events or the circumstances of members households so we could be accommodating (offering a creche at a conference maybe!) However I'm not sure these questions are as useful as we hoped.

What are better questions to ask that cover important problems like:

• How affordable is attending events in terms of travel and fee costs?
• Do your household circumstances present a potential barrier to participation? i.e. young or single parenting, carer, student under age of 16 etc

I think we don't need to ask about people's relationship status - its not useful or necessary. It could be helpful to ask about their households, but again I'm fuzzy on why. Best not if we're not sure. Katherine Bavage (WMUK) (talk) 10:01, 16 October 2013 (UTC)

Those are some really personal questions. If I was asked that in a survey from an organisation I was peripherally involved with, I wouldn't answer them, and I probably wouldn't respond to the survey at all. Do we really need to know if somebody is transgender, and if we do, do we really need to know what their gender was at birth? How does that help us? The same goes for sexual orientation, annual income, and relationship status. I can understand asking whether members are parents/carers, but do we really need to know how many children they have? There are also carers of disabled adults to consider. If there's something specific you want know - eg whether affordability is a barrier - why not ask that instead of making assumptions? Harry Mitchell (talk) 13:41, 16 October 2013 (UTC)

Thanks Harry. I'll try and break it down a bit cos you've raised a few points;

• "Do we really need to know if somebody is transgender, and if we do, do we really need to know what their gender was at birth?" - This is following best practice guidelines about asking questions about gender identity. Basically asking 'Are you a 'man', 'woman' or 'other' is just not good enough (NOT that I'm saying that's what you're suggesting.) If you look at what I was talking about on the intro page and the draft blurb for the demographics section, the reason you might monitor gender identity is both to do with trying to monitor if our membership is broadly mirroring that of wider society, and if not are there things we need to think about in terms of outreach work or barriers to participation. I'm really hoping this kind of data will feed into what people bring back from the diversity conference for instance. So that's the 'why'
• "I wouldn't answer them, and I probably wouldn't respond to the survey at all." - yep that's definitely a risk. I have on my to do list to do some reading about how to develop/encourage ways organisations can encourage people to disclose this information because they can see the value it will create. Again, this will be anonymous, all the questions will be optional as well, and I hope if we can link the ones we DO ask to data that informs decisions that might demonstrate value to people.
• "If there's something specific you want know - eg whether affordability is a barrier - why not ask that instead of making assumptions?" It's not that simple. You get responder bias about how relatively well off people are, and you cant compare that to office of national statistics data about national or regional trends we can benchmark against. I would welcome suggestions for better questions, and I'd probably err on the side of not including questions than asking rubbish ones. I like asking about caring commitments rather than number of children per se for example.
• Thanks so much for feedback. If you can make suggestions for questions that would help me lots, or just come and review what I've worked on next week IF you get chance (I know you're busy :-)) Katherine Bavage (WMUK) (talk) 13:57, 16 October 2013 (UTC)

Additional note - we should be asking about disability in this context too. Katherine Bavage (WMUK) (talk) 14:26, 16 October 2013 (UTC)

Good points. As I think I commented, I copied the text for the demographics questions from the best practice page on meta (as I can't think of a reason to not follow best practice). As for what to ask, my thought was to draft lots of potential questions and then select the ones we want to ask from that list. Thryduulf (talk: local | en.wp | en.wikt) 22:03, 16 October 2013 (UTC)

I too find the questions in the demographics section far too intrusive and too remote from the projects, in particular 2,3,4,7,8.9 and 10. I would normally bin any survey that asked such questions. We are in danger here of following so much best practice that we get the result that most of these surveys get of almost no replies. Following recent data protection lapses that have been covered in the media I think that most people have little confidence that their data will be kept confidential and it might be better to adopt a light touch with this survey. For gender, why not just ask people what gender they are and leave it at that? Let them fill in the reply free form and drop questions 2-4. Asking about relationship status and children also seems to go too far and it's hard to see why we need to do so. Honestly, what business does WMUK have asking about these things? Philafrenzy (talk) 18:48, 24 October 2013 (UTC)

Incidentally, Schedule 1 to the Data Protection Act states that sensitive personal data should not be processed unless it is "relevant" and "not excessive" in relation to the purpose for which it is processed. Philafrenzy (talk) 11:49, 25 October 2013 (UTC)

My supposition is that we are asking this information so that we can use it to compare our results with the population as a whole and with participation in Wikimedia projects as a whole. Ottomh I'm not sure why a distinction between married/civil partner/living with someone as if they were married is relevant though. Re gender, there was criticism from transgender groups about the Wikipedia data on transgender participation as there were only options "male","female","transgender" which they think meant that the number of transgender people was underreported as a transman for example would want to tick "male" and "transgender". What reposnsibilities people have for caring for children is important for planning of events and understanding barriers to participation at events. You have just given me the thought that we should explicitly ask whether the availability of childcare is a barrier to WMUK events, or whether we should be holding meetups in more child-friendly locations than pubs. Thryduulf (talk: local | en.wp | en.wikt) 12:07, 25 October 2013 (UTC)

Hi Philafrenzy. We had the same objections last year and we're going to move to a 'light touch' in the sense of the demographic questions a) Coming last b) Being optional and anonymous c) Having a clear statement on use, storage and purpose. I don't want to throw the baby out with the bathwater however - having NO questions about who are members are leaves us in a poor position to serve their needs.

It would be very helpful if people suggested better questions as well as saying they don't like existing ones :) I agree that asking if someone is married is basically irrelevant so better questions about circumstances that affect participation and engagement needed, but also ones that are somewhat objective (so not 'can you afford to attend events' because this tells us nothing useful about how we could help) Also if you have come across survey experts across the projects over the years do point them in the direction of this discussion and the meta page because I think we could all do a bit better on this front. Katherine Bavage (WMUK) (talk) 13:30, 25 October 2013 (UTC)

Can everyone drop in questions you would expect to have answered in here so I can try and factor that in? WMUK_membership_survey_2013/FAQs Katherine Bavage (WMUK) (talk) 13:53, 25 October 2013 (UTC)

The intro to the demographics sections currently includes the words "we want to reach out to less-well-represented groups and engage them.", would that be better phrased as "...engage with them"? Thryduulf (talk: local | en.wp | en.wikt) 21:19, 25 October 2013 (UTC)

How about a plain English re-write? That section seems packed with jargon, we don't have to inherit the management-speak from the WMF. As for 'engage' it reminds me of engaging the Borg. --Fæ (talk) 00:18, 26 October 2013 (UTC)

Please feel free to have a go :) Katherine Bavage (WMUK) (talk) 11:32, 26 October 2013 (UTC)

By all means ignore these comments if there are no resources to improve the survey this way.

I spent a lot of volunteer time writing up policies when we created the charity, being one of the few people around who had suitable qualifications and experience. Now there are people paid full time to do it, writing and improving this stuff for free seems far less worthwhile compared to using my wiki-time to add another 100,000 images to Commons. Happy to consider being paid should that ever arise. --Fæ (talk) 12:09, 26 October 2013 (UTC)

Yes you did and its how I know you're a good writer :) If you're too busy I understand - its just that its hard for me to rewrite what I wrote in a way you want - I thought it might be easier for you to just make some changes. Anyway, I'll have look at trying to split up sentences and simplify words - and anyone else reading this who could do that, thanks! Katherine Bavage (WMUK) (talk) 13:03, 26 October 2013 (UTC)

PS perhaps if we linked to engage the borg videos everytime I use the word engage this would dramatically increase participation :-) Katherine Bavage (WMUK) (talk) 13:05, 26 October 2013 (UTC)

The question order

Probably sucking eggs here.

General practice in such questions is to start with the most general questions and then move towards the specifics. This launches straight into questions that affect very small parts of the population. Should start with ethnicity, identified gender, age, etc and then work towards more specific things.

We rightly make sure that people realise such questions are optional. Jon Davies (WMUK) (talk) 10:48, 16 October 2013 (UTC)

The demographics questions will - in many cases - put people off straight away. They should be moved right to the end, I think. --MichaelMaggs (talk) 18:11, 16 October 2013 (UTC)

A few months back I spent a while filling in lots of professionally developed surveys about all sorts of things. From memory they were split about 60/40 asking demographics questions first/last. I don't think that provides strong guidance either way, so unless someone knows of a reason why they should be first moving them to the end would seem fine to me. The order of sections on the draft page reflects a combination of the order they were asked in last year (I don't know how that was decided) and what seemed logical to me at the end of the day yesterday.

I believe that all the questions will be optional, and that this will be clearly stated in the final presentation (on SurveyMonkey). Explicitly saying that the demographics questions are optional shouldn't be too difficult, but as we want people to answer them (otherwise we wouldn't ask) so we don't want to discourage people so some thought should go into the warning of any such marking.Thryduulf (talk: local | en.wp | en.wikt) 22:11, 16 October 2013 (UTC)

I'd agree that they should be moved to the end. The survey should start with the most important questions that are being asked, in order to maximise participation in that part of the survey. Optional sections such as the demographs should be last so that people that don't want to answer them can easily say 'no' and submit the rest of their responses, and also so that they aren't put off filling in the survey at the first page. If people have already spent time filling in most of a survey, and are then presented with a section they don't want to fill in, then they're more likely to submit their responses and use the option to skip the rest than with someone that is asked if they want to skip optional demograph questions right at the start without having already invested their time in the survey. Thanks. Mike Peel (talk) 22:31, 22 October 2013 (UTC)

Attendance at events

I've added a question to the draft about whether respondents have ever attended an event. I think it would be worthwhile to ask some more detailed questions about barriers to attendance (such as not knowing about events, no events in their area, being unavailable at weekends, etc) and whether addressing some of those might make them more inclined to come to an event. It might also be worth asking whether respondents have edited Wikipedia (and if no whether they'd like to) to see if we et any correlations to help make sense of the data. We could also ask how they first heard of WMUK, and it might be nice to ask for suggestions for future events - most of the responses might not be of much use, but we might get somebody who works for an organisation we want to partner with and is willing to act as a go-between for example. Harry Mitchell (talk) 14:04, 16 October 2013 (UTC)

Good plan. We did this a bit last time but it over focused on meetups. I'll have a go at drafting some questions :D Katherine Bavage (WMUK) (talk) 14:09, 16 October 2013 (UTC)

We should also ask about projects other than Wikipedia. Thryduulf (talk: local | en.wp | en.wikt) 22:12, 16 October 2013 (UTC)

"One of the main activities that WMUK organises are offline events." Does this read a little strangely to anyone else? I think it's the "one of...are", but I'm not sure and I can't think of a better way to phrase it myself. Thryduulf (talk: local | en.wp | en.wikt) 20:11, 25 October 2013 (UTC)

Gender questions

Why would we want to ask "If you are transgender, what was your assigned gender status at birth?". As the previous questions asked for gender identity and whether you are transgender, this extra detail seems oddly unnecessary and possibly offensive, particular if asking someone who was born intersex to fill in "Other (please specify)". I suggest this particular question is dropped as problematic.

Anyone reviewing this might find GLAAD's guide helpful.[1] --Fæ (talk) 17:01, 24 October 2013 (UTC)

Yes, drop it and a few others too. Philafrenzy (talk) 18:48, 24 October 2013 (UTC)

The question is copied verbatim from the best practices page on meta, which sites http://gaygenderresearch.org/ as the source for that question. It seems they have reorganised their website since the page on meta was written. I don't have time to hunt for the new location right now, sorry. Thryduulf (talk: local | en.wp | en.wikt) 11:56, 25 October 2013 (UTC)

If someone can track it down it would be useful to compare with. I suggest a key criteria for deciding which questions to ask in a WMUK survey should be that "if we don't understand and publicly explain how we would use specific personal data, then we don't collect it." With Wikipedia in the news with regard to poor treatment of transgender subjects, it would be better for WMUK to be criticised for excessive caution or political correctness, rather than a lack of care. --Fæ (talk) 12:42, 25 October 2013 (UTC)

For my twopennoth - this isn't my preferred approach to asking questions about gender - this is. As to why we might ask questions about gender - I'm guided by the twin principles of understanding and better serving a respondee group and in this case, asking questions about gender that represent a range of circumstances. It might seem tempting to just stick to asking if someone is 'male' or 'female' or 'transgender' but that's actually quite poor practice (I'm not suggesting that is what people have recommended, just explaining why a superficially simple solution isn't necessarily a good one)

Please be reassured I understand that in doing so the questions would have to be anonymous and optional - if you look elsewhere you'll see that we effectively plan to have the demographic questions as a second 'follow up' survey and encourage buy-in by demonstrating clearly how the data will be safe and could inform decision making. I can elaborate on that?

What is people's instinctual preference here? I think we have to choose between not asking about gender at all, or asking in a representative and accessible way. I personally prefer the latter but am happy to listen to views. Katherine Bavage (WMUK) (talk) 13:23, 25 October 2013 (UTC)

It's not instinct, rather understanding what we are going to do with the result and whether the outcome is then worth the burden of the question. Could you explain why we would want to know "assigned gender status at birth" if we already have asked if someone identified as transgender and they already gave their gender identity?

Considering that someone who might actually give an answer to the question has probably spent a long time moving on from how they were labeled at birth, and WMUK is neither a LGBT, gender or political specialist organization, this level of detail seems both intrusive and superfluous. GLAAD's general principle for journalists is to focus on current gender identity rather than making a big deal out of past or biological gender, as indeed is the evolving standard for Wikimedia (as far as I can tell, this was the whole point of the recent fracas at en.wp).

Oh, by the way, the survey may be anonymous, but we have very few members. This means that if only 50 people reply, and of these the stats say that only 5 people identify as LGBT and only two say something about being transexual, well, it's not that hard to work out who they must be, particularly for me with a background with Wikimedia LGBT. For these reasons any precautions of anonymity might be undermined by basic deduction. --Fæ (talk) 15:03, 25 October 2013 (UTC)

OK so to clarify the thinking behind the current questions

1. If we ask questions about gender, should they be representative of the lived transgender experience or just 'male'/'female'(Not other - I didn't like that question set and changed it yesterday)
2.  I think we need to understand the gender balance of our membership because it is still predominantly male and we need to address that. I hope that clarifies what we are going to do with the result - it will be reported feed into thinking across what the charity does and hopefully improve representation of groups that current have low engagement.
3. I don't think that the best way to do this is to effectively not allow people who identify as transgender to have that represented. Could you confirm you've had a look at the Equality and Human Rights commission guidance that I linked to? That explains why this might not be a good approach. I'd rather you refute that so I know we're operating from fundamentally different perspectives rather than a question of tweaking the survey draft. I'm actually not sure we *are* so far apart - the GLADD guide (another good one by the way, thanks) clarifies

   'When describing transgender people, please use the correct term or terms to describe their gender identity. For example, a person who is born male and transitions to become female is a transgender woman, whereas a person who is born female and transitions to become male is a transgender man.'

   If this is the case are we in consensus that we could ask questions, but better ones (and what would these be?) or do you maintain we just shouldn't ask because we're a generalist org not a representative one?
4.  The question of numbers/anonymity. If someone filled out this form then the person reviewing the response could possibly cross calculate characteristics to identify the person (E.g. This person is born in 1970, is male, and says their ethnicity is white irish - oh that must be Bob) That is always an issue when the pool of people you are surveying is small as you say. However - back to first principles - does that mean you don't survey small groups? In my opinion, no, it means you undertake the following precautions; a) All responses are anonymous b) All questions are optional c) Only a limited number of analysts review the individual responses as opposed to the meta data d) The meta data is reported according to best practice (again referring to guidance - '*'ing population numbers if the numbers are below a certain level) e) The individual responses are deleted after they have been reviewed f) a - e are clearly communicated to all participants.

I hope this is helpful. To help me it would be good if you have a look at the Equality and Human Rights commission guidance and then feel free to come back and tell me where this doesn't square with our community approaches as you refer to and I will definitely consider it. I'm in complete agreement about prioritising the needs of respondents and the burden of the question being worth the outcome. That is very important and I would probably rather reluctantly have no demographic data on members than cause an individual worry or embarrassment from clumsy questions. Its only because I think trying to have a sense of gender identity amongst members might help lead to better representation in the long term that this section matters. Anyway - value your input to make the right call even if we are approaching this from different angles. Katherine Bavage (WMUK) (talk) 12:06, 26 October 2013 (UTC)

With regard to the section title here, I was specifically raising the transgender questions as an issue, rather than wanting to get drawn into a more general discussion about surveying and analysing gender.

I did look at the Equality and Human Rights commission guidance, I don't need to look at it again, and it just did not seem pitched for the organization that WMUK is. That was the point I was making previously.

In terms of the current questions with regard to transgender, they still seem superfluous to anything the charity would actually do with them, if anyone answers the questions this becomes a risk if the data is used and reported in any form as the numbers would be so low as to reveal unnecessary personal information, and I don't understand how this is particularly helpful in delivering the mission of the charity. For example I don't know of anyone who was "officially" declared as intersex at birth, that is incredibly unlikely to have happened any date before the current century hence irrelevant to the target sample space, and the questions about "the process" of changing sex are intrusive verging on offensive, considering I cannot imagine why WMUK would want such information from the one or two people that might answer it.

Personally, I have spent about 2 hours on this, in contrast this week I have spent around 8 hours helping with a mass upload on the Welsh Wikipedia and around 20 minutes on a harassment problem on IRC. Having expressed my opinion fairly clearly on the record already, on balance I don't think that spending a long time refining one or two questions that extremely few people will give an answer to is worth diverting much more time on. If you would like more relevant alternative opinions to balance mine, I suggest raising a request for views on what is suitable at m:LGBT. Thanks --Fæ (talk) 13:25, 26 October 2013 (UTC)

"restricted reference material"

What does Q9 mean by "Access to restricted reference material", specifically what does "restricted" mean in this context? I'm guessing it's nothing to do with security classification? Thryduulf (talk: local | en.wp | en.wikt) 12:49, 25 October 2013 (UTC)

In the context of that question, I would think that mean restricted to members. -- Katie Chan (WMUK) (talk) 13:11, 25 October 2013 (UTC)

Actually I think Mike may have meant journals you pay to access or books you can't get hold of without buying or travelling to read? Katherine Bavage (WMUK) (talk) 13:24, 25 October 2013 (UTC)

Yup, I meant it to mean what Katherine's explained. Thanks. Mike Peel (talk) 15:48, 25 October 2013 (UTC)

Would perhaps "restricted-availability" express that better? Thryduulf (talk: local | en.wp | en.wikt) 19:32, 25 October 2013 (UTC)

Intro

In the intro the sentence about how long the data is kept isn't brilliantly worded. Currently it reads:

When the meta report about the results has been produced and any responses that can be followed up stored on your member contact record, all responses will be deleted within 60 days of the survey closing.

I think either

When the meta report about the results has been produced and any responses that can be followed up stored on your member contact record, all individual responses will be deleted. This will be within 60 days of the survey closing.

or

Within 60 days, when the meta report about the results has been produced and any responses that can be followed up stored on your member contact record, all individual responses will be delete.

would be better. I would simply make the change, but I can't decide which I prefer. Thryduulf (talk: local | en.wp | en.wikt) 19:30, 25 October 2013 (UTC)

I don't object as long as its clear to readers when the data will be looked at, reported on and then deleted :) Pick a reword that you feel is clearest! Katherine Bavage (WMUK) (talk) 11:33, 26 October 2013 (UTC)

NUTS regions

While I can see the logic in the concept of using NUTS, I'm not sure that they are the best structured for the purposes of locating events. Especially in Wales they do not correspond at all well to transport routes - Swansea or the Valleys to Cardiff is dead simple but they are in different regions, whereas Swansea to Caernarfon or Cardiff to Wrexham are intra-region but not easy or quick transportationally. Wales is best divided into South, Mid and North following the Severn Tunnel-Pembroke/Milford Haven, Shrewsbury-Aberystwyth and Chester-Hollyhead railway lines. In England the South East, South West and North West regions are too large to be meaningful - it's quicker and easier to get from Bristol to London than it is to get from Bristol to Cornwall for example (Dover is closer than Penzance!). Similarly London is much easier to get to from Oxford and Canterbury than it is to get between them. Thryduulf (talk: local | en.wp | en.wikt) 20:27, 25 October 2013 (UTC)

Urgh I know but it's hard to find a consistent measure that isn't really, really granular (i.e. at the counties level). Also we could ask about 'distance from home' but people are more likely to be able to travel cheaply to London than some places within 50miles of their home. I really don't know what the best solution is :( Katherine Bavage (WMUK) (talk) 12:42, 26 October 2013 (UTC)

Disability and accessibility

I've added a question on disability. I couldn't find any simple question explicitly described as best practice (only for entire surveys about disability, which is far more detail than we need), so I've used the question from the 2011 census http://www.ons.gov.uk/ons/guide-method/census/2011/the-2011-census/2011-census-questionnaire-content/2011-census-questions---england.pdf It doesn't give us much detail, but I don't really think we need any more in this part of the survey - access needs, etc. are dealt with in part 1.

Should we ask in part 1 though whether the WMUK wiki and/or hardcopy materials are accessible? If we do ask I think we should have a free text box for suggested improvements. Thryduulf (talk: local | en.wp | en.wikt) 12:37, 26 October 2013 (UTC)

Yes to adding the free text for suggestions and thanks for adding the questions - we need to make sure these are good and robust :-) Katherine Bavage (WMUK) (talk) 12:40, 26 October 2013 (UTC)