Talk:WMUK membership survey 2013/Survey draft

Demographics

Ok so in 2012 we asked questions about income, marital status, and children. What we were trying to do was get a sense of barriers to attend events or the circumstances of members households so we could be accommodating (offering a creche at a conference maybe!) However I'm not sure these questions are as useful as we hoped.

What are better questions to ask that cover important problems like:

• How affordable is attending events in terms of travel and fee costs?
• Do your household circumstances present a potential barrier to participation? i.e. young or single parenting, carer, student under age of 16 etc

I think we don't need to ask about people's relationship status - its not useful or necessary. It could be helpful to ask about their households, but again I'm fuzzy on why. Best not if we're not sure. Katherine Bavage (WMUK) (talk) 10:01, 16 October 2013 (UTC)

Those are some really personal questions. If I was asked that in a survey from an organisation I was peripherally involved with, I wouldn't answer them, and I probably wouldn't respond to the survey at all. Do we really need to know if somebody is transgender, and if we do, do we really need to know what their gender was at birth? How does that help us? The same goes for sexual orientation, annual income, and relationship status. I can understand asking whether members are parents/carers, but do we really need to know how many children they have? There are also carers of disabled adults to consider. If there's something specific you want know - eg whether affordability is a barrier - why not ask that instead of making assumptions? Harry Mitchell (talk) 13:41, 16 October 2013 (UTC)

Thanks Harry. I'll try and break it down a bit cos you've raised a few points;

• "Do we really need to know if somebody is transgender, and if we do, do we really need to know what their gender was at birth?" - This is following best practice guidelines about asking questions about gender identity. Basically asking 'Are you a 'man', 'woman' or 'other' is just not good enough (NOT that I'm saying that's what you're suggesting.) If you look at what I was talking about on the intro page and the draft blurb for the demographics section, the reason you might monitor gender identity is both to do with trying to monitor if our membership is broadly mirroring that of wider society, and if not are there things we need to think about in terms of outreach work or barriers to participation. I'm really hoping this kind of data will feed into what people bring back from the diversity conference for instance. So that's the 'why'
• "I wouldn't answer them, and I probably wouldn't respond to the survey at all." - yep that's definitely a risk. I have on my to do list to do some reading about how to develop/encourage ways organisations can encourage people to disclose this information because they can see the value it will create. Again, this will be anonymous, all the questions will be optional as well, and I hope if we can link the ones we DO ask to data that informs decisions that might demonstrate value to people.
• "If there's something specific you want know - eg whether affordability is a barrier - why not ask that instead of making assumptions?" It's not that simple. You get responder bias about how relatively well off people are, and you cant compare that to office of national statistics data about national or regional trends we can benchmark against. I would welcome suggestions for better questions, and I'd probably err on the side of not including questions than asking rubbish ones. I like asking about caring commitments rather than number of children per se for example.
• Thanks so much for feedback. If you can make suggestions for questions that would help me lots, or just come and review what I've worked on next week IF you get chance (I know you're busy :-)) Katherine Bavage (WMUK) (talk) 13:57, 16 October 2013 (UTC)

Additional note - we should be asking about disability in this context too. Katherine Bavage (WMUK) (talk) 14:26, 16 October 2013 (UTC)

Good points. As I think I commented, I copied the text for the demographics questions from the best practice page on meta (as I can't think of a reason to not follow best practice). As for what to ask, my thought was to draft lots of potential questions and then select the ones we want to ask from that list. Thryduulf (talk: local | en.wp | en.wikt) 22:03, 16 October 2013 (UTC)

I too find the questions in the demographics section far too intrusive and too remote from the projects, in particular 2,3,4,7,8.9 and 10. I would normally bin any survey that asked such questions. We are in danger here of following so much best practice that we get the result that most of these surveys get of almost no replies. Following recent data protection lapses that have been covered in the media I think that most people have little confidence that their data will be kept confidential and it might be better to adopt a light touch with this survey. For gender, why not just ask people what gender they are and leave it at that? Let them fill in the reply free form and drop questions 2-4. Asking about relationship status and children also seems to go too far and it's hard to see why we need to do so. Honestly, what business does WMUK have asking about these things? Philafrenzy (talk) 18:48, 24 October 2013 (UTC)

Incidentally, Schedule 1 to the Data Protection Act states that sensitive personal data should not be processed unless it is "relevant" and "not excessive" in relation to the purpose for which it is processed. Philafrenzy (talk) 11:49, 25 October 2013 (UTC)

My supposition is that we are asking this information so that we can use it to compare our results with the population as a whole and with participation in Wikimedia projects as a whole. Ottomh I'm not sure why a distinction between married/civil partner/living with someone as if they were married is relevant though. Re gender, there was criticism from transgender groups about the Wikipedia data on transgender participation as there were only options "male","female","transgender" which they think meant that the number of transgender people was underreported as a transman for example would want to tick "male" and "transgender". What reposnsibilities people have for caring for children is important for planning of events and understanding barriers to participation at events. You have just given me the thought that we should explicitly ask whether the availability of childcare is a barrier to WMUK events, or whether we should be holding meetups in more child-friendly locations than pubs. Thryduulf (talk: local | en.wp | en.wikt) 12:07, 25 October 2013 (UTC)

Hi Philafrenzy. We had the same objections last year and we're going to move to a 'light touch' in the sense of the demographic questions a) Coming last b) Being optional and anonymous c) Having a clear statement on use, storage and purpose. I don't want to throw the baby out with the bathwater however - having NO questions about who are members are leaves us in a poor position to serve their needs.

It would be very helpful if people suggested better questions as well as saying they don't like existing ones :) I agree that asking if someone is married is basically irrelevant so better questions about circumstances that affect participation and engagement needed, but also ones that are somewhat objective (so not 'can you afford to attend events' because this tells us nothing useful about how we could help) Also if you have come across survey experts across the projects over the years do point them in the direction of this discussion and the meta page because I think we could all do a bit better on this front. Katherine Bavage (WMUK) (talk) 13:30, 25 October 2013 (UTC)

Can everyone drop in questions you would expect to have answered in here so I can try and factor that in? WMUK_membership_survey_2013/FAQs Katherine Bavage (WMUK) (talk) 13:53, 25 October 2013 (UTC)

The question order

Probably sucking eggs here.

General practice in such questions is to start with the most general questions and then move towards the specifics. This launches straight into questions that affect very small parts of the population. Should start with ethnicity, identified gender, age, etc and then work towards more specific things.

We rightly make sure that people realise such questions are optional. Jon Davies (WMUK) (talk) 10:48, 16 October 2013 (UTC)

The demographics questions will - in many cases - put people off straight away. They should be moved right to the end, I think. --MichaelMaggs (talk) 18:11, 16 October 2013 (UTC)

A few months back I spent a while filling in lots of professionally developed surveys about all sorts of things. From memory they were split about 60/40 asking demographics questions first/last. I don't think that provides strong guidance either way, so unless someone knows of a reason why they should be first moving them to the end would seem fine to me. The order of sections on the draft page reflects a combination of the order they were asked in last year (I don't know how that was decided) and what seemed logical to me at the end of the day yesterday.

I believe that all the questions will be optional, and that this will be clearly stated in the final presentation (on SurveyMonkey). Explicitly saying that the demographics questions are optional shouldn't be too difficult, but as we want people to answer them (otherwise we wouldn't ask) so we don't want to discourage people so some thought should go into the warning of any such marking.Thryduulf (talk: local | en.wp | en.wikt) 22:11, 16 October 2013 (UTC)

I'd agree that they should be moved to the end. The survey should start with the most important questions that are being asked, in order to maximise participation in that part of the survey. Optional sections such as the demographs should be last so that people that don't want to answer them can easily say 'no' and submit the rest of their responses, and also so that they aren't put off filling in the survey at the first page. If people have already spent time filling in most of a survey, and are then presented with a section they don't want to fill in, then they're more likely to submit their responses and use the option to skip the rest than with someone that is asked if they want to skip optional demograph questions right at the start without having already invested their time in the survey. Thanks. Mike Peel (talk) 22:31, 22 October 2013 (UTC)

Attendance at events

I've added a question to the draft about whether respondents have ever attended an event. I think it would be worthwhile to ask some more detailed questions about barriers to attendance (such as not knowing about events, no events in their area, being unavailable at weekends, etc) and whether addressing some of those might make them more inclined to come to an event. It might also be worth asking whether respondents have edited Wikipedia (and if no whether they'd like to) to see if we et any correlations to help make sense of the data. We could also ask how they first heard of WMUK, and it might be nice to ask for suggestions for future events - most of the responses might not be of much use, but we might get somebody who works for an organisation we want to partner with and is willing to act as a go-between for example. Harry Mitchell (talk) 14:04, 16 October 2013 (UTC)

Good plan. We did this a bit last time but it over focused on meetups. I'll have a go at drafting some questions :D Katherine Bavage (WMUK) (talk) 14:09, 16 October 2013 (UTC)

We should also ask about projects other than Wikipedia. Thryduulf (talk: local | en.wp | en.wikt) 22:12, 16 October 2013 (UTC)

Transgender question

Why would we want to ask "If you are transgender, what was your assigned gender status at birth?". As the previous questions asked for gender identity and whether you are transgender, this extra detail seems oddly unnecessary and possibly offensive, particular if asking someone who was born intersex to fill in "Other (please specify)". I suggest this particular question is dropped as problematic.

Anyone reviewing this might find GLAAD's guide helpful.[1] --Fæ (talk) 17:01, 24 October 2013 (UTC)

Yes, drop it and a few others too. Philafrenzy (talk) 18:48, 24 October 2013 (UTC)

The question is copied verbatim from the best practices page on meta, which sites http://gaygenderresearch.org/ as the source for that question. It seems they have reorganised their website since the page on meta was written. I don't have time to hunt for the new location right now, sorry. Thryduulf (talk: local | en.wp | en.wikt) 11:56, 25 October 2013 (UTC)

If someone can track it down it would be useful to compare with. I suggest a key criteria for deciding which questions to ask in a WMUK survey should be that "if we don't understand and publicly explain how we would use specific personal data, then we don't collect it." With Wikipedia in the news with regard to poor treatment of transgender subjects, it would be better for WMUK to be criticised for excessive caution or political correctness, rather than a lack of care. --Fæ (talk) 12:42, 25 October 2013 (UTC)

For my twopennoth - this isn't my preferred approach to asking questions about gender - this is. As to why we might ask questions about gender - I'm guided by the twin principles of understanding and better serving a respondee group and in this case, asking questions about gender that represent a range of circumstances. It might seem tempting to just stick to asking if someone is 'male' or 'female' or 'transgender' but that's actually quite poor practice (I'm not suggesting that is what people have recommended, just explaining why a superficially simple solution isn't necessarily a good one)

Please be reassured I understand that in doing so the questions would have to be anonymous and optional - if you look elsewhere you'll see that we effectively plan to have the demographic questions as a second 'follow up' survey and encourage buy-in by demonstrating clearly how the data will be safe and could inform decision making. I can elaborate on that?

What is people's instinctual preference here? I think we have to choose between not asking about gender at all, or asking in a representative and accessible way. I personally prefer the latter but am happy to listen to views. Katherine Bavage (WMUK) (talk) 13:23, 25 October 2013 (UTC)

"restricted reference material"

What does Q9 mean by "Access to restricted reference material", specifically what does "restricted" mean in this context? I'm guessing it's nothing to do with security classification? Thryduulf (talk: local | en.wp | en.wikt) 12:49, 25 October 2013 (UTC)

In the context of that question, I would think that mean restricted to members. -- Katie Chan (WMUK) (talk) 13:11, 25 October 2013 (UTC)

Actually I think Mike may have meant journals you pay to access or books you can't get hold of without buying or travelling to read? Katherine Bavage (WMUK) (talk) 13:24, 25 October 2013 (UTC)