Microgrants/32GB usb stick for Commons dump

Overview

My bot, Faebot has been a busy bot lately. As a result my networked hard disk that holds my local dump of commonswiki.xml gets accessed constantly from several threads running on my macmini. I would like to move the 22GB file to flash memory on USB stick to avoid problems with wear and tear on my hardware. At the moment, the largest USB stick I have to hand is 8GB but the prices of 32GB have fallen dramatically, so this seems realistic.

Bot owners are expected to use a local dump of commonswiki.xml for lengthy or complex bot work, to avoid putting a lot of transactions on the Wikimedia servers. Naturally this means that the stress of high volume transactions moves to your own home kit!

Budget

£16 for a small form-factor 32GB USB stick. See example supplier.

Timeline

• Indefinitely. I would be unable to loan it, or use in on other machines as it would be in constant bot use from my home macmini. If I stop running Wikimedia scripts that use such a dump, or the Commons dump gets too large for a 32GB stick, I would be happy to lend it on to another volunteer that has a use for it.
• Progress will be seen on Commons:User:Faebot.
• I would expect this 32GB of storage to be good at least throughout 2013. At the moment a reasonable xml dump of Commons image pages is running at 22GB, it may well exceed 32GB by 2014.

Expected outcomes

• Reduce wear and tear on my home desktop drive (putting my personal backup and archives at risk).
• Enable Faebot to continue with the 2 million+ UK Geograph image categorization plus the other odd tasks it gets up to within scope. See Commons:User:Faebot/Geograph for current projects including sorting UK images by County/Borough using Open Street Map data (probably a year of slow bot work).

Who I am

I am Fæ, I do a lot on Commons.
--Fæ (talk) 19:38, 22 October 2012 (UTC)

Discussion

Hi Fæ. Thanks for submitting this microgrant application. :-) Since you're a current fellow trustee, I'm going to take a cautious approach to deciding on this application, and will ask a couple of other people to chime in prior to a decision being made. I hope that's OK with you.

With regards the application, on a technical basis, I'm not sure that a flash drive is the best solution here, both in terms of reliability (cheap flash drives won't last long under rigorous usage) or access speed (10MB/s read isn't particularly quick). Have you considered a small USB2 or firewire external hard drive? With regards ownership, I note that the drive would belong to WMUK and that it should be returned to the office once you've finished using it, either for recycling or reuse. Would that be OK with you? Thanks. Mike Peel (talk) 20:34, 22 October 2012 (UTC)

Hi both, you'll find that the newer cheap usb drives have a very limited life in terms of the number of write cycles they will sustain. But for around £16 for 32GB, you can almost start to think about them as solid state and much faster versions of DVR-R - i.e. consumables. I can't see any reason not to accept this, with the usual caveat that we are not to be seen to favour a trustee. Naturally, I'd be happy to lend my endorsement to other volunteers who were doing similar work and had similar needs in future. --RexxS (talk) 20:43, 22 October 2012 (UTC)

I have the feeling that this is a well documented and supported request. I see good arguments why this might be a good idea (it would help Faebot in its work), I see no immediate downsides and the requested amount falls within the limitations of the program. If the technical alternative that Mike suggests is significantly better and not extremely more expensive, I would prefer that. Volunteers should get the means to do their work as well as possible - which is the exact reason for this program. I realize Fae is a trustee, but while it is not allowed to give trustees advantage in these programs, I don't see benefit to giving them disadvantage. 82.139.72.116 20:51, 22 October 2012 (UTC)