User:Interwicket/code/mbwa
< User:Interwicket | code
#!/usr/bin/python
# -*- coding: utf-8 -*-
# wikipath en wiktionary User:Interwicket/code/mbwa
"""
This bot updates iwiki links between wiktionaries
It runs as a set of threads, each doing part of the process. The first group of threads generates
tasks (instances of class Task), from the wiki indexes, deletions from same, recent changes, and
null tasks. These are then picked up by the main thread, and distributed to 7 (identical) "hunter"
threads, which read the entries for the title. The pages are then passed to a group of 4 (again
identical) "replink" threads which write the changes (in module reciprocal.py)
Mbwa maintains a local database of all of the titles in all of the 170 wikts, so it knows which
entries will need links. The database is built and updated automatically, it has no trouble with
starting from "scratch". It is divided into 26 files, a to z on titles, titles outside a-z are
reduced modulo 26 on the first letter to a-z. This is done to prevent having to rebuild the entire
index when bsddb (eventually) corrupts one file; that file can be deleted and the process restarted.
Add tasks: this thread reads the title index and language links from each with the MW API. It
queues a task for each apparent discrepency between an entry and the index. In the case where the
title is "new", it adds the links to the index, and does not queue a task, this is essential to
startup (or re-building one index file), if there are links needed they will be found while reading
the index for another wikt (or the same one on the next pass). This thread also generates sets of
titles queued to the next thread (delete). The indexes are read with adjusted delays in between
until the interval is once per week per wikt.
Deletes: this compares each set of titles from add tasks to the indexes in the "inverse" direction,
looking for titles and links in the index not found in the live wikt. In each case it generates
a delete task, added to the main queue. This, in combination with the tasks queued to add links,
ensures that the links will be brought up to date, regardless of RC missed, page moves, reversions,
and any other odd events.
Recent changes: reads RC (from the MW API) from each wikt for new entries more than one hour old
and less than two days. It treats bot entries (including those not flagged as "bot", but where the
username ends with "...bot") as a lower priority when queuing them. This helps give RC entries
created by humans priority. The changes are read at adapted intervals: wikts that show changes are
read increasingly frequently, those that are not are read more seldomly, to a maximum of about 1 day.
The overall rate of API requests is thus kept at a minimum, while still being responsive in finding
changes.
Null: this thread generates a null task at intervals, to keep the main thread spinning.
Main thread: initialization, start other threads; then for each task: read the "primary" entry
referred to by the task object (in the case of a delete, see if another can be found), check the
links, if not complete, set up the task to be passed to a hunter thread.
Hunter: given the set of links from the page and the index, and others found along the way, read
each page to be (potentially) updated for this title. Then queue all the pages with complete sets
of links to the replink threads. All threads use a shared tick-tock timer to limit the page
read rate.
Replink: (in reciprocal.py) replace the links in the page and write if needed, this uses another
tick-tock to limit the page update rate.
Name:
"MBWA" can be given two meanings; the simplest is the acronym for Management By Wandering Around,
that is, the mbwa program just looks for things worth doing, not trying to do one particular list
in a particular order. However, it will -- eventually -- get to everything.
The other explanation is more complicated: in the 1970s I worked on a VLSI design system, that
produced NMC tapes for a photoplotter, that in turn produced the chrome and glass reticles used
to expose the pattern on silicon wafers.
The pattern tapes had the flashes (each a precisely positioned rectangle) sorted into the optimum
order for the photoplotter, to minimize the amount of motion between flashes. (As optimized runs
would still take 36 or more hours, this was very important!) The optimum order for typical arrays
such as memory cells was boustrophedonic -- up one row and down the next -- as the ox ploughs.
In jest, we referred to an unsorted order as "urocanic" -- as the dog pisses.
Since this program appears random in behaviour, and to a great extent is, that would seem to apply,
although it doesn't reduce the performance.
So mbwa: Kiswahili for "dog".
RLU 10.2.9
"""
import wikipedia
import xmlreader
import sys
import socket
import re
from time import *
from random import random, expovariate, choice
from mwapi import getwikitext, getedit, readapi, getticktock
from reciprocal import addrci, flws, getflstatus, replink, plock, updstatus, toreplink, setreptick
from iwlinks import getiwlinks
# borrow global:
from config import usernames
def srep(s):
return repr(u''+s)[2:-1]
def reval(s):
# tricky as repr uses either ' or ", but uses ' if both are present, escaping it
if "'" in s and '"' not in s: return eval('u"' + s + '"')
else: return eval("u'" + s + "'")
def safe(s): return srep(s)
reblank = re.compile(r'\[\[[A-Za-z-]+\s*:[^\]]*\]\]')
def isblank(t, p):
# like isEmpty in wikipedia, but much better and faster, reduces to (almost) identical
if len(t) > 20 and '[[' not in t[:20]: return False
# which is the 99% case, except for images atop, (and pl.wikt ;-)
if len(reblank.sub('',t).strip('\n ')) > 4: return False
return p.isEmpty() # resort to exact framework test, so as not to war with others
respace = re.compile(u'[ _\u00A0\u1680\u180E\u2000-\u200A\u2028\u2029' +
u'\u202F\u205F\u3000]+')
def fixtitle(t):
# fix a page title the way the server does to make DB keys, as pybot framework is out of date
# code from server:
# $dbkey = preg_replace( '/\xE2\x80[\x8E\x8F\xAA-\xAE]/S', '', $dbkey );
# (is BIDI, not done yet)
# $dbkey = preg_replace( '/[ _\xA0\x{1680}\x{180E}\x{2000}-\x{200A}\x{2028}\x{2029}
# \x{202F}\x{205F}\x{3000}]+/u', '_', $dbkey );
t = respace.sub(' ', t).strip(' ')
return t
import shelve
Lcode = { }
Exists = set()
site = { }
naps = { }
def now(): return int(clock())
Quit = False
import threading, Queue
tasq = Queue.PriorityQueue()
# 35K max to keep process memory < ~100MB
# 70K max to handle state of affairs (17.3.9), limit is soft in code below
# hunter queue, just big enough to distribute tasks to 7 or so hunters, soft limit
huntq = Queue.Queue()
# delete task queue, project sets for deltasks to check
delq = Queue.Queue()
# a single index file is 650MB (20.2.9) and if it gets corrupted, we start over
# so we use 26 of them, -a through -z, each title being in the file that is = mod 26 to the first char
# (note this will be different for some chars in a wide build, we just use the first "UTF-16" word)
# a corrupted file can just be deleted and will be re-built
mifs = range(0, 26) # list, to be shelves
# modulus ops on titles; note on a "narrow" build, this only uses the top half of surrogate pairs
def mif(s): return mifs[ord(s[0])%26]
def lkey(s): return 'hijklmnopqrstuvwxyzabcdefg'[ord(s[0])%26]
# Mi = shelve.open("mbwa-index")
milock = threading.RLock()
# (could use 26 locks, but this isn't going to matter that much?)
# mwba index is keyed with srep(title), value is tuple of time, links, redirs
def menc(ul, ur): return "%s,%s" % (" ".join(ul), " ".join(ur))
def mdec(s):
sp = s.split(',')
return sp[0].split(), sp[1].split()
# rather boring sequence of set routines:
def miget(title):
# get the links and redirs list as far as we know
with milock:
Mi = mif(title)
if srep(title) in Mi:
ul, ur = mdec(Mi[srep(title)])
return ul, ur
return [], []
def miadd(code, title, others = []):
# add a link from the wikt indexes, as well as others when first found
with milock:
Mi = mif(title)
if srep(title) in Mi:
ul, ur = mdec(Mi[srep(title)])
else:
ul = []
ur = []
if code in ul: return
if code in ur:
ur.remove(code)
if not ul:
us = set(others)
us.add(code)
us -= set(ur)
ul = sorted(us)
else:
ul.append(code)
Mi[srep(title)] = menc(ul, ur)
def midel(code, title):
# delete a link when we find no page
with milock:
Mi = mif(title)
if srep(title) in Mi:
ul, ur = mdec(Mi[srep(title)])
else:
return
if code in ul: ul.remove(code)
if code in ur: ur.remove(code)
if not ul and not ur: del Mi[srep(title)]
else: Mi[srep(title)] = menc(ul, ur)
def mired(code, title):
# found a redirect, move to redirs list
with milock:
Mi = mif(title)
if srep(title) in Mi:
ul, ur = mdec(Mi[srep(title)])
else:
ul = []
ur = []
if code in ur and code not in ul: return
if code in ul:
ul.remove(code)
if code not in ur:
ur.append(code)
Mi[srep(title)] = menc(ul, ur)
def miset(title, ul, ur):
# at completing an entry (title), record links and redirs
with milock:
Mi = mif(title)
Mi[srep(title)] = menc(ul, ur)
Mi.sync()
def miall(tix, nap = 1.0):
# get the index keys for one of the files (title is a-z)
# takes a non-trivial amount of time, but less than a minute
# then return title, links, redirs for each
# caller must take care not to block as we have lock held
# 200 on each lock, as locking on each entry is ridiculous amounts of CPU!
Mi = mif(tix)
with milock: mk = Mi.keys()
klim = len(mk)
k = 0
while k < klim:
i = 0
with milock:
while k+i < klim and i < 200:
if mk[k+i] in Mi: # (may have gone away ;-)
ul, ur = mdec(Mi[mk[k+i]])
yield reval(mk[k+i]), ul, ur
i += 1
k += 200
sleep(nap) # while not holding lock
# done
# read page titles and links from wikts, return apparent mismatches
from getlinks import getlinks
def livelinks(home):
redirs = flws[home].redirs
# sets of titles present, for delete scan
pset = { }
for k in 'abcdefghijklmnopqrstuvwxyz': pset[k] = set()
# read page title and links from the wikt, compare to our index
for title, links, redflag, bad in getlinks(flws[home].site, plock = plock):
if Quit: break
pset[lkey(title)].add(title)
# if a redirect, add as such, continue
if redflag:
mired(home, title)
continue
ul, ur = miget(title)
# ll is set, validate ...
ll = set()
for lc in links:
lc = str(lc) # not unicode (at Py2.6)
if lc not in flws:
# odd case, WMF server thinks it is a language (user copied links from 'pedias?)
# remove these: (will set lockedwikt in flw.__init__)
flws[lc].deletecode = True
if flws[lc].lockedwikt and not flws[lc].deletecode: continue
# we want to edit to delete in second case
ll.add(lc)
# if there are no links at all (not even home!) then this is a new title (to us)
# (there may have been redirects) don't do it on this pass (if it is itself okay, not bad)
if not ul and not bad:
miadd(home, title, sorted(ll))
continue
# make sure this wikt is present for title (it is not a redirect)
if home not in ul:
miadd(home, title)
ul, ur = miget(title)
if redirs: ul += ur
# compare links to ul, should match
# first add home to ll, then it should be identical
ll.add(home)
# if not redirs, but some present, is okay (at this point):
if not redirs and ur:
for lc in ur: ll.discard(lc)
# (also no point in trying to read them in hunt ;-)
# similar but different case for nolink, e.g. pl->ru
for lc in flws[home].nolink:
if lc in ul: ll.add(lc) # pretend present for comparison
# if apparent mismatch, or bad link(s) in the entry
if sorted(ll) != sorted(ul) or bad:
lcs = set(ul)
lcs.discard(home)
lnotu = [x for x in ll if x not in ul]
unotl = [x for x in ul if x not in ll]
# with plock: print " in LL, not in UNION:", lnotu
# with plock: print " in UNION, not in LL:", unotl
# some difference, so nk always > 1
yield title, lcs, ur, len(lnotu) + len(unotl) + 1, bad
# else: with plock: print "(%s matches)" % repr(title)
for k in 'abcdefghijklmnopqrstuvwxyz':
delq.put( (home, k, pset[k]) ) # start scans for deletions in this wikt
# extraordinarily silly omission from Python
def sign(x):
if x < 0: return -1
if x > 0: return 1
return 0
kloset = clock()
def klo(): return (clock() - kloset) / 1000.0
# Task class, with comparison key r controlled for time
# also slots optimization, (yes, tacky, but we gen up a lot of these)
class Task:
__slots__ = [ 'home', 'title', 'r', 'nk', 'src', 'force', 'page', 'onq' ]
def __init__(self, home='', title='', r=0, nk=0, src='', force=False, onq = 0):
self.home = home
self.title = title
self.r = r or expovariate(.01)
self.nk = nk
self.src = src
self.force = force
self.onq = onq or time()
self.page = None # added in main thread
# now set r to move forward, run queue, not bury things forever
while self.r > 4200.0: self.r -= 700.0
self.r += klo()
def __cmp__(self, other):
return sign(self.r - other.r)
# return tasks in random but prioritized order
qpw = { } # queue per wikt, to implement max and to-be-done
qpwlock = threading.Lock()
# keep track of least recently used (done) wikts, persists on disk
lru = shelve.open("mbwa-lru-list")
# last seen, a "timeout set", a set that elements magically disappear from after a time
from weakref import WeakValueDictionary
from heapq import heappush, heappop
class tmo(float): pass
class lastseen():
def __init__(s, timeout):
s.timeout = timeout
s.wdict = WeakValueDictionary()
s.theap = []
def add(s, key):
t = tmo(clock())
s.wdict[key] = t
heappush(s.theap, t)
def __contains__(s, key):
while s.theap and s.theap[0] < clock() - s.timeout: heappop(s.theap)
return key in s.wdict
def addtasks():
np = 0.0
# entries seen already, use Weak-Val-Dict directly, on titles->tasks
# title will be "in" seen if still on queue or hunt
seen = WeakValueDictionary()
# init lru:
for lc in Exists:
lc = str(lc) # not unicode
if flws[lc].lockedwikt: continue
if lc not in lru: lru[lc] = 0.0
# cleanup:
for lc in lru.keys():
if flws[lc].lockedwikt: del lru[lc]
lrulen = len(lru)
# one off 22.5.10
# lru['ml'] = 0
# for each, read everything, set priorities
mint = 20
while not Quit:
np += 1.0/lrulen # "pass" number
# find least recently done wikt:
home = sorted(lru.keys())[0] # or any other given code, not in unicode, but lru key
old = lru[home]
for lc in lru:
if lru[lc] < old:
old = lru[lc]
home = lc
# if less than a week has passed, sleep for a while, and then go ahead
# 70 min * 150 wikts is about a week (168 hours), is about the target
if lru[home] + (168 * 3600) > time():
mint = min(mint+2, 90)
else:
mint = max(mint-5, 20)
for i in range(mint, 0, -1):
with plock: print "(add tasks: sleeping, %s next in %d)" % (home, i)
sleep(60)
if Quit: break
if Quit: break
# record it now, so if we fail (or are aborted) we go on to the next one
lt = strftime("%d %B %Y %H:%M:%S", gmtime(lru[home])).lstrip('0')
if not lru[home]: lt = '[never]'
lru[home] = time()
lru.sync()
# queue all within process memory reason
qmax = (70000 - tasq.qsize()) / 2
with qpwlock:
if home not in qpw: qpw[home] = 0
qmax += qpw[home]
with plock: print "(reading links from %s.wikt, last done %s, qmax %d)" % (home, lt, qmax)
# skip codes we don't want
# we only want "home" wikts with bot or noflag status
flw = flws[home]
ponly = False
status = getflstatus(flw, nowrite = True)
if status not in ["bot", "noflag", "globalbot", "test", "blocked"]:
# we don't want this one at all
with plock: print "(%s status is %s, not reading at all)" % (home, flw.status)
continue
if status not in ["bot", "noflag", "globalbot"]:
with plock: print "(%s status is %s, not queueing from links)" % (home, flw.status)
ponly = True
tf = 0
qt = 0
for title, lcs, urs, nk, bad in livelinks(home):
if Quit: break
if ponly: # not bot or noflag, we are just counting them
tf += 1
continue
# if it is known as a redirect, skip it (odd case)
if home in urs: continue
# clip main page here if we can
if title.lower() == 'main page': continue
if title == flw.mainpage: continue
if title in seen: continue # queued already on this run, counted in qpw
tf += 1 # found a (new) task
if qpw[home] > qmax: continue # doesn't make the cut
with qpwlock: qpw[home] += 1
r = expovariate(.001)
t = Task(home=home, title=title, r=r, nk=nk, src='idx', force=bad)
seen[title] = t
# if lots on queue, no hurry ...
if tasq.qsize() > 500: sleep(1)
tasq.put(t)
qt += 1
if Quit: break
with plock:
print "(found %d tasks for %s, queued %d, queue size %d)" % (tf, home, qt, tasq.qsize())
if flw.status in ["bot", "noflag", "globalbot", "test"]:
flw.tbd = tf - qt + qpw[home] # total less queued on this pass plus on-queue (!)
updstatus(flw)
sleep(70) # rest between indexes. Is a huge effort, reading them ... (:-)
# (real reason is to keep process from spinning when little to do, main timer above)
# end of trueloop
with plock: print "(add tasks thread ending)"
# now recent changes set
def recent(home = 'en'):
# entries seen already, timeout set, keep for > 3 days (use 4)
seen = lastseen(4 * 86400)
# set up list of wikt codes to look at
qtime = { }
rcstart = { }
for lc in Exists:
if flws[lc].lockedwikt: continue
site[lc] = flws[lc].site
naps[lc] = 60 * choice(range(3, 91)) # scatter 3 to 90 minutes
if lc == home: naps[lc] = 0 # no wait for home wikt the first time
qtime[lc] = now() + naps[lc] # initial time
rcstart[lc] = ''
ny = 0
rcex = re.compile(r'<rc[^>]*title="(.+?)"[^>]*>')
rccont = re.compile(r'rcstart="(.+?)"')
rcisbot = re.compile(r'<rc[^>]*user="[^>]*bot"[^>]*>', re.I)
while not Quit:
# sleep until next one
nextq = now() + 1000000
nextlc = ''
for lc in qtime:
if qtime[lc] < nextq:
nextq = qtime[lc]
nextlc = lc
st = nextq - now()
# if st > 90:
# with plock: print "(%d, sleeping %d minutes, %s next)" % (now(), (st+29)/60, nextlc)
if st > 0:
sleep(st)
if st < -120:
with plock: print "(rc %d minutes behind)" % (-(st-29)/60)
lc = nextlc
if Quit: break
# for mbwa, only read rc from non-troublesome wikts
# this saves the bother of looking at closed wikts too
flw = flws[lc]
if getflstatus(flw, nowrite = True) not in ["bot", "noflag", "globalbot", "test"]:
with plock: print "(%s status is %s, not reading rc)" % (lc, flw.status)
qtime[lc] = now() + 86400 # look again tomorrow
continue
# read recentchanges, new entries, namespace 0, from site:
if True: # [indent]
# with plock: print "(%d, reading from %s.wikt)" % (now(), lc)
nf = 0
# set parameters
# from a little while ago (8 hours)
if not rcstart[lc]:
rcstart[lc] = '&rcstart=' + strftime('%Y-%m-%dT%H:%M:%SZ', gmtime(time() - 8*3600))
# up to one hour ago
rcend = '&rcend=' + strftime('%Y-%m-%dT%H:%M:%SZ', gmtime(time() - 3600))
# slow start, don't need to pick up too quickly
rclimit = "&rclimit=%d" % min(10 + ny/20, 200)
# with plock: print "(options " + rcend + rcshow + rclimit + ")"
try:
rct = readapi(flw.site,
"action=query&list=recentchanges&format=xml&rcprop=title|user|flags&rcdir=newer" +
"&rctype=new&rcnamespace=0"+rcend+rcstart[lc]+rclimit,
plock = plock)
except wikipedia.NoPage:
with plock: print "can't get recentchanges from %s.wikt" % lc
# rct = ''
# sleep(30)
qtime[lc] = now() + 700 # do other things for a bit
continue
except KeyError:
# local bogosity
with plock: print "keyerror"
sleep(20)
continue
# (They've borked the API by making gratuitous changes, we can't check
# to see if we have an empty "recentchanges" element, because it isn't
# always present now! Look for an attribute too. Sigh.)
if "<recentchanges />" in rct or 'recentchanges=""' in rct:
pass
elif '</recentchanges>' not in rct:
with plock: print "some bad return from recentchanges, end tag not found"
with plock: print safe(rct)
# rct = ''
sleep(30)
qtime[lc] = now() + 300 # do other things for a bit
continue
# continue parameter:
mo = rccont.search(rct)
if mo:
rcstart[lc] = "&rcstart=" + mo.group(1)
else: # we are up to date, set to one hour + 100 sec ago
rcstart[lc] = '&rcstart=' + strftime('%Y-%m-%dT%H:%M:%SZ', gmtime(time() - 3700))
found = False
for mo in rcex.finditer(rct):
if Quit: break
title = mo.group(1)
# unescape, API uses (e.g.) #039 for single '
title = wikipedia.html2unicode(title)
title = title.replace('_', ' ')
if ':' in title: continue
if not title: continue
isbot = False
if 'bot=""' in mo.group(0): isbot = True
if rcisbot.match(mo.group(0)): isbot = True
if lc + ':' + title not in seen:
lcs, urs = miget(title)
# new to us or not?
if lc in lcs:
lcs.remove(lc) # can happen on restarts (or entry re-created)
isnew = False
else:
isnew = True # new entry created with iwikis, do it anyway
nk = len(lcs) + len(urs) + 1
if nk == 1: continue # unique title
seen.add(lc + ':' + title)
if not isbot:
t = Task(home=lc, title=title, nk=nk, src='rc', force=isnew)
else:
t = Task(home=lc, title=title, nk=nk, src='bot', r = expovariate(.0014))
tasq.put(t)
ny += 1
nf += 1
found = True
if found:
naps[lc] /= 2
naps[lc] = max(naps[lc], 300) # five minutes
else:
mn = naps[lc]/300 # one-fifth, in minutes
naps[lc] += 60 * choice(range(5, 10 + mn))
# add 5-10 minutes or longer if we don't find anything
maxnap = 60 * choice(range(1400, 1500)) # around 24 hours
naps[lc] = min(naps[lc], maxnap)
qtime[lc] = now() + naps[lc]
with plock: print "(rc found %d in %s, next in %d minutes)" % (nf, lc, (naps[lc]+29)/60)
"""
if naps[lc] > 90:
with plock:
elif naps[lc] > 30:
with plock: print "(rc found %d in %s, next in %d seconds)" % (nf, lc, naps[lc])
else:
with plock: print "(rc found %d in %s, next immediately)" % (nf, lc)
"""
with plock: print "(recent changes thread ending)"
def deltasks():
# incoming queue is sets of titles, already sorted by modulus key
# this is a bit more complicated than just a set for the whole wikt, but allows
# us to release memory for each set as we go
psets = { } # dict by key letter and then lc of sets queued to us
found = { } # dict by lc of number found
tbc = 0 # total titles to be checked
while True:
# scan our whole local db, looking for entries that list a title, when the title is not in wikt
for tix in 'abcdefghijklmnopqrstuvwxyz':
# look for new task sets, block as we must have one, or not enough to be worth it yet
# pick up whatever is available now, add to our little structure
while delq.qsize() > 0 or not psets or tbc < 100000:
lc, k, pset = delq.get()
if k not in psets: psets[k] = { }
if lc in psets[k]: tbc -= len(psets[k][lc])
psets[k][lc] = pset # if we have somehow wrapped all the way 'round use new one!
found[lc] = 0 # make sure it exists
tbc += len(pset)
# any for this letter?
if tix not in psets: continue
ptix = psets[tix]
with plock:
print "(starting delete scan for %s/%s, tbc %d)" % (','.join(ptix.keys()), tix, tbc)
for lc in found: found[lc] = 0
# read index file
for t, ul, ur in miall(tix, nap = 1.0): # pole, pole, hakuna matata!
# NOTE milock held here by miall(), don't take other locks or block
# we do take tasq sync lock implicitly, and release it
if Quit: return # should unwrap everything?
for lc in ptix:
if t not in ptix[lc] and (lc in ul or lc in ur):
if t != fixtitle(t):
# got a bad one somewhere, delete from db now
midel(lc, t)
continue # no need to check the fixed title?
""" better handled by 'exists' for 'del' below? more general case?
if lc == 'ml' and u'\u0d4d\u200d' in t:
# "bad" titles from before forced 5.1 "normalization", don't do delete
# op, it will add bad iwikis
# this prevents thrashing, but doesn't solve problem
continue
"""
# queue up a delete task
task = Task(home=lc, title=t, src='del')
tasq.put(task)
found[lc] += 1
break # don't look at other wikts, one task is enough
for lc in ptix:
if found[lc]:
with plock: print "(delete scan for %s/%s, %d found)" % (lc, tix, found[lc])
tbc -= len(ptix[lc])
ptix = None
del psets[tix] # done with all for letter key, discard
# end of True-loop
with plock: print "(delete thread ending)"
def nulltask():
wasrt = 10.0
# keep main task queue and thread slithy et lubriceaux
while not Quit:
sleep( min(tasq.qsize() + 70, 350) )
tasq.put( Task(src='null') )
# adjust rate (mostly this is for fun, though it is useful in spreading load ;-)
# below 200, is 7 sec, above 2700 is 2 seconds, at 7000 1 second, at 10K no reptick
rt = min(max((3700.0-tasq.qsize())/500.0, 2.0), 7.0)
# and corrections outside range: (cover the range, so no restarts, this is the serious advant)
if tasq.qsize() > 5000: rt = 1.5
if tasq.qsize() > 7000: rt = 1.0
if tasq.qsize() > 10000: rt = 0.0
if tasq.qsize() < 10: rt = 10.0
if int(wasrt*10) != int(rt*10):
with plock: print "(replink ticktock was %.3f, now %.3f)" % (wasrt, rt)
setreptick(rt)
wasrt = rt
with plock: print "(null thread exiting)"
def main():
socket.setdefaulttimeout(70)
with plock: flws['en'].site.forceLogin()
# setup basics
for c in 'hijklmnopqrstuvwxyzabcdefg':
mifs[ord(c)%26] = shelve.open('mbwa/mbwa-index-' + c, protocol = 2)
enw = wikipedia.getSite(code = "en", fam = "wiktionary")
# make sure we have an flw for everything claimed to be in family (including stops)
for code in flws['en'].site.family.langs: foo = flws[code]
# get active wikt list
# minus crap. Tokipona? what are they thinking? Klingon? ;-) deleted ISO code
# se has no wiktionary (not even closed), as is locked (but not shown locked in table?)
Lstops = ['tokipona', 'tlh', 'sh', 'se', 'as']
sitematrix = readapi(enw, "action=sitematrix&format=xml")
rematrix = re.compile(r'//([a-z-]+)\.wiktionary')
sms = set()
for code in rematrix.findall(sitematrix):
sms.add(code)
# print "found code", code, len(sms)
if code in Lstops: continue
Exists.add(code)
foo = flws[code]
# see if we have a login in user config, else pretend we do
# has to be done before any call, or login status gets confused!
if code not in usernames['wiktionary']:
usernames['wiktionary'][code] = "Interwicket"
# set delete for anything not in matrix:
for lc in flws:
if lc not in sms: flws[lc].deletecode = True
with plock: print "found %d active wikts" % len(Exists)
if len(Exists) < 150: return
for lc in Exists:
site[lc] = wikipedia.getSite(lc, "wiktionary")
naps[lc] = 0 # nil, might be referenced by hunt()
with plock: print "starting ..."
# start task generation threads, then yield queue entries:
tt = threading.Thread(target=addtasks)
tt.daemon = True # kill silently on exit (:-)
tt.name = 'get link tasks'
tt.start()
rt = threading.Thread(target=recent)
rt.daemon = True # kill silently on exit (:-)
rt.name = 'get recent changes'
rt.start()
dt = threading.Thread(target=deltasks)
dt.daemon = True # kill silently on exit (:-)
dt.name = 'delete scan'
dt.start()
nt = threading.Thread(target=nulltask)
nt.daemon = True # kill silently on exit (:-)
nt.name = 'null task generator'
nt.start()
# now "hunter tasks"
for i in range(1, 8):
ht = threading.Thread(target=hunter)
ht.daemon = True
ht.name = 'hunter %d' % i
ht.start()
nt = 0
while True:
task = tasq.get()
if task.src == 'null':
with plock:
print '(null points r %.4f on queue %.2f seconds clock %.1f queue %d ' \
'hunt %d replink %d tick tock %.1f)' \
% (task.r, time() - task.onq, clock() - kloset, tasq.qsize(),
huntq.qsize(), toreplink.qsize(), getticktock())
continue
# queue limit from/for addtasks (;-)
if task.src == 'idx':
with qpwlock: qpw[task.home] -= 1
nt += 1
# Task:
with plock: print nt, '('+task.src+')', task.home, srep(task.title), \
"links", task.nk, "random", "%.4f"%(task.r), "queue", tasq.qsize()
# locals, and coerce types
home = task.home
title = task.title
ul, ur = miget(title)
lcs = set(ul)
urs = set(ur)
lcs.discard(home)
urs.discard(home)
mysite = wikipedia.getSite(home, 'wiktionary')
page = wikipedia.Page(mysite, fixtitle(title))
task.page = page
title = task.title = page.title()
if ':' in title: continue # redundant, but eh?
if title.lower() == 'main page': continue
if not title: continue
# with plock: print "%s:%s" % (home, srep(title))
# structure of code here is leftover from source (;-)
tag = True
if tag:
# ... pick up current version
try:
# text = page.get()
text = getwikitext(page, plock = plock)
oldtext = text
if isblank(text, page):
# we don't want to update other entries, but treat this as missing
# we will look at it again every few days, it may then have content
with plock: print " ... page is effectively blank"
midel(home, title)
text = ''
except wikipedia.NoPage:
with plock: print " ... %s not in %s.wikt" % (safe(page.title()), safe(home))
midel(home, title)
# if task.src == 'del' and lcs:
if lcs: # hmmm...
# others?
home = lcs.pop()
if flws[home].status in ['bot', 'globalbot', 'test', 'noflag']:
# requeue to ourselves: (can happen more than once)
task.home = home
task.src = 'delrq' # hmmm...
tasq.put(task)
text = ''
except wikipedia.IsRedirectPage:
with plock: print " ... redirect page"
mired(home, title)
text = ''
except KeyError:
# annoying local error, from crappy framework code
with plock: print "KeyError"
sleep(20)
continue
except Exception, e:
with plock: print "unknown exception from getwikitext", repr(e)
sleep(30)
continue
if not text: continue
# if case was delete, and exists, we are done
# this covers the Malayam (ml) Unicode 5.1 force case, page appears to exist
if task.src == 'del':
with plock: print " ...", srep(title), "exists now"
continue
act = ''
# use our newer code, not framework
ls = getiwlinks(text, flws).keys()
# special case for pl here ...
for lc in flws[home].nolink:
if lc not in ls: lcs.discard(lc)
# wikt links to redirs
if flws[home].redirs: lcs |= urs
# list of iwikis in entry should match lcs, if not, we need to update
if sorted(ls) == sorted(lcs) and not task.force:
with plock: print " ...", srep(title), "is okay"
miadd(home, title) # ensure present in rc case (added with iwikis?)
continue
# if not always adding redirs to this wikt, but some present, is ok
# also nolink wikts
if (not flws[home].redirs or flws[home].nolink) and not task.force:
ok = True
# need to remove something
for s in ls:
if s not in lcs and s not in urs and s not in flws[home].nolink: ok = False
# need to add something
for s in lcs:
if s not in ls: ok = False
if ok:
with plock: print " ...", srep(title), "is okay (may have redirects or nolinks)"
miadd(home, title)
continue
# go hunt down some iwikis, add reciprocals when needed
with plock: print " ... hunting iwikis for", srep(title)
sleep(huntq.qsize()*5) # q limit to reasonable?
huntq.put(task)
# loop on task ends
# done
def hunter():
while not Quit:
task = huntq.get()
# locals, and coerce types
home = task.home
title = task.title
page = task.page
links, redirs, complete = hunt(page)
if Quit: break # return from hunt will not be valid
# and update this page:
addrci(page, flws[home].site, links = links, redirs = redirs, remove = complete)
# record this title as done, links and redirs known
if complete:
ul = set(links.keys())
ul.add(home)
ur = set(redirs.keys())
# sorted is nice, and makes lists again
miset(title, sorted(ul - ur), sorted(ur))
# else it will get done again at some point, hopefully without exceptions
with plock: print "(hunter thread ending)"
# wiki-hunt ... see if a word is in other wikts, return list ...
def hunt(page):
word = page.title()
text = getwikitext(page, plock = plock) # will just return _contents
home = page.site().lang
ul, ur = miget(word)
totry = set(ul) | set(ur)
done = set()
fps = set()
links = { }
redirs = { }
# reiw = re.compile(r'\[\[([a-z-]{2,11}):' + re.escape(word) + r'\]\]')
# simple scan for existing iwikis, use improved code
# for lc in reiw.findall(text):
iws = getiwlinks(text, flws)
for lc in iws:
lc = str(lc) # not unicode
# if lc in site:
totry.add(lc)
# not home:
totry.discard(home)
done.add(home)
exceptions = False
while totry:
lc = totry.pop()
if flws[lc].lockedwikt or flws[lc].deletecode: continue
if Quit: return None, None, False
try:
fpage = wikipedia.Page(site[lc], word)
text = getwikitext(fpage, plock = plock)
if isblank(text, fpage):
# we don't want to link to entirely blank pages
with plock: print " ", srep(word), "in", lc, "is blank or empty"
done.add(lc)
continue # not adding to links
with plock: print " ", srep(word), "found in", lc
except wikipedia.NoPage:
with plock: print " ", srep(word), "not in", lc
done.add(lc)
continue
except wikipedia.IsRedirectPage:
redirs[lc] = fpage
with plock: print " ", srep(word), "found in", lc, "(redirect)"
except Exception, e:
exceptions = True
with plock: print "exception testing existence of word", str(e)
done.add(lc)
continue
done.add(lc)
links[lc] = fpage
# add to list to add reciprocal link, or complete set, don't (can't :-) update redirects
if lc not in redirs: fps.add(fpage)
# look for iwikis in the page, add to to-be-tried if not already done
iws = getiwlinks(text, flws)
for lc in iws:
lc = str(lc) # not in unicode
if lc not in done and lc not in totry:
with plock: print " found further iwiki", lc
totry.add(lc)
# all done, now add reciprocals
# don't remove anything if there were exceptions because hunt may be incomplete
# if no exceptions, hunt is complete for these entries (there may be others not seen,
# but then they aren't linked, as we've looked at all links ...), so remove any
# links not found:
for fpage in fps:
if Quit: return None, None, False
addrci(fpage, site[home], links=links, redirs=redirs, remove=not exceptions)
# return list of all links and redirects, and flag if complete
return links, redirs, not exceptions
# end? Finally?
if __name__ == "__main__":
try:
main()
except KeyboardInterrupt:
print "(keyboard interrupt)"
# mostly just suppress traceback
except Exception, e:
print "exception", repr(e)
finally:
Quit = True
replink(end = True)
sleep(210) # give a bit of a chance for add tasks/hunt/rc to stop cleanly
for i in range(0, 26):
print "closing index file for", 'hijklmnopqrstuvwxyzabcdefg'[i]
sleep(1) # time for print
with milock: mifs[i].close()
lru.close()
wikipedia.stopme()