python實現Simhash算法
1、simhash步驟
simhash包含分詞、hash、加權、合並、降維五大步驟
simhash代碼如下:
import jieba import jieba.analyse import numpy as np class SimHash(object): def simHash(self, content): seg = jieba.cut(content) # jieba.analyse.set_stop_words('stopword.txt') # jieba基於TF-IDF提取關鍵詞 keyWords = jieba.analyse.extract_tags("|".join(seg), topK=10, withWeight=True) keyList = [] for feature, weight in keyWords: # print('feature:' + feature) print('weight: {}'.format(weight)) # weight = math.ceil(weight) weight = int(weight) binstr = self.string_hash(feature) print('feature: %s , string_hash %s' % (feature, binstr)) temp = [] for c in binstr: if (c == '1'): temp.append(weight) else: temp.append(-weight) keyList.append(temp) listSum = np.sum(np.array(keyList), axis=0) if (keyList == []): return '00' simhash = '' for i in listSum: if (i > 0): simhash = simhash + '1' else: simhash = simhash + '0' return simhash def string_hash(self, source): if source == "": return 0 else: temp = source[0] temp1 = ord(temp) x = ord(source[0]) << 7 m = 1000003 mask = 2 ** 128 - 1 for c in source: x = ((x * m) ^ ord(c)) & mask x ^= len(source) if x == -1: x = -2 x = bin(x).replace('0b', '').zfill(64)[-64:] return str(x) def getDistance(self, hashstr1, hashstr2): ''' 計算兩個simhash的漢明距離 ''' length = 0 for index, char in enumerate(hashstr1): if char == hashstr2[index]: continue else: length += 1 return length
1.1分詞
分詞是將文本文檔進行分割成不同的詞組,比如詞1為:今天星期四,詞2為:今天星期五
得出分詞結果為【今天,星期四】【今天,星期五】
1.2hash
hash是將分詞結果取hash值
星期四hash為:0010001100100000101001101010000000101111011010010001100011011110
今天hash為:0010001111010100010011110001110010100011110111111011001011110101
星期五hash為:0010001100100000101001101010000000101111011010010000000010010001
1.3加權
1.4合並
1.5降維
降維是將合並的結果進行降維,如果值大於0,則置為1小於0 則置為0,因此得到的結果為:
2、simhash比對
一般simhash采用海明距離來進行計算相似度,海明距離計算如下:
對於A,B兩個n維二進制數
二者的海明距離為:
其中:
舉例:
1000與1111的海明距離為3
到此這篇關於python實現Simhash算法的文章就介紹到這瞭,更多相關python實現Simhash算法內容請搜索WalkonNet以前的文章或繼續瀏覽下面的相關文章希望大傢以後多多支持WalkonNet!