tf-idf-1

我把数据全部都读到内存了,不适合大数据... ...

文本中某行

教育#287543.txt:英国 少年 16 岁 创业 赚 人生 第一 个 100万 16 岁 时 你 在 干 什么 ? 忙 着 做 功课 还是 玩 ? 英国 少年 克里斯琴·欧文斯 16 岁 时 通过 经营 公司 , 赚 得 他 人生 的 第一 个 100万 英镑 ( 约合 150万 美元 ) 。

file2Map.java

    首先把所有文本转为HashMap<类别,<文档名,文档内容>>

IDF.java

    计算IDF

TF.java

    计算TF

TFIDF.java

    计算文档中词的tf*idf值

输出结果:

游戏 [冰=0.06649351276067433, 您=0.06073662514664949, 域=0.060263931983040295, 开放=0.05557732752017935, 等级=0.0518109175859678, OL=0.04051887865604911, 妖=0.03615835918982418, BOSS=0.03369636721454702, 印=0.030389158992036834, ----=0.028575283680407116, 树木=0.028575283680407116, ?=0.028079854569499733, 塔=0.02678567251571151, 上古=0.026597405104269733, 时而=0.026597405104269733, 着急=0.025194077921578118, 勇敢=0.024105572793216117, 等待=0.023295984183343395, 地图=0.023047924049131092, 全新=0.023047924049131092, 魔=0.02213209617385788, 精致=0.02072436703438712, .=0.02072436703438712, 封=0.02072436703438712, 渐渐=0.01983499358661173, 天=0.019540493227624835, 编=0.019444541354057893, 场景=0.018746488458249732, --=0.018746488458249732, 啦=0.018431666403920112, 女=0.018099094325397698, 8日=0.01759337246769251, 吗=0.017503882329393167, 无情=0.01597824471961806, 触角=0.01597824471961806, 密室=0.01597824471961806, 鼎城=0.01597824471961806, 苍凉=0.01597824471961806, 无尽=0.01597824471961806, 黄沙=0.01597824471961806, 积聚=0.01597824471961806, 湮没=0.01597824471961806, 黄土地=0.01597824471961806, 妖媚=0.01597824471961806, 群英=0.01597824471961806...]

[时空=0.14261736417145798, 完美=0.10284800603587496, ChinaJoy=0.09243287028770744, 国际化=0.03633270363698692, 》=0.034204375411664475, 《=0.034204375411664475, 游戏=0.03391942881607323, 主题=0.032861977417901415, 西游=0.025842525976502582, 09CJ=0.025842525976502582, 互动=0.025139025100883783, 玩=0.025091876168048415, 正式=0.023288220838431746, CJ=0.02310821757192686, 诛=0.021508749689843374, 赤壁=0.021508749689843374, 派对=0.020373909167351142, 大作=0.018774441285267653, 口袋=0.01816635181849346, 专题=0.01816635181849346, 即=0.017944096378914, 2009年=0.017944096378914, 世界=0.017112849707057384, 成果=0.016759350067255856, 舞=0.016759350067255856, 家=0.016679488503440767, 分享=0.01638337302660763, 仙=0.01638337302660763, 发展=0.015924898160869026, 2008年=0.015160333067923847, 广大=0.013832575531834248, 快乐=0.01364906462203191, 款=0.013624763863870093, 累累=0.012921262988251291, 硕果=0.012921262988251291, 喜讯=0.012921262988251291, perfect=0.012921262988251291, 2009CJ=0.012921262988251291, 场区=0.012921262988251291, games=0.012921262988251291, 一年一度=0.012921262988251291, live=0.012921262988251291, show=0.012921262988251291, QQ96061=0.012921262988251291...]

[天下无双=0.1149312339481299, 诡=0.08808922036817231, 阶段=0.06400394590238564, 级=0.0624110684464318, 江湖=0.05882741361546084, 死神=0.05872614691211488, 流=0.055907774300559224, 阎罗=0.0492562431206271, 巅峰=0.03883297724754522, 步入=0.03578436741214549, 职业=0.0319176391484404, 你=0.029592110672873737, 越发=0.02936307345605744, 》=0.025353243177944282, 《=0.025353243177944282, 降临=0.024770137456688237, 杀手=0.023856244941430325, 新=0.021884607937829485, 苦=0.02182383838449717, 成为=0.021097066222905043, 传说=0.02038182298373636, 哦=0.019980606403919136, !=0.019303822568260014, 此后=0.018939807582975553, 至=0.01708890387471992, 邪魔=0.016418747706875698, 101=0.016418747706875698, 侯=0.016418747706875698, 冥=0.016418747706875698, 行者=0.016418747706875698, 名副其实=0.016418747706875698, 疾风=0.016418747706875698, 吃苦耐劳=0.016418747706875698, 洗礼=0.016418747706875698, 41-60=0.016418747706875698, 迈入=0.016418747706875698, 闻风丧胆=0.016418747706875698, 诡怪=0.016418747706875698, 21-40=0.016418747706875698, 亦=0.01631700986987005, 派=0.016134715914126317, 玩=0.015941831099248804, 充满=0.015465385625281594, 暗影=0.01468153672802872, 历=0.0146815367280287...]