Gushwell's Dev Notes

   Tweet

72. 素性抽出
極性分析に有用そうな素性を各自で設計し，学習データから素性を抽出せよ．素性としては，レビューからストップワードを除去し，各単語をステミング処理したものが最低限のベースラインとなるであろう．

import re
from nltk import stem

class Stopwords:
    words = ['a', 'about', 'all', 'an', 'and', 'any', 'are', 'as', \
            'at', 'be', 'been', 'but', 'by', 'can', 'could', 'do', \
            'does', 'for', 'from', 'has', 'have', 'he', 'her', 'his', \
            'how', 'i', 'if', 'in', 'into', 'is', 'it', 'its', 'made', \
            'make', 'may', 'me', 'my', 'no', 'not', 'of', 'on', 'one', \
            'or', 'out', 'she', 'should', 'so', 'some', 'than', 'that', \
            'the', 'their', 'them', 'there', 'then', 'they', 'this', \
            'those', 'to', 'too', 'us', 'was', 'we', 'what', 'when',\
            'which', 'who', 'with', 'would', 'you', 'your', ''
        ]

    @staticmethod
    def exists(word):
        return word in  Stopwords.words

class SentimentFeatures:
    def __init__(self, filename):
        self.filename = filename
        self.stemmer = stem.PorterStemmer()

    @staticmethod
    def isValid(word):
        if word == '' or len(word) <= 2:
            return False
        if re.match(r'^[-=!@#$%^&*()_+|;";,.<>/?]+$', word):
            return False
        return not Stopwords.exists(word)

    def getFromLine(self, line):
        sentiment = line[:3]
        array = re.split(r'\s|,|\.|\(|\)|\'|/|\'|\[|\]|-', line[3:])
        # こういう時はlambda キーワードいらないんですね。
        words = filter(self.isValid, array)
        xs = map(self.stemmer.stem, words)
        return sentiment, xs

    def enumerate(self):
        with open(self.filename, 'r') as fin:
            for line in fin:
                yield self.getFromLine(line)

def main():
    sf = SentimentFeatures('chapter08/sentiment.txt')
    for sentiment, features in list(sf.enumerate())[:50]:
        print(sentiment, " ".join(features))

if __name__ == '__main__':
    main()

-1  film depress life itself
-1  ill fit tuxedo strictli off rack
-1  clich escap perfervid treatment gang warfar call ce wild
-1  circuit queen won learn thing busi curs film strateg place white sheet
-1  waterlog script plumb unchart depth stupid incoher sub sophomor sexual banter
+1  imax strap pair goggl shut real world take vicari voyag last frontier space
-1  analyz movi three word thumb friggin down
+1  stori haven seen big screen befor stori american human be know
-1  bray complet sea noth savag garden music video resum clue make movi
+1  teen review such recommend onli under year age onli veri mild rental

   Tweet

71. ストップワード
英語のストップワードのリスト（ストップリスト）を適当に作成せよ．さらに，引数に与えられた単語（文字列）がストップリストに含まれている場合は真，それ以外は偽を返す関数を実装せよ．さらに，その関数に対するテストを記述せよ．

class Stopwords:
    words = ['a', 'about', 'all', 'an', 'and', 'any', 'are', 'as', \
            'at', 'be', 'been', 'but', 'by', 'can', 'could', 'do', \
            'does', 'for', 'from', 'has', 'have', 'he', 'her', 'his', \
            'how', 'i', 'if', 'in', 'into', 'is', 'it', 'its', 'made', \
            'make', 'may', 'me', 'my', 'no', 'not', 'of', 'on', 'one', \
            'or', 'out', 'she', 'should', 'so', 'some', 'than', 'that', \
            'the', 'their', 'them', 'there', 'then', 'they', 'this', \
            'those', 'to', 'too', 'us', 'was', 'we', 'what', 'when',\
            'which', 'who', 'with', 'would', 'you', 'your', ''
        ]

    @staticmethod
    def exists(word):
        return word in  Stopwords.words

def test():
    w = 'in'
    print(w, Stopwords.exists(w))
    w = 'what'
    print(w, Stopwords.exists(w))
    w = 'list'
    print(w, Stopwords.exists(w))
    w = 'yes'
    print(w, Stopwords.exists(w))

def main():
    test()

if __name__ == '__main__':
    main()

in True
what True
list False
yes False

   Tweet

本章では，Bo Pang氏とLillian Lee氏が公開しているMovie Review Dataのsentence polarity dataset v1.0を用い，文を肯定的（ポジティブ）もしくは否定的（ネガティブ）に分類するタスク（極性分析）に取り組む．

70. データの入手・整形
文に関する極性分析の正解データを用い，以下の要領で正解データ（sentiment.txt）を作成せよ．

1. rt-polarity.posの各行の先頭に"+1 "という文字列を追加する（極性ラベル"+1"とスペースに続けて肯定的な文の内容が続く）
2. rt-polarity.negの各行の先頭に"-1 "という文字列を追加する（極性ラベル"-1"とスペースに続けて否定的な文の内容が続く）
3. 上述1と2の内容を結合（concatenate）し，行をランダムに並び替える

sentiment.txtを作成したら，正例（肯定的な文）の数と負例（否定的な文）の数を確認せよ．

import random

def AddPlus():
    lines = []
    with open('rt-polarity.pos', 'r', encoding='cp1252') as fin:
        for line in fin:
            lines.append('+1 ' + line)
    return lines

def AddMinus():
    lines = []
    with open('rt-polarity.neg', 'r', encoding='cp1252') as fin:
        for line in fin:
            lines.append('-1 ' + line)
    return lines

def CreateText():
    lines = AddPlus() + AddMinus()
    random.shuffle(lines)
    with open('sentiment.txt', 'w') as fout:
        fout.writelines(lines)


def main():
    CreateText()

if __name__ == '__main__':
    main()

+1 a wonderful character-based comedy . 
-1 a ragbag of cliches . 
+1 gently humorous and touching . 
-1 the first fatal attraction was vile enough . do we really need the tiger beat version ? 
-1 shafer's feature doesn't offer much in terms of plot or acting . 
-1 the material and the production itself are little more than routine . 
+1 it's a beautifully accomplished lyrical meditation on a bunch of despondent and vulnerable characters living in the renown chelsea hotel . . . 
+1 captivates and shows how a skillful filmmaker can impart a message without bludgeoning the audience over the head . 

   Tweet

69. Webアプリケーションの作成
ユーザから入力された検索条件に合致するアーティストの情報を表示するWebアプリケーションを作成せよ．アーティスト名，アーティストの別名，タグ等で検索条件を指定し，アーティスト情報のリストをレーティングの高い順などで整列して表示せよ．

conda list flask

flask                     0.12                     py36_0
flask-cors                3.0.2                    py36_0

<!DOCTYPE html>
<html>
<head>
    <link href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css" rel="stylesheet">
    <title>言語処理100本ノック:No69</title>
    <style>
        body {
            margin: 20px;
        }
    </style>
<body>
    <div class="container-fluid">
    {% block content %}
    <!-- ここにメインコンテンツが挿入される  -->
    {% endblock %}
    </div>
</body>
</head>

{% extends "layout.html" %}
{% block content %}

<form method="post" class="form-horizontal">
    <div  class="form-group">
        <label class="control-label col-xs-2">名前: </label>
        <input name="name" type="text" >
    </div>
    <div  class="form-group">
        <label class="control-label col-xs-2">地域: </label>
        <input name="area" type="text" >
    </div>
    <div  class="form-group">
        <label class="control-label col-xs-2">タグ: </label>
        <input name="tag" type="text" >
    </div>
    <div  class="form-group">
        <div class="control-label col-xs-2"></div>
        <button class="btn btn-primary" type="submit">検索</button>
    </div>    
</form>
{% endblock %}

{% extends "layout.html" %}
{% block content %}
<h3>結果一覧</h3>
<p>
<a href="/">戻る</a>
</p>
<div class="table-responsive">
    <table class="table table-striped">
        <thead>
            <tr>
                <th>名前</th>
                <th>地域</th>
                <th>性別</th>
                <th>タグ</th>
                <th>別名</th>
                <th>タイプ</th>
            </tr>
        </thead>
        <tbody>
            {% for artist in artists %}
            <tr>
                <td>{{ artist.name }}</td>
                <td>{{ artist.area }}</td>
                <td>{{ artist.gender }}</td>
                <td>
                    {% for tag in artist['tags'] %}
                        {{ tag.value }}<br>
                    {% endfor %}
                </td>
                <td>
                    {% for alias in artist['aliases'] %}
                        {{ alias.name }}<br>
                    {% endfor %}
                </td>
                <td>{{ artist.type }}</td>
                </tr>
            {% else %}
                <td>書籍が登録されていません</td>
            {% endfor %}
        </tbody>
    </table>
</div>
{% endblock %}

import pymongo
from flask import Flask, render_template, request

class SearchInfo:
    def __init__(self, name, area, tag):
        self.name = name
        self.area = area
        self.tag = tag

class Searcher:
    def __init__(self):
        self.client = pymongo.MongoClient('localhost', 27017)
        self.db = self.client['MusicBrainzDb']
        self.co = self.db['artists']

    def search(self, info):
        query = { '$and' : [ ]}
        if info.name != '':
            cond = []
            cond.append({ 'aliases.name': info.name })
            cond.append({ 'name': info.name })
            query['$and'].append({ '$or': cond })
        if info.area != '':
            query['$and'].append({ 'area': info.area })
        if info.tag != '':
            query['$and'].append({ 'tags.value': info.tag })
        q2 = self.co.find(query).sort('rating.count', pymongo.DESCENDING)
        return list(q2)[:100]

app = Flask(__name__)

@app.route('/', methods=['GET', 'POST'])
def index():
    if request.method == 'POST':
        searcher = Searcher()
        results = searcher.search(SearchInfo(request.form['name'], request.form['area'], request.form['tag']))
        return render_template('result.html', artists=results) 
    else:
        return render_template('search.html') 

if __name__ == "__main__":
    app.run(debug=True)

app = Flask(__name__)

def index():

@app.route('/', methods=['GET', 'POST'])

artists=results

 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger pin code: 148-857-886

   Tweet

68. ソート
"dance"というタグを付与されたアーティストの中でレーティングの投票数が多いアーティスト・トップ10を求めよ．

import json
import pymongo

def findDance():
    client = pymongo.MongoClient('localhost', 27017)
    db = client['MusicBrainzDb']
    co = db['artists']
    return co.find({'tags.value': 'dance'}).sort('rating.count', pymongo.DESCENDING)

def main():
    for d in findDance()[0:10]:
        print(f'{} - {}'.format(d['name'], d['rating']['count']))
        #print(f'{d['name']} - {d['rating']['count']}')

if __name__ == '__main__':
    main()

co.find({'tags.value': 'dance'})

co.find({'tags.value': 'dance'}).sort('rating.count', pymongo.DESCENDING)

d['rating']['count'])

print('{} - {}'.format(d['name'], d['rating']['count']))

Madonna - 26
Bjork - 23
The Prodigy - 23
Rihanna - 15
Britney Spears - 13
Maroon 5 - 11
Adam Lambert - 7
Fatboy Slim - 7
Basement Jaxx - 6
Cornershop - 5

   Tweet

67. 複数のドキュメントの取得
特定の（指定した）別名を持つアーティストを検索せよ．

import json
import pymongo
from bson.objectid import ObjectId

def ObjectIdToStr(o):
    if isinstance(o, ObjectId):
        return str(o)
    raise TypeError(repr(o) + " is not sported")

def findAlias(alias):
    client = pymongo.MongoClient('localhost', 27017)
    db = client['MusicBrainzDb']
    co = db['artists']
    return co.find({'aliases.name': alias})

def printArtist(artist):
    print('')
    print(json.dumps(artist, indent=4, ensure_ascii=False, sort_keys=True, default=ObjectIdToStr))

def main():
    alias = input('=>')
    for d in findAlias(alias):
        printArtist(d)

if __name__ == '__main__':
    main()

ireturn co.find({'aliases.name': alias})

=>サザン

{
    "_id": "5a653c60323575566c9d9167",
    "aliases": [
        {
            "name": "Southern All Stars",
            "sort_name": "Southern All Stars"
        },
        {
            "name": "SAS .",
            "sort_name": "SAS ."
        },
        {
            "name": "サザン",
            "sort_name": "サザン"
        }
    ],
    "area": "Japan",
    "begin": {
        "year": 1975
    },
    "ended": true,
    "gid": "24d9e513-c215-4901-87ee-b66aa769ab9d",
    "id": 9549,
    "name": "サザンオールスターズ",
    "rating": {
        "count": 2,
        "value": 70
    },
    "sort_name": "Southern All Stars",
    "tags": [
        {
            "count": 1,
            "value": "japanese"
        },
        {
            "count": 1,
            "value": "asian"
        },
        {
            "count": 1,
            "value": "fixme label mess"
        }
    ],
    "type": "Group"
}