Air9

And it's turtles all the way down...

⇝

Air9

And it's turtles all the way down...

Spider-04-sqlite3

2016-04-05

知道创宇爬虫设计第四天:sqlite3

这部分比较简单,需要注意的几点

  • connect, cursor
  • execute
  • commit, rollback
  • close

既然之前已经学会了使用logging,就可以尝试下配合着使用

代码

MySqlite.py
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import sqlite3
import logging
import logging.config
logging.config.fileConfig('logging.conf')
levels = {
1: 'CRITICAL',
2: 'ERROR',
3: 'WARNING',
4: 'INFO',
5: 'DEBUG',
}
loglevel = 4
logger = logging.getLogger('spider')
logger.setLevel(levels[loglevel])
class MySqlite(object):
def __init__(self, dbfile):
try:
self.conn = sqlite3.connect(dbfile)
logger.info("Open database %s" % dbfile)
logger.debug("Open database %s" % dbfile)
except sqlite3.Error as e:
# print "Fail to connect %s: %s" % (dbfile, e) # e.args[0]
logger.error("Fail to connect %s: %s" % (dbfile, e))
return
self.cursor = self.conn.cursor()
def create(self, table):
try:
logger.info("Create table %s" % table)
self.cursor.execute(
"CREATE TABLE IF NOT EXISTS %s(Id INTEGER PRIMARY KEY \
AUTOINCREMENT, Data VARCHAR(40))" % table
)
self.conn.commit()
except sqlite3.Error as e:
logger.error("Fail to create %s: %s" % (table, e))
self.conn.rollback()
def insert(self, table, data):
try:
logger.info("Insert %s into table %s" % (data, table))
self.cursor.execute(
"INSERT INTO %s(Data) VALUES('%s')" % (table, data))
self.conn.commit()
except sqlite3.Error as e:
logger.error("Fail to insert %s into %s: %s" % (data, table, e))
self.conn.rollback()
def close(self):
logger.info("Close database")
self.cursor.close()
self.conn.close()
if __name__ == '__main__':
ms = MySqlite('spider.db')
ms.create('t1')
ms.insert('t1', 'test')
ms.close()
logging.conf
[loggers]
keys = root, spider
[handlers]
keys = consoleHandler, fileHandler
[formatters]
keys = simpleFormatter
[logger_root]
level = DEBUG
handlers = consoleHandler
[logger_spider]
level = DEBUG
handlers = consoleHandler, fileHandler
qualname = spider
propagate = 0
[handler_consoleHandler]
class = StreamHandler
level = DEBUG
formatter = simpleFormatter
args = (sys.stdout,)
[handler_fileHandler]
class = FileHandler
level = DEBUG
formatter = simpleFormatter
args = ('spider.log', 'w')
[formatter_simpleFormatter]
format = %(asctime)s - %(name)s - %(levelname)s - %(message)s
datefmt =

参考

  • https://docs.python.org/2/library/sqlite3.html
  • http://blog.sina.com.cn/s/blog_72603eac01013pbc.html
  • http://blog.csdn.net/jeepxiaozi/article/details/8808435
  • http://dongweiming.github.io/blog/archives/pa-chong-lian-xi/
  • http://devdocs.io/python~2.7/library/logging.config
  • Python
  • Spider
  • sqlite
  • Python

扫一扫,分享到微信

微信分享二维码
Sublime Preferences
Spider-03-argparse
© 2018 Air9
Hexo Theme Yilia by Litten
  • ⇝

tag:

  • Privoxy
  • Shadowsocks
  • gfw
  • Coursera
  • JavaScript
  • AngularJS
  • ngResource
  • Python
  • Flask-SQLAlchemy
  • Alembic
  • Flask-Migrate
  • Flask
  • relationship
  • backref
  • Linux
  • x86
  • elf
  • main
  • libc
  • start
  • init
  • gdb
  • sorted
  • key
  • Closure
  • Decorator
  • func_defaults
  • Q
  • asm
  • align
  • capstone
  • ida
  • Spider
  • ThreadPool
  • logging
  • argparse
  • sqlite
  • requests
  • doctest
  • SublimeREPL
  • ExpandRegion
  • Anaconda
  • MarkdownEditing
  • Archlinux
  • pip
  • vmtools
  • bootstrap
  • keyboard
  • tagIndex
  • Brackets
  • gulp
  • browser-sync
  • python
  • ropgadget
  • Refactoring
  • git
  • github
  • ssh
  • hexo
  • rss
  • sitemap
  • idapython
  • idc
  • ionic
  • rop
  • keystone
  • unicorn
  • Mac
  • mongodb
  • MySQL
  • UTF-8
  • markdown
  • evernote
  • sublime
  • vscode
  • proxy
  • iTerm
  • polipo
  • Charles
  • 科学上网
  • Powerline
  • cookbook
  • closure
  • factory
  • string
  • python-magic
  • mime
  • apt-get
  • raspi
  • ANSI
  • termcolor
  • colorama
  • redis
  • brew
  • tmux
  • 字符集
  • 编码
  • ASCII
  • Unicode

    缺失模块。
    1、请确保node版本大于6.2
    2、在博客根目录(注意不是yilia根目录)执行以下命令:
    npm i hexo-generator-json-content --save

    3、在根目录_config.yml里添加配置:

      jsonContent:
        meta: false
        pages: false
        posts:
          title: true
          date: true
          path: true
          text: false
          raw: false
          content: false
          slug: false
          updated: false
          comments: false
          link: false
          permalink: false
          excerpt: false
          categories: false
          tags: true
    

  • 树莓派初体验

    2018-05-19

    #raspi

  • Markdown 同步 Evernote

    2017-06-01

    #markdown#evernote#sublime#vscode

  • polipo+ss实现 iTerm2 终端科学上网

    2017-04-03

    #Shadowsocks#proxy#iTerm#polipo#Charles#科学上网

  • Linux x86 elf 程序启动过程

    2017-03-16

    #Linux#x86#elf#main#libc#start#init#gdb

  • keystone

    2016-11-30

    #capstone#python#rop#keystone#unicorn

  • Python 控制终端输出颜色 & ANSI 控制码

    2016-10-12

    #Python#ANSI#termcolor#colorama

  • python-magic has no attribute 'from_file'

    2016-10-11

    #Python#pip#python-magic#mime#apt-get

  • idapython-elf-gadget

    2016-10-08

    #ida#python#idapython

  • ionic 使用 popover 时的一点小问题

    2016-09-27

    #JavaScript#AngularJS#ngResource#ionic

  • Code Smell

    2016-09-25

    #Refactoring

  • Mac 安装 mongodb

    2016-09-22

    #Mac#mongodb

  • 命令行调用 idapython 脚本

    2016-09-20

    #ida#python#idapython#idc

  • Mac 下使用 homebrew 安装 redis

    2016-09-12

    #Mac#redis#brew

  • Q 判断 Gadget 时遇到 align 伪指令的问题

    2016-08-23

    #Q#asm#align#capstone#ida

  • Flask-SQLAlchemy 中的 relationship & backref

    2016-08-16

    #Flask-SQLAlchemy#Flask#relationship#backref

  • hexo-rss-sitemap

    2016-07-28

    #hexo#rss#sitemap

  • Python 中的闭包与装饰器

    2016-07-25

    #Python#Closure#Decorator#func_defaults

  • 字符集与编码

    2016-07-06

    #Python#UTF-8#ANSI#字符集#编码#ASCII#Unicode

  • AngularJS-ngResource-save

    2016-07-06

    #Coursera#JavaScript#AngularJS#ngResource

  • brackets-live-preview-gulp

    2016-07-05

    #Coursera#JavaScript#AngularJS#Brackets#gulp#browser-sync

  • bootstrap-keyboard-tagIndex

    2016-06-28

    #Coursera#JavaScript#bootstrap#keyboard#tagIndex

  • Permission denied (publickey)

    2016-06-16

    #git#github#ssh

  • Python-Cookbook-1.09 简化 translate 方法

    2016-05-25

    #Python#cookbook#closure#factory#string

  • Python-sorted函数中key的用法

    2016-05-19

    #Python#sorted#key

  • Flask 数据库更新问题

    2016-05-16

    #Python#Flask-SQLAlchemy#Alembic#Flask-Migrate

  • Spider-07-doctest

    2016-04-08

    #Python#Spider#doctest

  • Mac 安装 MySQL 并设置 utf-8

    2016-04-07

    #MySQL#UTF-8

  • Spider-06-requests

    2016-04-07

    #Python#Spider#requests

  • Spider-05-Spider

    2016-04-06

    #Python#Spider

  • Sublime Preferences

    2016-04-05

    #SublimeREPL#ExpandRegion#Anaconda#MarkdownEditing

  • Spider-04-sqlite3

    2016-04-05

    #Python#Spider#sqlite

  • Spider-03-argparse

    2016-04-04

    #Python#Spider#argparse

  • Spider-02-logging

    2016-04-03

    #Python#Spider#logging

  • Spider-01-MyThreadPool

    2016-04-02

    #Python#Spider#ThreadPool

  • ropgadget capstone

    2016-03-10

    #capstone#python#ropgadget

  • hexo github

    2016-03-09

    #git#github#hexo

  • 10.11 privoxy [Errno 61] Connection refused

    2016-03-09

    #Privoxy#Shadowsocks#gfw

  • arch vmtools

    2016-03-09

    #Archlinux#vmtools

  • arch gdb pip ropgadget

    2016-03-09

    #gdb#Archlinux#pip

  • tmux copy2clipboard

    2016-03-09

    #Archlinux#tmux

  • iTerm下配置Powerline

    2016-03-09

    #iTerm#Powerline