Python網絡數據采集 (影印版)第2版
作 者: (美)瑞安·米切爾(Ryan Mitchell) 著
定 價: 89
出?版?社: 東南大學出版社
出版日期: 2018年11月01日
頁 數: 288
裝 幀: 平裝
ISBN: 9787564179779
●Preface
Part Ⅰ.Building Scrapers
1.Your First Web Scraper
Connecting
An Introduction to BeautifulSoup
Installing BeautifulSoup
Running BeautifulSoup
Connecting Reliably and Handling Exceptions
2.Advanced HTML Parsing
You Don't Always Need a Hammer
Another Serving of BeautifulSoup
findo and findallo with BeautifulSoup
Other BeautifulSoup Objects
Navigating Trees
Regular Expressions
Regular Expressions and BeautifulSoup
Accessing Attributes
Lambda Expressions
3.Writing Web Crawlers
Traversing a Single Domain
Crawling an Entire Site
Collecting Data Across an Entire Site
Crawling Across the Internet
4.Web Crawling Models
Planning and Defining Objects
Dealing with Different Website Layouts
Structuring Crawlers
Crawling Sites Through Search
Crawling Sites Through Links
Crawling Multiple Page Types
Thinking About Web Crawler Models
5.Scrapy
Installing Scrapy
Initializing a New Spider
Writing a Simple Scraper
Spidering with Rules
Creating Items
Outputting Items
The Item Pipeline
Logging with Scrapy
More Resources
6.St0ring Data
Media Files
Storing Data to CSV
MySQL
Installing MySQL
Some Basic Commands
Integrating with Python
Database Techniques and Good Practice
"Six Degrees" in MySQL
Email
Part Ⅱ.Advanced Scraping
7.Reading Documents
Document Encoding
Text
Text Encoding and the Global Internet
CSV
Reading CSV Files
PDF
Microsoft Word and .docx
8.Cleaning Your Dirty Data
Cleaning in Code
Data Normalization
Cleaning After the Fact
OpenRefine
9.Reading and Writing Natural Languages
Summarizing Data
Markov Models
Six Degrees of Wikipedia:Conclusion
Natural Language Toolkit
Installation and Setup
Statistical Analysis with NLTK
Lexicographical Analysis with NLTK
Additional Resources
10.Crawling Through Forms and Logins
Python Requests Library
Submitting a Basic Form
Radio Buttons,Checkboxes,and Other Inputs
Submitting Files and Images
Handling Logins and Cookies
HTTP Basic Access Authentication
Other Form Problems
11.Scraping JavaScript
A Brief Introduction to JavaScript
Common JavaScript Libraries
Ajax and Dynamic HTML
Executing JavaScript in Python with Selenium
Additional Selenium Webdrivers
Handling Redirects
A Final Note on JavaScript
12.Crawling Through APIs
A Brief Introduction to APIs
HTTP Methods and APIs
More About API Responses
Parsing JSON
Undocumented APIs
Finding Undocumented APIs
Documenting Undocumented APIs
Finding and Documenting APIs Automatically
Combining APIs with Other Data Sources
More About APIs
13.Image Processing and Text Recognition
Overview of Libraries
Pillow
Tesseract
NumPy
Processing Well-Formatted Text
Adjusting Images Automatically
Scraping Text from Images on Websites
Reading CAPTCHAs and Training Tesseract
Training Tesseract
Retrieving CAPTCHAs and Submitting Solutions
14.Avoiding Scraping Traps
A Note on Ethics
Looking Like a Human
Adjust Your Headers
Handling Cookies with JavaScript
Timing Is Everything
Common Form Security Features
Hidden Input Field Values
Avoiding Honeypots
The Human Checklist
15.Testing Your Website with Scrapers
An Introduction to Testing
What Are Unit Tests?
Python unittest
Testing Wikipedia
Testing with Selenium
Interacting with the Site
unittest or Selenium?
16.Web Crawling in Parallel
Processes versus Threads
Multithreaded Crawling
Race Conditions and Queues
The threading Module
Multiprocess Crawling
Multiprocess Crawling
Communicating Between Processes
Multiprocess Crawling--Another Approach
17.Scraping Rem0tely
Why Use Remote Servers?
Avoiding IP Address Blocking
Portability and Extensibility
Tor
PySocks
Remote Hosting
Running from a Website-Hosting Account
Running from the Cloud
Additional Resources
18.The Legalities and Ethics of Web Scraping
Trademarks,Copyrights,Patents,Oh My!
Copyright Law
Trespass to Chattels
The Computer Fraud and Abuse Act
robots.txt and Terms of Service
Three Web Scrapers
eBay versus Bidder's Edge and Trespass to Chattels
United States v.Auernheimer and The Computer Fraud and Abuse Act
Field v.Google:Copyright and robots.txt
Moving Forward
Index
內容簡介
如果編程是魔法,那麼網絡數據采集肯定就是某種巫術。編寫一個簡單的自動化程序,你就可以查詢Web服務器,請求數據,解析數據以提取所需的信息。這本實用書籍的擴充版不但介紹了網絡數據采集,更是從現代網絡中抓取幾乎各類數據的綜合指南。《Python網絡數據采集(第2版·影印版·英文版)》部分側重於網絡數據采集機制:使用Python向Web服務器請求信息,對服務器響應信息做基本的處理,自動與站點展開交互。第二部分探討了各種更具體的工具和應用程序,以應對你可能遇到的任何網絡數據采集場景。
(美)瑞安·米切爾(Ryan Mitchell) 著
瑞安·米切爾,位於波士頓的HedgeServ的不錯軟件工程師,負責開發公司的API和數據分析工具。她畢業於歐林工程學院,擁有哈佛大學擴展學院(Harvard Urliversity Exterlsion School)軟件工程碩士學位以及數據科學證書。在加入HedgeServ之前,她曾就職於Abine,負責使用Python開發網絡數據采集工具和自動化工具。她經常從事零售、金融和制藥行業的網絡數據采集項目的咨詢工作,還曾經在東北大學和歐林工程學院擔任課程顧問和兼職教員。
"