当前位置:博客首页 > Python > 正文

python如何建立属于自己的可用代理IP池

作者: Jarvan 分类: Python 发布时间: 2019-02-28 08:54 百度已收录

一、建立属于自己的代理IP池
[cce]
import requests
from bs4 import BeautifulSoup
import random

ip_list = []

def get_ip_list(url):
  headers = {
  ‘User-Agent’: ‘Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36’
  }
  res = requests.get(url, headers=headers)
  res = BeautifulSoup(res.text, ‘html.parser’)
  results = res.select(‘#ip_list tr’)
  for result in results[1:]:
  ip = result.select(‘td’)[1].text
  port = result.select(‘td’)[2].text
  judge(ip, port)

def judge(ip, port):
  proxy = {‘http’: ip+’:’+port}
  try:
  res = requests.get(‘https://www.baidu.com’, proxies=proxy)
  except Exception:
  print(‘该ip:’ + ip + ‘无效’)
  return False
  else:
  if 200 <= res.status_code < 300:
  ip_list.append((ip, port))
  return True
  else:
  print(‘该ip:’ + ip + ‘无效’)
  return False

def get_random_ip():
  ip, port = random.choice(ip_list)
  result = judge(ip, port)
  if result:
  return ip + ‘:’ + port
  else:
  ip_list.remove((ip, port))

if __name__ == “__main__”:
  get_ip_list(“https://www.xicidaili.com/wt/”)
  print(ip_list)
[/cce]

二、成果:获取的可用代理IP

发表评论