Advanced Crawling: IP Proxies

Background

It was exam-score season again, and this time the teaching assistant tried something different: by sending a request to the server, the score would be printed directly on the page. However, the server recorded the visiting IP address and limited how many times each IP could query scores per day.

So is there a way to break through this limit? The answer is the famous “IP proxy.”

IP proxies are actually very common. The “ladders” people use in daily life also work through IP proxies: access through overseas IP addresses and then browse indirectly through a proxy.

A more powerful use case for IP proxies is web crawling. Through a proxy, you can achieve highly anonymous access and easily break through a website’s IP blocking.

This post records the hands-on process so I can refer back to it later.

Usage: attach the proxies= parameter to a requests call, and put the obtained proxy information into it.

You can visit specific websites, such as http://icanhazip.com/, to check whether the current IP proxy is working.

Proxy format:

proxy = {
  'http': ip:port
  'https': ip:port
}

Obtaining an IP

It is actually simple: obtain the house number (IP address) plus identity/key (authKey/authSecret).

Because domestic IP pools are very cheap, sometimes registration even comes with free credits at the cost of leaking personal information, some websites such as https://www.juliangip.com/ provide 1000 free IPs every day. For low-intensity proxy crawling, that is already enough.

The exact way to obtain IPs is generally through the interface provided by the website. For that part, RTFM.

Example Code

import time
import re
import requests
from requests.auth import HTTPProxyAuth

headers = {
  'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.193 Safari/537.36'
}

url = 'http://101.42.221.61:9090?id='  # Final target URL

ip = ''  # IP extracted from the IP pool

proxies = {
  'http': ip,
  'https': ip
}

auth = HTTPProxyAuth("USERNAME", "USERPWD")  # Account and password for IP authentication

# Request a new IP from the proxy provider
def getIP():
  getIPUrl = "http://v2.api.juliangip.com/dynamic/getips?area=%E6%B1%9F%E8%8B%8F,%E5%8D%97%E4%BA%AC&filter=1&isp=%E7%94%B5%E4%BF%A1&num=1&pt=1&result_type=text&split=1&trade_no=1152845013074502&sign=e30c961c58d3541005b58f5d70967497"
  newIp = requests.get(url=getIPUrl, auth=auth).text
  print("newIP: " + newIp)
  return newIp

# Update the proxy
def updateProxy():
  newIP = getIP()
  proxies['http'] = newIP
  proxies['https'] = newIP

nameList = [211830066,
          211850009,
          215220019,
          215220010,
          215220008,
          215220005,
          215220002,
          211870171,
          211840154,
          211840112,
          211503026,
          211200015,
          211180111,
          211098324,
          205220026,
          201830099,
          201810069,
          191870226,
          191870072,
          191830074,
          191220048,
          191180008,
          211870298,
          215220008,
          215220002,
          211870202,
          211870180,
          211840245,
          211820282,
          211220167,
          211108100,
          205220011,
          201840100,
          191180123
          ]

ansList = {}

for name in nameList:
  updateProxy()
  curUrl = url + str(name)
  print(curUrl)
  # print(proxies['http'])
  res = requests.get(headers=headers, url=curUrl, proxies=proxies, verify=False).text
  curIPis = requests.get(url="http://icanhazip.com/", proxies=proxies).text # Verify whether the current proxy IP is working.
  print(curIPis)
  print(res)
  pts = re.findall(r"[0-9]+", res)
  if len(pts) != 0 and pts[0] != 58:
      ansList[name] = pts[0]
      print(str(name) + " " + str(pts[0]))
  time.sleep(1)

with open("./res.txt", 'w', encoding='utf-8') as fp:
  for x in ansList:
      string = str(x) + " " + str(ansList[x])
      fp.write(string + '\n')
print('finished!')

Screenshot of the final crawler result: