picsearch bandwidth hog

registered · Regular Joined: Feb 16, 2005 Posts: 67 Location: Canada

Hello all, I havent posted much lately but I recently came back from holidays and found a huge drain on my site from a robot called picsearch.com. This site's robots used up over 1gig of bandwidth searching my amazon module for.. i guess pics..lol in one month!! Well any site that does not limit how much they pull from me is not a robot that I want.
I have added a entry to my .htacces to deny any hits from picsearch.com.. but my question is this.. How can i add this site and all it's robots to my .htaccess without doing a general entry like this:

Code:

deny from picsearch.com

What I am worried about is that apache will have to try and resolve to the domain name for every visitor to the site, and from what i understand is that this slows down apache. I can find an ip but will this block all spiders from picsearch?.. eg. spider1.picsearch.com, spider2.picsearch.com etc

Thanks in advance, oh and Raven if this is in the wrong place please move, thanks again.

Site Admin/Owner Joined: Aug 27, 2002 Posts: 17088

wrote:

Information for domain picsearch.com
IP: 217.212.245.198 [get RIPE/ARIN IP info] (Resolved: www.picsearch.com)
Reverse IP: there are 2 domains on this IP, click here to get them all
Name Server: dns5.telia.com Found 966 domains, hosted on this NS
ns3.picsearch.com Found 2 domains, hosted on this NS
(click to any NS server to get all domains, hosted on this NS)
IP Location: Sweden [SE] - Stockholm
IP owner: TeliaSonera AB
IP assigned to: Picsearch AB
Domain status: REGISTRAR-LOCK
Domain Registrar: NETWORK SOLUTIONS, LLC.
Created: 15-mar-1999
Expires: 15-mar-2009

Here are the IP's/CIDR assigned to that domain name: 217.212.224.0/19

I would add deny from 217.212.224.0/19 instead

Posted: Fri Jan 27, 2006 5:33 pm

Add PicSpider/1.1 to your harvester blocker and write to htaccess.
This is only one 1 IP:inetnum: 217.212.224.128 - 217.212.224.255
netname: SE-PICSEARCH
More info: http://www.picsearch.com/bot.html

Posted: Sat Jan 28, 2006 12:21 am

thank you both.. Smile

Posted: Sat Jan 28, 2006 7:23 am

Raven wrote:

Here are the IP's/CIDR assigned to that domain name: 217.212.224.0/19

Incorrect ban because that bans the whole network

Susann wrote:

217.212.224.128 - 217.212.224.255

So the correct CIDR should be: 217.212.224.128/25

Posted: Sat Jan 28, 2006 10:27 am

I did that intentionally, DJ, because their Lookup revealed that they could use any of the band, but it's his choice.

Posted: Sat Jan 28, 2006 11:30 am

I understand but if you lookup ripe http://www.ripe.net/whois?searchtext=217.212.224.0 that is owned by TradeDoubler AB

If you check the /19 CIDR you will notice you ban a lot more then the range specified.
The CIDR specified by RIPE is meant as being anything in sweden aka Top-Level

217.212.224.0/19 = 217.212.224.0 - 217.212.255.255

As you see /19 is the wrong here.

Here's some code you can use to check it yourself Wink

Code:

<?php


$ip = '217.212.224.0';


$cidr = 19;





$ip = ip2long($ip);


$mask = 0xffffffff << (32 - $cidr);


echo "$ip - ".long2ip($ip - $mask);

Posted: Sat Jan 28, 2006 12:19 pm

DJ, I completely understand how to calculate CIDR's, but thanks. Yours is the more precise/correct CIDR to use. When I started looking up the 900+ domains in their name servers, I just went for overkill. Use the /25 as DJ has mentioned.

Posted: Sun Jan 29, 2006 8:01 am

Thanks Djmaze
For those who have problems to understand the cidr´ s I found a helpful site with examples and some pictures for better understanding.

http://www.freesoft.org/CIE/Course/Subnet/5.htm

Posted: Sun Jan 29, 2006 8:19 am

Here's some more info Wink

http://en.wikipedia.org/wiki/IPv4_subnetting_reference
http://en.wikipedia.org/wiki/Classless_inter-domain_routing