banned googlebot problem

Hangin' Around Joined: Jul 18, 2005 Posts: 33

hello, i have been having this problem for about a week now, each day i get an email from sentinel saying googlebot was blocked because it tries to access admin.php files.... this is my robots.txt file

User-agent: Mediapartners-Google*
Disallow:

User-agent: *
Disallow: /admin.php
Disallow: /admin/
Disallow: /images/
Disallow: /includes/
Disallow: /themes/
Disallow: /blocks/
Disallow: /modules/
Disallow: /language/

i dont want to add googlebot to the protected ip range because i dont think it would a good idea.

now..i dont think it's someone tryin to impersonate google's bot because the ip address is from google's corporation.

i have sentinel 2.3.2 on platinum 7.6
how can i fix this?

Posted: Mon Jul 18, 2005 2:56 pm

btw, is it bad if googlebot gets banned?
does it mean that probably it wont crawl my site in the future?

Regular Joined: Apr 17, 2005 Posts: 64

Boar thats The same problem I am having, google trying to access admin.php files ADMIN is the reason sentinel gets banned, I have the same robots file as you have, I have nuke 7.5 and sentinel 2.3.1, Maybe we could just rename the admin file, but lets see what the experts have to say... Smile

registered · Worker Joined: Oct 07, 2003 Posts: 211

I ran into this a little bit ago. Try this:

http://www.ravenphpscripts.com/posts2598.html

Hope that helps.

Posted: Mon Jul 18, 2005 6:32 pm

I will try that

registered · Posted: Mon Jul 18, 2005 6:59 pm

Google bots aren't going to hurt anything. They're stupid! All they are doing is using the same link as some script kiddie hacker wannabe, that tried to hack your site, and it got picked up through the PR system. Punish a Google bot, and you punish yourself!

Taken to its logical conclusion, let's say a Google bot set itself up as an admin on your site, and/or got into your admin panel. What's a Google bot going to do, you know? It's not like it can think for itself!

I exclude all Google bots from being banned. I recommend you do the same... Wink

Posted: Mon Jul 18, 2005 7:45 pm

Well, i dropped all the sentinel's tables and started with fresh ones, i hope that's the problem because i dont understand why we are few having this problem while others probably using the same version of nuke and same version of sentinel aren't having this problem... now.. i have to wait and see if google's bot gets banned.

VinDSL, perhaps.. but, like i said, i dont understand why others aren't having this problem :/

Google's bot is a Spider On The Nuke? hehe

Posted: Mon Jul 18, 2005 8:09 pm

VinDSL So you think I should just keep the google bot banned

Posted: Mon Jul 18, 2005 9:14 pm

sonicman. dont do that unless you DONT want google's bot to crawl your site.

ok, google's bot got banned again even with new tables.. so the problem isn't there :/

Raven wrote:

The parentheses are causing it. Nuke should also forbid that string even if Sentinel was not installed. Comment out the Sentinel include statement in mainfile.php and try it.

i should do this?

/*****************************************************/
/* Security - Sentinel v.2.3.1 START */
/*****************************************************/
/*if (defined('FORUM_ADMIN')) {
@include_once("../../../includes/nukesentinel.php");
} elseif (defined('INSIDE_MOD')) {
@include_once("../../includes/nukesentinel.php");
} else {
@include_once("includes/nukesentinel.php");
}*/
/*****************************************************/
/* Security - Sentinel v.2.3.1 END */
/*****************************************************/

if i do it, it wont do anything bad? (something else not working..)

Posted: Mon Jul 18, 2005 9:45 pm

Vin is saying protect the ips of the googlebots. (I think) LOL.

Don't Comment out sentinel in the mainfile and leave it that way. What Raven was showing is that string was being banned, and it couldn't be looked up even without sentinel because Nuke doesn't allow that string either.

Just protect the IPs of Google if you have to. Googlebots are a GOOD thing. You don't want them banned.

Posted: Mon Jul 18, 2005 10:52 pm

well, i will eventually expose my administration area to the public if i let Googlebots to crawl through my admin.php file.... I am not saying googlebots are bad, i never thought that way either, what i am trying to accomplish here is to deny access to googlesbots or any other bot to my admin.php file like other websites do using sentinel.

Posted: Tue Jul 19, 2005 1:03 am

Dauthus wrote:

Vin is saying protect the ips of the googlebots. (I think) LOL... Just protect the IPs of Google if you have to. Googlebots are a GOOD thing. You don't want them banned.

Exactly!

Posted: Tue Jul 19, 2005 1:16 am

Okay!

i wont protect googlebots IP's.....

New Member Joined: Aug 24, 2003 Posts: 11

This entry here grants this particular googlebot access to all areas on your website.

Quote:

User-agent: Mediapartners-Google*
Disallow:

I don't know if this one needs to be broken out separately, but if it does you have to list which files and directories you don't want accessed because this bot won't read the second entry.

The second entry you posted looks okay and is what the other googlebots should obey. If it's one of them triggering sentinel and you trace the IP back to their company, I recommend contacting Google and tell them you think their bot is misbehaving. They may need to adjust their software or there might be something in your robots.txt which wasn't posted that's throwing it off.

Posted: Tue Jul 19, 2005 8:51 am

ok..i guess i should search some more.

registered · Regular Joined: Feb 16, 2005 Posts: 67 Location: Canada

I had the same problem as well. The adsense I was using on the site was on everypage and even tho the robots.txt file told all bots to not access admin.php and folder, googlebot still tried to crawl the url's that I had visited while logged in as admin. I mean everytime, login as admin and goto say modules admin, then log out and bamn, googlebot banned for trying to access the exact page. So eventually after trying many things to prevent googlebot (any any other crawler) from accessing any admin url's I went extreme. I placed a allow deny rule in .htaccess to block any ip's except for ones that I list. Yes people might say this is a bit radical, but for me, it is worth it, even tho I have to change the ip's once and a while.

Code:

<Files "admin.php">


  Order Deny,Allow


  Deny from all


  Allow from xx.xx.xx.xx


  Allow from xx.xx.xxx.xxx


</Files>

Posted: Tue Jul 19, 2005 7:04 pm

Dauthus
we discussed this on your site, all I would have to do is add the google bots ips in the protected ip range right

Posted: Tue Jul 19, 2005 7:41 pm

sonicman wrote:

Dauthus
we discussed this on your site, all I would have to do is add the google bots ips in the protected ip range right

Yep.

Posted: Tue Jul 19, 2005 8:10 pm

This is my theory...

When Google bots spider your site, under normal conditions, I believe they follow the rules. However, many ppl, including admins, hackers, et cetera, are running Google Toolbar in their browser, with PR (PageRank) enabled. When you go to your admin page, a hacker tries to run an exploit against your site, or whatever, and PR is enabled, Google sends a bot to check out the URI. It doesn't matter if it's you accessing your admin page, or a hacker running a UNION exploit, Google (PR) bots will blindly pick up the link and run with it.

The point I was trying to make, above - and I guess I didn't state it clearly enough - even if a Google bot accesses your admin panel, what's it going to do? Answer: sit there... Laughing

Personally, I think it's a bad form to ban Google bots, even if they're acting silly - accessing your admin panel, running exploits against you, or whatever - but that's just me. I've never had a spider crack my site yet, knock on wood...

Posted: Tue Jul 19, 2005 8:57 pm

vinDSL
Cool I will just add the google bots IP address range in sentinel

Thanks

Posted: Wed Jul 20, 2005 2:49 am

VinDSL, i am not using any type of third-parties toolbars or anything other than a clean browser. Now, my site is still under consctruction, no one knows about it besides few ppl.

I have emailed google, let's see what they have to say about it.

Everyone with Platinum 7.6 (if that matters) and Sentinel having problems with googlebot please raise your hand! only few ppl? mm how come others dont have this problem? :/

for now, i have set the Admin blocker to Email and Redirect everyone to the main site...i havent had any new emails YET!. i hope they didnt removed google's bot from crawling my site, i hope it's temporary if they did.

Posted: Thu Jul 21, 2005 10:59 am

well, googlebot isnt crawling my site anymore thanks to NukeSentinel!.

i will stop complaining now.

registered · Regular Joined: May 28, 2005 Posts: 90

i have another question: why rename admin.php while in robots.txt the new name is visible to the world? :/

Posted: Thu Jul 21, 2005 11:48 am

i was thinking the same exact thing for a couple of months now.. i havent asked that the question... i guess few people know about robots.txt file? i know..