Google Search

rustyblade · Post by **rustyblade** » Wed Nov 08, 2006 8:48 pm

Why does SMF not come up when you google search specific shaving terms that you know exist here? Entries from B&B always come up on google.

If SMF did show up, we would get more new users perhaps.

Anybody technical know why SMF content doesn't get cataloged by google?

It is a shame to have all this content here that does not seem to be available to the outside world.

With The Grain · Post by **With The Grain** » Wed Nov 08, 2006 8:56 pm

if you search "how to shave" neither appear on the first page, it would be great if we could make SMF a sponsored result for that but probably costly

rustyblade · Post by **rustyblade** » Wed Nov 08, 2006 9:03 pm

With The Grain wrote:if you search "how to shave" neither appear on the first page, it would be great if we could make SMF a sponsored result for that but probably costly

Not just that, but the content here will not show up, whereas B&B will.

For example, something that I can remember is my sticky razors. I searched "The Wilkinson Sword Razor"

SMF doesn't show up. That text exists here also. There are many instances that anyone could be searching for shave stuff or a better way and all that valuable info here is not showing on google. Yet B&B does.

JohnP · Post by **JohnP** » Wed Nov 08, 2006 9:06 pm

Just forwarded this up to the admins,
if anyone would know Johnnie or Chris should be able to answer this better than I.
John P.

Sue · Post by **Sue** » Wed Nov 08, 2006 9:14 pm

The Administrator can set up an account (if desired ~ there is a fee) with Google "Adwords". Choose 'key words', i.e. 'straight razor', wet-shaving, 'strop', etc. Those key words will trigger SMF to appear. There are many vaiables that would be up to the orner or owner/administrator. There may also be other ways to do so but this one works.
Sue

ScottS · Post by **ScottS** » Thu Nov 09, 2006 6:20 am

We don't seem to be hurting for participation. As more and more folk link to here, our google hits will go up. We don't sell anything, or accept advertising, so our hit rate doesn't really matter. My experience with paying google is that if people don't click through, your ad falls off the bottom anyway. I just don't think its worth paying for.

rustyblade · Post by **rustyblade** » Thu Nov 09, 2006 6:29 am

I'm not talking about the ads, that doesn't matter. Our content isn't showing up in the regular search listings. I have noticed this for a while now when I am searching for a specific shave soap, for example. B&B content will show up whereas old and newer SMF content doesn't (and we have a lot of valuable content here).

No, we are not hurting, but I am positive we would get more new users and that can't be a bad thing.

I'm just wondering why this is...?

MindSpin · Post by **MindSpin** » Sat Nov 11, 2006 10:54 pm

Inside some of the HTML documents you can put in META keywords that describe the web page and it's contents. You can aldo put in keywords that get logged by search "spiders" that seek all this stuff out. Then you can submit it to a search site, Google for example offers this for free, but offers no guarantees. You can also subsribe and pay for that too.
I've done this for my wife's decorating website. It worked pretty good. And I believe it has a bit of a snowball affect. The more times it is found, the more likely it is to be found as time goes on.

I just took a look at this site's META contents (right click on the page and "View Source"), and there is very little in the way of META content.

Dave

timc · Post by **timc** » Sun Nov 12, 2006 9:17 am

rustyblade wrote:
With The Grain wrote:if you search "how to shave" neither appear on the first page, it would be great if we could make SMF a sponsored result for that but probably costly
Not just that, but the content here will not show up, whereas B&B will.

For example, something that I can remember is my sticky razors. I searched "The Wilkinson Sword Razor"

SMF doesn't show up. That text exists here also. There are many instances that anyone could be searching for shave stuff or a better way and all that valuable info here is not showing on google. Yet B&B does.

That search doesn't even return any results if you add "site:shavemyface.com" on the end of the search term. It seems as if that page isn't even being indexed.

rustyblade · Post by **rustyblade** » Sun Nov 12, 2006 9:22 am

timc wrote:
rustyblade wrote:
With The Grain wrote:if you search "how to shave" neither appear on the first page, it would be great if we could make SMF a sponsored result for that but probably costly
Not just that, but the content here will not show up, whereas B&B will.

For example, something that I can remember is my sticky razors. I searched "The Wilkinson Sword Razor"

SMF doesn't show up. That text exists here also. There are many instances that anyone could be searching for shave stuff or a better way and all that valuable info here is not showing on google. Yet B&B does.
That search doesn't even return any results if you add "site:shavemyface.com" on the end of the search term. It seems as if that page isn't even being indexed.

None of the text within the forum is ending up on google. It is bothering me and I don't know why. I added the site to to google manually a few days ago, but I think there is some way the database here is setup that google is not indexing it.

It is stopping many people from finding this site. They are finding B&B instead (which has had a large influx of new users). This could be good/bad depending on how you feel about new users, but it can't be a bad thing for the future.

Gatorade · Post by **Gatorade** » Sun Nov 12, 2006 10:31 am

I remember googlebot stopping in a few times a while back. I assume it was to archive the site. Shouldn't that pick up the text at the same time?

yoyology · Post by **yoyology** » Sun Nov 12, 2006 1:20 pm

I am by no means a web technology expert, but I do teach classes on searching, and it may be that SMF is falling into the memory hole that's called "The Invisible Web".

If the back-end software for the forum generates static pages that are modified each time there's a new post, then bots like Googlebot are able to index each page in the forum.

However, if pages are generated dynamically, on the fly each time (for instance, by concatenating individual replies archived in a database) then Google has no way to find the information, because the page literally doesn't exist until it's created by a click on a link.

The software appears to be called phpBB. Since php is a database system, I'm guessing that dynamic pages are the problem. Badger and Blade runs on vBulletin.

Of course, it could be that there's some other issue (bots may be disallowed by default in the software, the admins may have chosen not to have the forums indexed, etc.) but this would be the first thing I'd look at.

Here's a thread on the phpBB forums that might give the admins some insight.

timc · Post by **timc** » Sun Nov 12, 2006 1:38 pm

yoyology wrote: Of course, it could be that there's some other issue (bots may be disallowed by default in the software, the admins may have chosen not to have the forums indexed, etc.) but this would be the first thing I'd look at.

Actually, the first thing to check would be the robots.txt file http://www.shavemyface.com/robots.txt

And here it is:

# robots.txt for http://www.shavemyface.com

User-agent: *
Disallow: /forum
Disallow: /gallery
Disallow: /personal

So, the robots.txt file tells search engines to not index anything in "/forum", which is why forum posts aren't showing up. At least, that seems to be the best explanation that I have.

rustyblade · Post by **rustyblade** » Sun Nov 12, 2006 3:59 pm

Tim, that explains it. Nice bit of detective work, thanks.

Now, I can see the reason for this if it is indeed deliberate. I said earlier that B&B is getting more new users, but this is not necessarily always desirable. It has been known for a while that more 'experts' reside on SMF. All I'm going to say is B&B has a much different "flavour" in that regard and the serious shavegeek information is retained over here. This is not a knock on B&B, just a conclusion I came to a while ago.

If you need serious info on razors, brushes, soaps/creams, and technique it mostly happens here.

Post by **VinceFX** » Mon Nov 13, 2006 3:46 pm

HAHAHAH BUSTED! Yes, it's true. I put this file in place to prevent Google from indexing the forum content. Why? Because Google was sucking up all my bandwidth by downloading every damn page on our server.

Personally, it doesn't really bother me if we're ranked kinda low on Google. Errrr...should I be concerned? Hooray for word of mouth! LOL

Vince

timc wrote:
yoyology wrote: Of course, it could be that there's some other issue (bots may be disallowed by default in the software, the admins may have chosen not to have the forums indexed, etc.) but this would be the first thing I'd look at.
Actually, the first thing to check would be the robots.txt file http://www.shavemyface.com/robots.txt

And here it is:

# robots.txt for http://www.shavemyface.com

User-agent: *
Disallow: /forum
Disallow: /gallery
Disallow: /personal
So, the robots.txt file tells search engines to not index anything in "/forum", which is why forum posts aren't showing up. At least, that seems to be the best explanation that I have.

Post by **VinceFX** » Mon Nov 13, 2006 3:52 pm

oh god....you guys got me to have second thoughts about this...

I went ahead and removed the disallow flags from the robots.txt file.

I hope my server doesn't blow up now! LOL

Vince

Post by **VinceFX** » Mon Nov 13, 2006 3:58 pm

rustyblade wrote:Tim, that explains it. Nice bit of detective work, thanks.

Now, I can see the reason for this if it is indeed deliberate. I said earlier that B&B is getting more new users, but this is not necessarily always desirable. It has been known for a while that more 'experts' reside on SMF. All I'm going to say is B&B has a much different "flavour" in that regard and the serious shavegeek information is retained over here. This is not a knock on B&B, just a conclusion I came to a while ago.

If you need serious info on razors, brushes, soaps/creams, and technique it mostly happens here.

Yea, I'm sure that by limiting our exposure we have also kept our growth down. I would theorize that this lends itself to a forum that is easier to manage and therefore higher in quality. Yes, that's right. We are the best!

fallingwickets · Post by **fallingwickets** » Mon Nov 13, 2006 5:15 pm

rustyblade wrote:Tim, that explains it. Nice bit of detective work, thanks.

Now, I can see the reason for this if it is indeed deliberate. I said earlier that B&B is getting more new users, but this is not necessarily always desirable. It has been known for a while that more 'experts' reside on SMF. All I'm going to say is B&B has a much different "flavour" in that regard and the serious shavegeek information is retained over here. This is not a knock on B&B, just a conclusion I came to a while ago.

If you need serious info on razors, brushes, soaps/creams, and technique it mostly happens here.

I was intent on finding out how to take care of my new razor and went page after page through the google results to my query (I guess I could take a lesson or two on searching google....lol) and somehow by sheer luck I guess came to SMF. All this to say that if you are serious avout shaving, you'll get to niverna eventually!

rustyblade · Post by **rustyblade** » Mon Nov 13, 2006 5:54 pm

Vince, it will be an interesting experiment if nothing else.

ScottS · Post by **ScottS** » Mon Nov 13, 2006 5:55 pm

VinceFX wrote:oh god....you guys got me to have second thoughts about this...

I went ahead and removed the disallow flags from the robots.txt file.

I hope my server doesn't blow up now! LOL

Vince

By all means, keep an eye on your bandwidth!