Robot concerns

Welcome to the New Sony Alpha Shooters Community
Our community was only launched in August 2020 but is already thriving thanks to many amazing and helpful members. We'd love to welcome you on board too!
Join Now

garuda

Well-known Member
Original poster
Pro Member
Pro Member
Followers
4
Following
0
286
186
Sep 14, 2020
Name
Mike
City/State
Illinois
Tim, should there be concern with these numerous, repeated robot queries? Do we know what they are extracting? See pic below:

'
Robot Entries.JPG

Robot Entries-2.JPG
 
Last edited:

Jeff A

Well-known Member
Pro Member
Pro Member
Followers
4
Following
0
247
104
Oct 31, 2020
Name
Jeff
Country
United States
City/State
Elk Grove, California
I for one, have never seen what you have shown above.
 

garuda

Well-known Member
Original poster
Pro Member
Pro Member
Followers
4
Following
0
286
186
Sep 14, 2020
Name
Mike
City/State
Illinois
I for one, have never seen what you have shown above.

I was browsing about the forum to learn all the features and just noticed this. Seems those little guys are quite busy. I thought Tim should know about it, if he doesn't already. It may not be a problem, since nowadays bots are everywhere. Depends on what they are extracting, I guess.
 

Timothy Mayo

Admin
Pro Member
Pro Member
Followers
64
Following
24
741
510
Mar 1, 2020
Name
Tim
Country
United Kingdom
City/State
Malvern
These are just your typical Google/Bing Search Robots. They crawl the site looking for new content to include in their Search Results Pages.


They are able to view everything that none registered members can see, which is most things, although they can't view profiles. Some images might also be indexed if they are included in threads that have been indexed.

I try to block a lot of the gallery images from being indexed by search engines as there is no need for them to be really, I also only allow useful threads to be indexed at this time.
 
Last edited:
  • Like
Reactions: dschuffert

Ivanturas

Active Member
Pro Member
Pro Member
Followers
0
Following
0
57
19
Nov 17, 2020
Name
Iván Fuentes Hagar
XenForo (which I believe is the engine underneath these forums) is pretty good at handling search engine crawlers. I wouldn't worry much; as Tim said, they will only index what's public.
 

garuda

Well-known Member
Original poster
Pro Member
Pro Member
Followers
4
Following
0
286
186
Sep 14, 2020
Name
Mike
City/State
Illinois
...... they will only index what's public.
.
Q#1: When you say they index what is public, describe how indexing works, and what parts of the indices are available directly to public surfing internet? Except for profile data, isn't everything on all forums public and available to surfers?

Q#2: While surfing internet, I've noticed references to forum posts on various forums. When you click them, you go to the post to read. So if a photo is attached or embedded in that post, is the photo also available to surfers?

Q#3: Can Google extract photos and place them in their "Goggle Photos" library? Just curious.
 

Timothy Mayo

Admin
Pro Member
Pro Member
Followers
64
Following
24
741
510
Mar 1, 2020
Name
Tim
Country
United Kingdom
City/State
Malvern
Q#1: When you say they index what is public, describe how indexing works, and what parts of the indices are available directly to public surfing internet? Except for profile data, isn't everything on all forums public and available to surfers?
Search engines can index anything that is viewable without logging into the site, which is most content here apart from user profiles. I can add add a tag rel="noindex" to pages/threads/posts that tells search engines that you would rather that particular page isn't included in the search results, and they may or may not adhere to that advice. Probably around 95% of content here has the rel="noindex" tag, as if you allow Google to index everything from a forum you can run into problems with thin content penalties (like a single short thread with no replies), then the site doesn't rank well in Google and the traffic/members drop.

Q#2: While surfing internet, I've noticed references to forum posts on various forums. When you click them, you go to the post to read. So if a photo is attached or embedded in that post, is the photo also available to surfers?
Yes the photo will also be available to surfers which will be guests on the forum until they register, unless it's in a personal album that you have restricted to members only.


Q#3: Can Google extract photos and place them in their "Goggle Photos" library? Just curious.
Yes it's possible for Google to include images on the forum in their image search. I'm not sure what the latest dimensions are but typically if they do this then it's a low resolution image with a link back to the forum.
 

Ivanturas

Active Member
Pro Member
Pro Member
Followers
0
Following
0
57
19
Nov 17, 2020
Name
Iván Fuentes Hagar
.
Q#1: When you say they index what is public, describe how indexing works, and what parts of the indices are available directly to public surfing internet? Except for profile data, isn't everything on all forums public and available to surfers?

Q#2: While surfing internet, I've noticed references to forum posts on various forums. When you click them, you go to the post to read. So if a photo is attached or embedded in that post, is the photo also available to surfers?

Q#3: Can Google extract photos and place them in their "Goggle Photos" library? Just curious.
I think Tim replied great to all of these. I'd just add:

Not necessarily! Forums allow admins to define both public and private content, including categories. As an example, I used to participate on a gearhead forum, where the whole "show us your project car" category was private for security reasons, as some people had a load of cash thrown in their cars.

If by "Google Photos" you mean "Google Image Search", then as Tim said, yeah, they are indexed. That doesn't give Google any rights over the images, but it is possible for people to download a low res version of the pic, and depending on how tech savvy they are, download the same pic being displayed here, without ever entering the forum.

Now, considering "how the internet works", regardless of how search engines work, or the forum's privacy settings, if what you want is for no one to download any of your photos without you knowing, then the only way is to not upload them at all, unless it's something like a private cloud drive or your own server (like the laptop that's in my home office's closet running Ubuntu server)... and even then, your stuff is still "connected" to the world, just behind some walls, but that's a more technical thing not relevant to what we're talking right now.

Perhaps it could be a good idea to make all showcase categories private, while keeping "conversation" categories public? But then again, there's a lot of really useful information in the showcase threads that would be excluded, and we should actually want for more people to find it, read more and more until they sign up and hopefully become good contributors in the community. I believe I actually got here while searching for info after getting my Alpha! (switched from Nikon about... 4 months ago?).
 
  • Like
Reactions: Timothy Mayo

Timothy Mayo

Admin
Pro Member
Pro Member
Followers
64
Following
24
741
510
Mar 1, 2020
Name
Tim
Country
United Kingdom
City/State
Malvern
Perhaps it could be a good idea to make all showcase categories private, while keeping "conversation" categories public? But then again, there's a lot of really useful information in the showcase threads that would be excluded, and we should actually want for more people to find it, read more and more until they sign up and hopefully become good contributors in the community. I believe I actually got here while searching for info after getting my Alpha! (switched from Nikon about... 4 months ago?).
I might create a private pro member forum that is restricted to pro members only. But obviously making all of the other categories restricted means that not many people would find the site, since many are discovering it via search engines like yourself. If that was the case it might not even be worth running it as very few people would join up and the community here would be very quiet.
 

Ivanturas

Active Member
Pro Member
Pro Member
Followers
0
Following
0
57
19
Nov 17, 2020
Name
Iván Fuentes Hagar
I might create a private pro member forum that is restricted to pro members only. But obviously making all of the other categories restricted means that not many people would find the site, since many are discovering it via search engines like yourself. If that was the case it might not even be worth running it as very few people would join up and the community here would be very quiet.
Absolutely agree! The private Pro forum does sound like a really good idea. Hadn't thought about that!
 

Pro Membership Upgrade

  • If you'd like to support this community you can easily do so by becoming a Pro Member and unlocking the following benefits:

    ✓ No Adverts in the Forums
    ✓ Pro Member Competitions
    ✓ Sell Items in the Marketplace
    ✓ Unlimited Gallery Uploads
    ✓ Higher Resolution Gallery Uploads
    ✓ Forum Profile Signature
    ✓ Custom Profile Cover Photo
    ✓ Advertise Your Photography Workshops

    Pro Member Upgrade

Shopping Links

You can also support this site by purchasing gear through the following affiliate links:
Although I would recommend supporting your local camera store if you have one.

User Menu

alpha shooters text logo

© 2020 Alpha Shooters. All Rights Reserved.
Disclaimer: Alpha Shooters is an independently run website and is not affiliated with Sony.com
As an Amazon Associate I earn from qualifying purchases.