Pesquisar

What are some best practices relating to bots and spider activity?

Environment:

Bots, Spiders  
Oracle B2C Service

Resolution:

 What does Oracle do to prevent bot activity? And what can and should you do?

When known bots or spiders access your site, the activity is not counted against your billable sessions.  

• What does Oracle do to filter out bot and spider activity?
  • Oracle B2C Service Development has an algorithm that detects and filters out bot and spider traffic.  This algorithm evaluates the traffic and then filters it out if it is following specific bot or spider-like patterns. Oracle also maintains a list of known spiders and bots that are then filtered out.  
  • Oracle Cloud Operations has installed a robots.txt file on each site to minimize the issues that spiders present.
    • See Allowing other search engines to index the Oracle B2C Service application for more information.
       
• What can and should you as the customer do to filter out bot and spider activity?
  • More information about this is available here: Large number of spiders in the Internet Spider Activity Report
    • Of note from this answer:
      Because of the consistent expansion of web spiders there might be times when a new and unknown spider indexes your site which does cause a spike in your answer views and or sessions. When you discover new spiders hitting your site you can always add them to the configuration setting:
      
      SEC_SPIDER_USER_AGENT
      Defines custom User Agents that are known web spiders. Valid entries include a comma-separated list of User Agent sub-strings (for example, MyCustomSpider or InternalGoogle). Values for this setting are case sensitive. Browsers supplying this User Agent will not experience different behavior on end-user pages in order to be in compliance with the major web search providers, though they will be considered a web spider for statistics collection. Default is blank.
       
• You can edit the SEC_INVALID_ENDUSER_HOSTS configuration to explicitly list which hosts are not allowed to access the end-user interface: 
  • Refer to Restricting computers or hosts that can access the Oracle B2C Service console and end-user pages for more information.
     
• Monitor your site activity
  • See Session usage information for more information.

For additional information, please see Session usage information (includes Demystifying Session Usage.pdf).