Good and bad bots

Bots are becoming a part of our life progressively. So, we are not sure anymore who we are dealing with – human or bot. You are lucky to have encountered a “good” bot, but what if it’s a bad one? And how did they emerge in the first place?

Web crawlers became the first major introduction of bots into the Internet. It was clear from the beginning that the unified standard of management of behavior of such bots was necessary, so that they could not harm resources or download the information that was not intended for their usage. The common known robots.txt file, which can be found almost at every site in the Internet, hit the web. Unfortunately, along with the positive research new threats were being developed: plotters were learning to use bots to make DoS and DDoS attacks.

Characteristics of a good bot are the following:
1. It is identified correctly as a bot
2. It scans the site sparingly and considers the directives of the webmaster, made in robots.txt

As we know, every medal has the reverse side, so harmful bots also exist. What are they and why are they harmful? The most harmful bots are spambots, which spy, collect information from guest books and contact forms, carry out DoS and DDoS attacks, which flood the network with irrelevant information.

Characteristics of a bad bot are the following:
1. It is harmful to the site owner 2. It scans the site aggressively and ignores robots.txt.

The statistics of 20 000 sites was analyzed within 90 days in 2018. It was found that bots account for 56% of all website traffic at the average. At that, 29% were malicious in nature. It was also revealed that the larger the site is, the more bad bots there are. The conclusion is obvious: the price of protection is clearly lower then potential losses from malicious bots. Isn’t it time to think about it?

What is DoS-attack?

DoS attack (a denial-of-service attack) is a cyber-attack where the perpetrator (usually hacker) seeks to make a machine or network resource unavailable, i.e. creating such conditions, when legitimate users of the system can’t get access to the provided system resources (servers), or the access is hindered.

Now DoS and DDoS attacks are most popular since they make it possible to shut down almost any system without leaving legally valid evidence. The cost of attack organization is negligible. An hour long 10GBit/s attack costs approximately EUR/USD 50, and anyone, who visits a special hacker service in the Internet, can arrange it. A distributed denial-of-service (DDoS) attack occurs when multiple systems flood the bandwidth or resources of a targeted system. Under current conditions DDoS attacks involve not only computers but other consumer devices with access to the Internet as well.

First, the perpetrator scans the network using specially designed scenarios, which reveal potentially weak nodes. The selected nodes are attacked, and the perpetrator gets the administrator permissions. Trojan programs, running in background mode, compromise the captured nodes. Now these computers are called zombie computers: their users don’t even suspect that they are potential participants of DDoS attack.

Then the perpetrator sends certain commands to the captured computers, and they, in their turn, carry out a powerful attack on the targeted Internet service. In some cases unintentional action leads to actual DDoS attack, for example, adding a link on a popular Internet resource to a site, located on a rather slow server (slashdot effect). Massive increase in traffic leads to overload and, respectively, to denial of service for some users.

How to identify and block “bad” bots

Ben Goodsell is a lead SEO for RKG Merkle, with deep experience in technical SEO, social media, link building, and content strategy. Ben has worked with some of the largest sites and brands on the web. According to him, any competent optimizer is familiar with using Log Files to understand Googlebot behavior, but few seem to know they can be used to identify bad bots crawling your site. The main danger is that bad bots are executing JavaScript automatically, inflating analytics numbers, expending your resources and scraping and duplicating content.

According to the data, bots account for 56% of all website traffic at the average. At that, 29% were malicious in nature and caused noticeable damage. As part of the study, the expert group of the agency looked at 20,000 websites (of all sizes) over a 90-day period. Additional insight showed the more you build your brand, the larger a target you become.

Bot Protection

A number of filters, connected to the high capacity Internet channel, are used for defense against network attacks. The filters analyze the passing traffic gradually by revealing non-standard network activity and errors. The number of the analyzed non-standard traffic patterns comprises all currently known attack methods, including those made via distributed botnets.

Internet resources try to protect themselves from such attacks by creating various security tools at their servers or requesting protection from their provider. Unfortunately, such protection is useless since attacks with power of tens of GBit/s, ordinary now, will simply flood the network from the provider to the attacked server or, more likely, will block the network used by the provider to access the Internet.

Thus, direct protection of the Internet service or provider within their territory is almost always useless. It is necessary to protect the attacked Internet service by intercepting the traffic before it reaches the bottleneck – channel between the provider and the Internet backbone.

Cloud technologies are used for that end, when the system, conducting defense, is situated as close to backbones as possible, has multiple high-speed Internet connection to several backbones, ideally – geographically distributed over different regions of the planet.

Myth: DDoS attack – it is expensive

“DDoS attack is a very expensive action, only business magnates can afford it. In extreme case, it is scheming of secret services!”

Actually, such actions are very affordable.

Now we see that DDoS-activity is constantly increasing. It’s affordable and monstrously cheap to order an attack. There will be no proof links to avoid propaganda accusations. But take our word for it, it is so.

call me back
to top
Call me back!