E-mail address harvesting

E-mail harvesting is the process of obtaining lists of e-mail addresses using various methods for use in bulk e-mail or other purposes usually grouped as spam.

Methods

The simplest method involves spammers purchasing or trading lists of e-mail addresses from other spammers.

Another common method is the use of special software known as "harvesting bots" or "harvesters", which spider Web pages, postings on Usenet, mailing list archives, internet forums and other online sources to obtain e-mail addresses from public data.

Spammers may also use a form of dictionary attack in order to harvest e-mail addresses, known as a directory harvest attack, where valid e-mail addresses at a specific domain are found by guessing e-mail address using common usernames in email addresses at that domain. For example, trying alan@example.com, alana@example.com, alanb@example.com, etc and any that are accepted for delivery by the recipient email server, instead of rejected, are added to the list of theoretically valid e-mail addresses for that domain.

Another method of e-mail address harvesting is to offer a product or service free of charge as long as the user provides a valid e-mail address, and then use the addresses collected from users as spam targets. Common products and services offered are jokes of the day, daily bible quotes, news or stock alerts, free merchandise, or even registered sex offender alerts for one's area. Another technique was used in late 2007 by the company iDate, which used e-mail harvesting directed at subscribers to the Quechup website to spam the victim's friends and contacts.

Spam differs from other forms of direct marketing in many ways, one of them being that it costs little more to send to a larger number of recipients than a smaller number. For this reason, there is little pressure upon spammers to limit the number of addresses targeted in a spam run, or to restrict it to persons likely to be interested. One consequence of this fact is that many people receive spam written in languages they cannot read — a good deal of spam sent to English-speaking recipients is in Chinese or Korean, for instance. Likewise, lists of addresses sold for use in spam frequently contain malformed addresses, duplicate addresses, and addresses of role accounts such as postmaster.

Spammers may harvest e-mail addresses from a number of sources. A popular method uses e-mail addresses which their owners have published for other purposes. Usenet posts, especially those in archives such as Google Groups, frequently yield addresses. Simply searching the Web for pages with addresses — such as corporate staff directories or membership lists of professional societies — using spambots can yield thousands of addresses, most of them deliverable. Spammers have also subscribed to discussion mailing lists for the purpose of gathering the addresses of posters. The DNS and WHOIS systems require the publication of technical contact information for all Internet domains; spammers have illegally trawled these resources for email addresses. Many spammers use programs called web spiders to find email addresses on web pages. Usenet article message-IDs often look enough like email addresses that they are harvested as well.

Spammer viruses may include a function which scans the victimized computer's disk drives (and possibly its network interfaces) for email addresses. These scanners discover email addresses which have never been exposed on the Web or in Whois. A compromised computer located on a shared network segment may capture email addresses from traffic addressed to its network neighbors. The harvested addresses are then returned to the spammer through the bot-net created by the virus.

A recent, controversial tactic, called "e-pending", involves the appending of e-mail addresses to direct-marketing databases. Direct marketers normally obtain lists of prospects from sources such as magazine subscriptions and customer lists. By searching the Web and other resources for e-mail addresses corresponding to the names and street addresses in their records, direct marketers can send targeted spam e-mail. However, as with most spammer "targeting", this is imprecise; users have reported, for instance, receiving solicitations to mortgage their house at a specific street address — with the address being clearly a business address including mail stop and office number.

Spammers sometimes use various means to confirm addresses as deliverable. For instance, including a hidden Web bug in a spam message written in HTML may cause the recipient's mail client to transmit the recipient's address, or any other unique key, to the spammer's Web site. Users can defend against such abuses by turning off their mail program's option to display images, or by reading email as plain-text rather than formatted.

Likewise, spammers sometimes operate Web pages which purport to remove submitted addresses from spam lists. In several cases, these have been found to subscribe the entered addresses to receive more spam.

When persons fill out a form it is often sold to a spammer using a web service or http post to transfer the data. This is immediate and will drop the email in various spammer databases. The revenue made from the spammer is shared with the source. For instance if someone applies online for a mortgage, the owner of this site may have made a deal with a spammer to sell the address. These are considered the best emails by spammers, because they are fresh and the user has just signed up for a product or service that often is marketed by spam.

Legality

In Australia, the creation or use of email-address harvesting programs (address harvesting software) is illegal according to the 2003 anti-spam legislation. [1] [2]. The legislation is intended to prohibit emails with 'an Australian connection' - spam originating in Australia being sent elsewhere, and spam being sent to an Australian address.

In The United States of America, the CAN-SPAM Act of 2003 [3] made it illegal to initiate e-mail to a recipient where the electronic mail address of the recipient was obtained:

• Using an automated means that generates possible electronic mail addresses by combining names, letters, or numbers into numerous permutations.

• Using an automated means to extract electronic mail addresses from an Internet website or proprietary online service operated by another person, and such website or online service included, at the time the address was obtained, a notice stating that the operator of such website or online service will not give, sell, or otherwise transfer addresses maintained by such website or online service to any other party for the purposes of initiating, or enabling others to initiate, electronic mail messages.

Anti-harvesting Methods

An automated method to attack automated e-mail address harvesters involves List poisoning, a technique that may fill the harvested lists with dynamically generated fake e-mail addresses, thus theoretically rendering the harvested list useless.

On an individual level, users who post e-mail addresses on websites can use Address munging to make it harder to harvest. For example by changing "bob@example.domain" to "bob at example dot domain" to keep the address from being harvested by simple bots. Putting email addresses in images instead of plain text is another technique.

A method that can be implemented on a website, is to provide a contact form instead of an e-mail address. The contact form provides a textarea for the message, and an input for the sender's e-mail address. The server-side script that processes the posted form data, is then responsible for sending the actual message, which means that the e-mail address of the recipient is never exposed. Note that contact forms have other drawbacks: the user cannot use his preferred e-mail client to compose the message, and insecure contact forms may be subject to other types of automated abuse, such as bots that post spam via the form.

Another way to hide an email address is Google's reCAPTCHA Mailhide email protection, which can create a link for you to use that degrades gracefully without JavaScript (but does require a human to click, open a new popup, solve a two-word Captcha via images or listening, then offers a mailto link). reCAPTCHA can also be embedded to solve some contact form spam problems.

A method that can be implemented at the recipient email server for combatting directory harvesting attacks is to reject all e-mail addresses as invalid from any sender that has specified more than one invalid recipient address, however this carries a risk of legitimate email being blocked too.

For CAN-SPAM Act of 2003 harvesting protection, operators of web sites and online services should include a notice that the site or service will not give, sell, or otherwise transfer addresses maintained by such website or online service to any other party for the purposes of initiating, or enabling others to initiate, electronic mail messages.

If you want allow visitors of your site to email you but hide your email address from spammers, you can scramble or encrypt it. One way to do this is by placing the encrypted email in a script that runs when the web page is loaded in a JavaScript-enabled browser. This makes it difficult for spambots to find and decrypt the email address.

See also

• Botnet
• List poisoning
• Spamtrap
• Anti-spam techniques (e-mail)

External links

• Spam laws
• Spamwise website harvesting-risk checker
• CDT study of address-harvesting
• ZDNet report on harvesting from Twitter