How do social listening tools collect data?

This article will show how the data collection of social media listening and tracking tools works. We will use BuzzMetrics for demonstration. BuzzMetrics is a social listening tool for tracking campaigns and brands managed by major corporations such as Samsung, Coca-Cola, Unilever and global agencies such as Ogilvy, Maxus, Leo Burnett, Phibious, Performics... in Vietnam. Social Listening is a new business form of market research. And just like traditional market research processes, social media research also goes through five stages: 1. Data Collection - 2. Take samples - 3. Data classification - 4. Data Analysis - 5. Presentation of research report. Data collection is the first step in the process of carrying out research on social media. Social listening tools will have to build a data pool for each country they study (panels), and this could take months or even years. At the same time, for each campaign, agencies will request the addition of channels used for that campaign, such as microsites, new fanpages, which the social listening tool will have to respond to. Collecting market-wide data is necessary and allows researchers to produce trend and industry studies. There are currently two main methods for collecting data: Keywords and Sites.

‍COLLECTING DATA WITH KEYWORDS

‍This method is applied to global social networks such as Facebook, Google Plus, Youtube, Twitter, Instagram... where social listening tools connect to APIs (Application Programming Interface) of social networks and request permission to retrieve data containing keywords.

• Advantages: Allows access to data from the entire social network, including individual pages
• Disadvantages: Similar to Facebook's hashtag search engine, retrieving data using APIs depends on the limitations of social networks. The API is unstable and changeable, so the data returned by the API may be incomplete and inconsistent over time. Currently there is no clear statistic of how much of a discussion can be obtained by API data retrieval.

‍DATA COLLECTION BY SITES

‍It is the collection of data by site in which the sites with the data you want to collect must be listed in the system. This method helps to collect the entire data of the covered channels. The system will access the websites to be collected, click on article links, find article readings and record information of value for research effectively without affecting the websites being collected.

With the method of collecting data using sites, the system performs operations as a user when accessing the page to be collected. After clicking on the Thread, the system will copy the entire content displayed on the screen including Title, Replies, Authors, URL, etc. Then the system will return to each page as soon as possible to find a new thread and a new reply for the old thread. Each reply will count as 1 mention and each thread will also count as 1 mention. Therefore, a thread with 100 replies will count as 101 mentions, or 101 consumer opinions.

• Advantages: Allows access to all data in the site
• Disadvantages: Obtaining data by site can lead to a temporary lack of data for the following reasons:

- Site structure changes

- Site blocking: automatic collection of data, in case of misuse of resources of websites, will be blocked by admin

- Site down: Unable to access to collect data when the page is in maintenance or problems occur

- Problems come from Internet channel

Building a list of all forums, e-newspapers and fanpages in Vietnam is a long process. That's why foreign Social Listening Tools often lack data due to insufficient site collection, or when faced with one of the above problems, there is no human resources to deal with it immediately. In the process of ensuring complete data for campaigns and brands, social listening tools such as Buzzmetrics have to use a human resource department that applies a variety of computer-assisted methods to check whether data on critical channels is being collected enough or not. This is the difference between free tools and all-inclusive social listening solutions, which is the guarantee of complete data for customers. Social listening technology, as well as search engine technology, is a model that aggregates market data. A social listening tool must store data for at least two years for research purposes. The pressure to store and process data also increases over time, so the investment in hardware infrastructure systems is also huge and continuous.