Insights from Our Experts

Blog image

How SayOne implemented data scraping for an aggregator app!

Author Image

Shamily VA,Software Engineer

SayOne Technologies has proven expertise in data scraping and crawling. Our best practices of web crawling for the extraction of relevant data and its structuring for analysis have yielded rewards. SayOne’s crawling experts specialize in web data crawling, extraction and information integration. We understand the clients’ need for processed information and help them analyze, visualize and monitor it closely. We build interactive applications that facilitate further analysis to help the end user make the most of the obtained data.

 

Our customized crawlers crawl through and analyze numerous data sources to fetch relevant data. This data is then turned into insights using the latest open source tools. We backtrack and monitor the entire cycle to improve process and boost performance. Our built-in tools and applications engage the user and allow them to work with available value-added information effectively. 

 

Aggregator apps is an area where SayOne has employed scraping. Aggregators are fast gaining popularity since they provide a wealth of information at the fingertips which consolidates the best of data from diverse sources. The term refers to a website or computer software that combines or assembles data or information from multiple online sources and provides a repository to access these. Some of the popular aggregators in vogue are rottentomatoes (movie reviews), Alatest (product reviews), AOTY (music), Google (search), Flipboard (news) and Foodpanda (cuisine). 

 

 

Aggregators collect data from various sources through different methods. Important among these is web scraping. This is done by directly implementing http or by embedding a web browser. Web scraping is related to web indexing using a bot or web crawler. The latter transforms unstructured data on the web, usually html, into structured data that can be stored and analyzed in a central local database or spreadsheet.  Some other methods for web scraping are HTML parsers, vertical aggregation platforms, etc. At SayOne we load the full content of a website using Python programming language and combine it with parsing tools to extract relevant information. 

 

An instance of successful scraping by SayOne was for an indoor maps application. This application is to indoors what Google Maps is to outdoors. It helps easier search of destinations and also points of interest within a location such as a shopping mall. The app offers customers timely alerts, collects intelligent data and sends out push notifications about deals specially personalized for the customer. It serves as an ideal shopping friend, albeit virtual, who enhances the shopping experience manifold. Data specific to shops, offers and deals was scraped from the websites of shops with who we were in tie-up. We used the Scrapy framework supported by Scrapy Cloud, a cloud-based web crawling platform of Scraping leader Scrapinghub. It enables smooth deployment of crawlers and the capability to scale them quite easily. Spiders were extended in clicks using add-ons. The scraped data is stored in a high-availability database, shared using dashboard.