Bing Search Scraper

🔎 What is a Bing Search Scraper? Our SERP scraper crawls Bing Search Results Pages (SERPs) and extracts data from those web pages in structured formats such as JSON, XML, CSV, or Excel. With this SERP Scraper API, you will be able to extract the following Bing data from each Bing page:

🌱 Organic results

🛍 Paid results

🙋‍♀️ People Also Ask

🔍 Additional attributes

## 💯 How many results can you scrape with Bing Search Scraper? Bing Search Scraper can return up to 2000 results for one keyword. This is due to the limitations of Bing itself: although it shows as if it has millions of results for a given search query, Bing will not give you new results past page 200 (if we have 10 results per page). While we regularly run Actor tests to keep the benchmarks in check, the results may also fluctuate without our knowing. The best way to know for sure for your particular use case is to do a test run yourself. ## 💸 How much will scraping Bing cost you? When it comes to scraping, it can be challenging to estimate the resources needed to extract data as use cases may vary significantly. That's why the best course of action is to run a test scrape with a small sample of input data and limited output. You’ll get your price per scrape, which you’ll then multiply by the number of scrapes you intend to do. ## ⬇️ Bing Search Scraper data input The scraper gives you really good control over what kind of Bing Search results you'll get. You can specify the following settings: - Query phrases or raw Bing search URLs 🔍 - Search market 🗺 - Language of search 🇬🇧 - Number of results per page 🔟 ### How to scrape Bing Search results by URL or keyword There are two ways you can scrape Bing search pages: either by URL or by search term. 1. Scraping by URL will get you data from Bing Search results page. You'll get Bing data for the query thats included in the URL. You can add as many URLs as you want. 2. Scraping by search term will also get you data from Bing Search results page. You can also add as many search terms as you want. #### Example input for scraping Bing Search search term It is also easy to get Bing search data by search term. Just enter the search term and a number of Bing pages to scrape. With this option, you also can: - scrape by multiple keywords in parallel by adding more search terms and separating them by a new line - indicate how many results you want to see per each Bing page (10-50) - indicate the market of search (locale) and language Here's is example of an input in JSON: json { "ignoreSoftBlocking": false, "maxPagesPerQuery": 1, "queries": "apify\ncrawlee", "resultsPerPage": 50, "saveHtml": false, "saveHtmlSnapshotUrls": false, "marketCode": "en-US", "languageCode": "en" } #### Scrape Bing Search results by URL To input URLs instead, simply replace queries with full URLs: json "queries": "https://www.bing.com/search?q=apify&form=QBLH&sp=-1&ghc=1&lq=0&pq=apify&sc=11-5&qs=n&sk=&cvid=E54654768E75474GA4316BFD06FF2CA715&ghsh=0&ghacc=0&ghpl=", Note that the Actor will extract only the query from the URL (in this example, "apify"), while all other settings (e.g., language) will be taken from your input. ## ⬆️ Bing Search data output The scraper stores its result in the default dataset associated with the scraper run, from which you can export it to various formats, such as JSON, XML, CSV, or Excel. ### Output example For each Bing Search results page, the dataset will contain a single record. Here is example in JSON: json { "searchQuery": { "term": "apify", "resultsPerPage": 10, "page": 1, "url": "https://www.bing.com/search?q=apify&mkt=en-US&setLang=en&count=10&first=1", "marketCode": "en-US", "languageCode": "en" }, "html": null, "htmlSnapshotUrl": null, "resultsTotal": 30100, "organicResults": [ { "iconUrl": "https://th.bing.com/th?id=ODLS.f0918ea1-202b-4bbd-abfe-46cd7093840f&w=32&h=32&qlt=90&pcl=fffffa&o=6&cb=13&pid=1.2", "displayedUrl": "https://apify.com", "title": "Apify: Full-stack web scraping and data extraction platform", "url": "https://apify.com/", "description": "Cloud platform for web scraping, browser automation, and data for AI. Use 2,000+ ready-made tools, code templates, or order a custom solution.", "emphasizedKeywords": ["web"], "date": null, "type": "organic", "position": 1, "productInfo": {}, "siteLinks": [ { "url": "https://apify.com/pricing", "title": "Plans & Pricing", "description": "Plans and pricing" } // ... and more ] } // ... and many more ], "paidResults": [], "peopleAlsoAsk": [ { "url": "https://docs.apify.com/platform", "question": "What is apify and how does it work?", "answer": "Apify is a cloud platform that helps you build reliable web scrapers, fast, and automate anything you can do manually in a web browser. Actors are serverless cloud programs running on the Apify platform that can easily crawl websites with millions of pages, but also perform arbitrary computing jobs such as sending emails or data transformations." } // ... and more ], "relatedQueries": [ { "title": "apify login", "url": "https://www.bing.com/search?q=apify+login&FORM=QSRE1" } // ... and more ] }
You can download the results directly from the platform using a button or from the Get dataset items API endpoint: json <https://api.apify.com/v2/datasets/[DATASET_ID]/items?format=[FORMAT]> where [DATASET_ID] is the ID of the dataset and [FORMAT]can be csv, html, xlsx, xml, rss or json. ## 🔒 Soft Blocking in Bing Scraper When scraping Bing, you may encounter soft blocking, where Bing gently degrades the quality of results rather than issuing a hard error. Typical signs include: - Limited results: Bing may ignore requests for more than the default count and only return 10 results on page 1, regardless of your resultsPerPage setting. - Empty pages: Sometimes Bing displays a page as having no results, even though there are results for your query. - Unexpected results: Occasionally, Bing may return results unrelated to your search terms when soft blocking kicks in. This actor automatically retries whenever it detects soft blocking and, by default, fires multiple concurrent requests with different extra url parameters for page 1 to improve efficiency. Once it successfully fetches a non soft blocked page, it cancels any other queued requests for that page that haven’t yet begun processing. If you’d like to disable these features, you can by disabling ignoreSoftBlocking. ## ❓FAQ ### 📚 Resources on how to use Bing SERP API to scrape Bing - Is web scraping legal? - blogpost on ethical scraping. - Platform pricing page with pricing specifications. #### How do I scrape millions of results from Bing? Please note that, although Bing always shows that it has found millions of results, Bing will never display more than a few thousands results for a single search query 🤥 If your goal is to get as many results as possible, try creating many similar queries and combine different parameters and locations. #### Is it legal to scrape Bing search results? Web scraping is legal if you are extracting publicly available data, but you should respect boundaries such as personal data and intellectual property regulations. Rule of thumb is: you should only scrape personal data if you have a legitimate reason to do so, factoring in Bing's Terms of Use as well. If you're unsure whether your reason is legitimate, consult your lawyers. We also recommend that you read our blog post on the subject: is web scraping legal? #### How can I use data scraped from Bing Search? Bing SERP API has a lot to offer in terms of how extracted Bing data can be applied: 🔸 Use it for search engine optimization (SEO) and monitor how your website performs on Bing for certain queries over time. 🔹 Monitor how frequently a search term has been used on Bing, and how it compares with total search volume. 🔸 Analyze display ads for a given set of keywords. 🔹 Monitor your competition in both organic and paid results. 🔸 Build a URL list for certain keywords. This is useful if, for example, you need good relevant starting points when scraping web pages containing specific phrases. 🔹 Analyze Bing algorithm and identify its main trends #### Can I use Bing Scraper with the API? The Apify API gives you programmatic access to the Apify platform. The API is organized around RESTful HTTP endpoints that enable you to manage, schedule, and run Apify Actors. The API also lets you access any datasets, monitor actor performance, fetch results, create and update versions, and more. To access the API using Node.js, use the apify-client NPM package. To access the API using Python, use the apify-client PyPi package.