director, software developer, qualified tas / ict teacher.

Published: 6 months ago

Getting meaningful data from the Bing API, using C#

For a recent side project, I wanted to be able to take some user given content, scour the web for it and see if it pops up anywhere else. It’s a pretty broad requirement and something I’ve thought about a lot off and on for the last few years. I looked into it a while back, when the Google API wasn’t so restrictive, but due to the complexity of what I wanted to do with the data, it quickly got thrown into the too hard basket.

Recently someone suggested that I checkout the Bing API to see what’s on offer. With 5,000 free requests a month it’s a decent start. While it may not be perfect for my use, it’s definitely worth checking out.

The Bing API allows you to return news results, web results images etc, it’s fairly comprehensive. For this demo I was only interested in the web results, which is great because Microsoft has provisioned its own plan for web results only. You can sign up for it yourself here: http://datamarket.azure.com/dataset/bing/searchweb If you find that 5,000 results are too little, you could probably mix and match API keys with the free 5,000 requests on the standard API here: http://datamarket.azure.com/dataset/bing/search

The Azure data market does provide some support for the Bing API, suggesting some basic examples, but they’re all pretty average in terms of readability and getting meaningful data back from a search result.

This demo takes your through building a decent Bing API hook in C#, so that you can easily query the Bing API and get data back in a useful format. If you’re interested in the end result checkout the code here: https://github.com/LucasMoffitt/Bing-Search also checkout the unit and integration tests if you decide to implement something similar in a production environment.

Starting with the domain

The Bing API Result returns a mess of Json, which is great for us because there’s already a stack of powerful Json tools that can handle Deserialization. Only problem being getting the data into the right shape for them to do the job. After inspecting the Json results from fiddler and looking at tools like http://json2csharp.com/ the C# domain looks a bit like this:

public class BingSearch
{
public List<Result> Results { get; set; }


public class Result
{
public Metadata __Metadata { get; set; }
public string ID { get; set; }
public string Title { get; set; }
public string Description { get; set; }
public string DisplayUrl { get; set; }
public string Url { get; set; }
}

public class Metadata
{
public string Uri { get; set; }
public string Type { get; set; }
}
}

Creating the RestSharpClient

With our domain setup and ready to go, we need to start looking at how to make the request in C#, I prefer to use RestSharp for simple projects like this, but there’s no reason you couldn’t just use the HttpClient.

If you used the links above to create an account, you should now be the proud owner of a shiny API key. When connecting to the Bing Rest API with Basic Auth, simply provide the API key as the username and password. Using rest sharp it’s as easy as this:

private readonly RestClient _client;

public Searcher(string apiKey)
{
if (string.IsNullOrEmpty(apiKey))
throw new ArgumentNullException("apiKey");


_client = new RestClient();
_client.Authenticator = new HttpBasicAuthenticator(apiKey, apiKey);
}

The one thing that the RestSharp Json Deserializer isn’t good with, is dynamic objects. Because the Bing API returns Json wrapped in a “d” object and properties are lavishly prefixed with underscores and difference casing, we want a serious Json Deserializer to bash the results into our nice clean domain. Thankfully RestSharp lets us do this by plugging in our own handlers, in this case to deal with Json results.

public class DynamicJsonDeserializer : IDeserializer
{
public string RootElement { get; set; }
public string Namespace { get; set; }
public string DateFormat { get; set; }


public T Deserialize<T>(IRestResponse response)
{
return JsonConvert.DeserializeObject<dynamic>(response.Content);
}
}

and then adding it to the RestClient.

_client.AddHandler("application/json", new DynamicJsonDeserializer());

Building the request

So we have our nice and clean domain, we’ve got our rest client setup, now we need to build our request. The Bing API has a stack of different parameters you can apply, but for this example to show you the basic, we’re only going to worry about the actual search query and selecting the amount of results the API should return. Following the format below, it would be very easy to build upon that if required.

The API logic requires that if we make a response count request, that it be above 0. So easily enough we check if there’s an input number higher than 0, and if so we append the query. If not, we simply add the search query (called “Query”, which is mandatory).

The API also wants us to wrap the “Query” parameter with ‘ so we do that also. The request resource will change depending on if you’re using web results or the default API, in the web results only case, it should look like this: https://API.datamarket.azure.com/Bing/SearchWeb/v1/Web

Here’s my completed request logic.

public RestRequest BuildBingRequest(string query, int resultsCount)
{
var request = new RestRequest
{
Resource = "https://api.datamarket.azure.com/Bing/SearchWeb/v1/Web",
Method = Method.GET,
RequestFormat = DataFormat.Json
};


if (resultsCount > 0)
{
request.AddParameter(new Parameter
{
Name = "$top",
Value = resultsCount,
Type = ParameterType.GetOrPost
});
}


request.AddParameter(new Parameter
{
Name = "Query",
Value = string.Format("'{0}'", query.Trim()),
Type = ParameterType.GetOrPost
});


return request;
}

Fire the laser

We’ve built our request. We’ve got our client and our shiny domain, now all we need to do is fire the request and get the data.

Using our Json.Net dynamic Deserializer we can easily bypass the pesky “d” wrapper and get straight to the search results.

public IEnumerable<BingSearch.Result> Search(string query, int resultsCount)
{
var request = BuildBingRequest(query, resultsCount);
var response = _client.Execute<dynamic>(request);
if (response.ResponseStatus == ResponseStatus.Error)
{
if (response.StatusCode == HttpStatusCode.Unauthorized)
throw new AuthenticationException("Api Key is not correct");


throw new Exception(string.Format("Request Error Message: {0}. Content: {1}.", response.ErrorMessage, response.Content));
}


return response.Data == null ? null : JsonConvert.DeserializeObject<BingSearch>(response.Data.d.ToString()).Results;
}

Now we’ve got something useful!

Bing Results Example

Just like any other collection of objects, you can now do whatever you want with the search results. In less than 100 lines of code, we can bend the Bing search platform to our will.

If you’ve got improvements to the demo code feel free to send a pull request, and same goes if you can think of more questionable areas that could use more testing.

tl:dr https://github.com/LucasMoffitt/bing-search

Have a Comment?

Some HTML is OK