Build a News Article Search Engine with The New York Times, The Guardian, and Wikipedia APIs.

APIs are a hot topic for developers and entrepreneurs alike. An API is a way for different applications to communicate with each other by sending and receiving information. For example, your company may want to have a live feed of all tweets about itself displayed on its website. The way to access these tweets is to ask Twitter’s API for the relevant information. Twitter’s API will then send back the information your company wants and the code on your company’s website can display that information.

I recently had an idea for a web application that would return relevant search results by asking the APIs of various trusted news sources for any articles they had on the topic. Before beginning development, I scoured the internet for tutorials on how to use the three APIs I wanted use — The New York Times API, The Guardian API, and the The MediaWiki API. I did not find anything that showed me exactly how to implement what I wanted to do so now, after finishing my application, I am writing a quick tutorial that should get you up and running with these three free news APIs.

In the first section, I will throw up my HTML, CSS, and JavaScript code so that developers can read it and take from it the parts they need. In the second section, I will highlight the parts of the code that are most relevant to querying the three APIs my application communicates with and explain how they work. My hope is that this post will help anyone comfortable with HTML, CSS, jQuery, and JavaScript make the leap into working with APIs. For anyone interested in working with APIs but unfamiliar with the aforementioned technologies you can click the following links to some great resources for learning them — HTML & CSS, jQuery, JavaScript. If you want all the code you can clone my project’s GitHub repository and if you want to see the site in action you can check out the live site.

Section 1 - The Code

HTML

pop-uptext
  1. <!DOCTYPE html>
  2. <html>
  3. <head>
  4.     <title>Anchorbase | Find news better</title>
  5.     <link href="css/bootstrap.min.css" rel="stylesheet" media="screen">
  6.     <link rel="stylesheet" type="text/css" href="index.css">
  7. </head>
  8. <body>
  9.     <div class="container">
  10.         <h3 id="anchorbase-logo" class="text-center">Anchorbase</h3>
  11.         <div class = "row">
  12.             <div class="col-md-8 col-md-offset-2">
  13.                 <form id="search-bar" class="span4" role="form">
  14.                     <input id="user-search" class="form-control span3" placeholder="Search news better" autocomplete="off">
  15.                 </form>
  16.             </div>
  17.         </div>
  18.         <div class="row" id="sortable">
  19.             <div class="row-sort">
  20.                 <div class="col-md-4 text-center" id="NYT">
  21.                     <table id="NYT-results-table-header results-table-header" class="NYT-results-table-header table table-bordered">
  22.                         <tr class="NYT-results-header">
  23.                             <th class="text-center" >New York Times</th>
  24.                         </tr>
  25.                     </table> 
  26.                     <div id="NYT-results-table-scrolable-container" class="results-table-scrolable-container" >
  27.                         <table id="NYT-results-table" class="NYT-results-table table">                                 
  28.                             <tr id='NYT-last-row'>
  29.                                 <td></td>
  30.                             </tr>
  31.                         </table>
  32.                     </div>
  33.                 </div> 
  34.                 <div class="col-md-4 text-center" id="Guardian">
  35.                     <table id="G-results-table" class="G-results-table table table table-bordered">
  36.                         <tr class="guardian-results-header results-table-header">
  37.                             <th class="text-center" >The Guardian</th>
  38.                         </tr>
  39.                     </table>
  40.                     <div id="guardian-results-table-scrolable-container" class="results-table-scrolable-container" >
  41.                         <table id="guardian-results-table" class="guardian-results-table table">    
  42.                             <tr id='guardian-last-row'>
  43.                                 <td></td>
  44.                             </tr>
  45.                         </table>
  46.                     </div>
  47.                 </div>
  48.                 <div class="col-md-4 text-center" id="Wiki">
  49.                     <table id="G-results-table" class="G-results-table table table table-bordered">
  50.                         <tr class="G-results-header results-table-header">
  51.                             <th class="text-center" >Wikipedia</th>
  52.                         </tr>
  53.                     </table>
  54.                     <div id="wikipedia-results-table-scrolable-container" class="results-table-scrolable-container" >
  55.                         <table id="wikipeida-results-table" class="wikipeida-results-table table">    
  56.                             <tr id='wikipedia-last-row'>
  57.                                 <td></td>
  58.                             </tr>
  59.                         </table>
  60.                         <div>    
  61.                         </div>
  62.                     </div>
  63.                 </div>
  64.             </div>    
  65.         </div>
  66.     </div>
  67.     <script src="http://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js" ></script>
  68.     <script src="http://code.jquery.com/ui/1.11.2/jquery-ui.js"></script>
  69.     <script type='text/javascript' src='js/jquery.js'></script>
  70.     <script>
  71.     $(document).ready(function() {
  72.         $('#sortable > .row-sort').sortable();
  73.     }); 
  74.     </script>
  75. </body>
  76. </html>

CSS custom

pop-uptext
  2. #anchorbase-logo {
  3.     margin-top: 100px;
  4. }
  5. #search-bar {
  6.     margin-top: 30px;
  7.     margin-bottom: 40px;
  8. }
  9. .results-table-scrolable-container{
  10.     height: 500px;
  11.     overflow: hidden;
  12. }
  13. .results-table-scrolable-container:hover {
  14.     overflow: auto;
  15.     overflow-y: scroll;
  16. }
  17. .NYT-results-header{
  18.     height: 0px;
  19. }
  20. .NYT-results-header:hover {
  21. background-color: #f3fbfb;
  22. }
  23. .guardian-results-header:hover {
  24. background-color: #f3fbfb;    
  25. }
  26. .G-results-header:hover {
  27. background-color: #f3fbfb;
  28. }
  29. .form-control:focus {
  30.   border-color: #D8F2F3;
  31.  }
  32. ::-webkit-scrollbar {
  33.     -webkit-appearance: none;
  34.     width: 7px;
  35. }
  36. ::-webkit-scrollbar-thumb {
  37.     border-radius: 4px;
  38.     background-color: rgba(0,0,0,.5);
  39.     -webkit-box-shadow: 0 0 1px rgba(255,255,255,.5);
  40. }

JavaScript/jQuery

pop-uptext
  1. $(document).ready(function(){    
  2.     
  3.     var userSearchTerm = "",
  4.         
  5.         NYTapiKey = "--------------google New York Times API key to get your own free key and then paste it here as a string---------------------------",
  6.         NYTtimesShown = 0,
  7.         NYTresultsArray = [],
  8.         
  9.         guardianApiKey = "----------google Guardian API key to get your own free key and then paste it here as a string--------------------------",
  10.         guardianTimesShown = 1,
  11.         guardianResultsArray = [],
  12.         wikipediaResultsArray =[],
  13.         wikipediaContinueString = "";
  14.     function ResultsObject(url, headline, leadParagraph, index) {
  15.         this.url = url;
  16.         this.headline = headline;
  17.         this.leadParagraph = leadParagraph;
  18.         this.index = index;
  19.     }
  20.     var apiCallNYT = function(){
  21.         $.ajax({
  22.             url: "http://api.nytimes.com/svc/search/v2/articlesearch.json?q=" + userSearchTerm.replace(' ', '+') + "&page=" + NYTtimesShown + "&api-key=" + NYTapiKey,  
  23.             dataType: 'json',
  24.             success: function(results){
  25.                 var resultsString = JSON.stringify(results),
  26.                     resultsObject = results,
  27.                     documentsArray = resultsObject['response']['docs'];
  28.                
  29.                 documentsArray.forEach(function(element, index, array){
  30.                     if (element['web_url'] == null || element['headline']['main'] == null || element['lead_paragraph'] == null) {
  31.                     } else {
  32.                     article = new ResultsObject(element['web_url'], element['headline']['main'], element['lead_paragraph'],  NYTresultsArray.length);
  33.                     NYTresultsArray.push(article);
  34.                     }
  35.                 });
  36.                 resultsPrintNYT(NYTresultsArray);
  37.                 NYTtimesShown++;
  38.             }
  39.         });
  40.     };
  41.     
  42.     var resultsPrintNYT = function(resultsArray){
  43.         for (var i = 0; i < resultsArray.length; i++){
  44.             if (resultsArray[i] !== undefined) {
  45.                 $("<tr id='NYTresult"+resultsArray[i].index+"'><td class='text-left NYTresult'><a href='" + resultsArray[i].url + "'>" + resultsArray[i].headline +"</a><p>" + resultsArray[i].leadParagraph + "</p></td></tr>").insertBefore('#NYT-last-row');
  46.             } 
  47.         }
  48.     };
  49.     var apiCallGaurdain = function() {        
  50.         $.ajax({
  51.             url: "http://content.guardianapis.com/search?api-key="+guardianApiKey+"&show-fields=all&page="+ guardianTimesShown + "&q="+userSearchTerm.replace(' ', '%'),
  52.             dataType: 'jsonp',
  53.             success: function( results ) {
  54.             var results = results;
  55.             //console.log(results);
  56.             var documentsArray = results['response']['results'];
  57.             //console.log(documentsArray);
  58.             documentsArray.forEach(function(element, index, array){
  59.                     if (element['webUrl'] == null || element['webTitle'] == null || element['fields']['standfirst'] == null) {
  60.                     } else {
  61.                     article = new ResultsObject(element['webUrl'], element['webTitle'], element['fields']['standfirst'],  guardianResultsArray.length);
  62.                     guardianResultsArray.push(article);
  63.                     }
  64.                 });
  65.                 resultsPrintGuardian(guardianResultsArray);
  66.                 guardianTimesShown++;
  67.             }
  68.         });
  69.     };
  70.     var resultsPrintGuardian = function(resultsArray){
  71.         for (var i = 0; i < resultsArray.length; i++){
  72.             if (resultsArray[i] !== undefined) {
  73.                 $("<tr id='guardianResult"+resultsArray[i].index+"'><td class='text-left guardianResult'><a href='" + resultsArray[i].url + "'>" + resultsArray[i].headline +"</a><p>" + resultsArray[i].leadParagraph + "</p></td></tr>").insertBefore('#guardian-last-row');
  74.             } 
  75.         }
  76.     };
  77.     var apiCallWikipedia = function(){
  78.         $.ajax({
  79.             url: "http://en.wikipedia.org/w/api.php?format=json&action=query&generator=allpages&gaplimit=10&gapfrom="+ userSearchTerm.replace(' ', '+')+"&prop=pageimages|extracts&pilimit=max&exintro&explaintext&exsentences=1&exlimit=max&gapcontinue=" + wikipediaContinueString,
  80.             dataType: 'jsonp',
  81.             success: function(results) {
  82.             var results = results;
  83.             console.log(results);
  84.             var documentsArray = results['query']['pages'];
  85.             console.log(documentsArray);
  86.             console.log(typeof documentsArray);
  87.             wikipediaContinueString = results['query-continue']['allpages']['gapcontinue'];
  88.             for (var prop in documentsArray) {
  89.                 if (documentsArray.hasOwnProperty(prop)){
  90.                     article = new ResultsObject( "https://en.wikipedia.org/wiki/"+ encodeURI(documentsArray[prop]['title']), documentsArray[prop]['title'], documentsArray[prop]['extract'],  wikipediaResultsArray.length)
  91.                     wikipediaResultsArray.push(article);
  92.                 }
  93.             }
  94.                  resultsPrintWikipedia(wikipediaResultsArray);
  95.                  console.log(wikipediaResultsArray);
  96.             }
  97.         });
  98.     }
  99.     var resultsPrintWikipedia = function(resultsArray){
  100.         for (var i = 0; i < resultsArray.length; i++){
  101.             if (resultsArray[i] !== undefined) {
  102.                 $("<tr id='wikipeidaResult"+resultsArray[i].index+"'><td class='text-left wikipediaResult'><a href='" + resultsArray[i].url + "'>" + resultsArray[i].headline +"</a><p>" + resultsArray[i].leadParagraph + "</p></td></tr>").insertBefore('#wikipedia-last-row');
  103.             } 
  104.         }
  105.     };
  106.    
  107.     $('#search-bar').bind('keypress',function (event){
  108.         if (event.keyCode === 13){
  109.             event.preventDefault();
  110.             
  111.             userSearchTerm = $('#user-search').val();
  112.             
  113.             NYTtimesShown = 0;
  114.             NYTresultsArray = [];
  115.             $('.NYTresult').remove();
  116.             
  117.             guardianTimesShown =1;
  118.             guardianResultsArray = [];
  119.             $('.guardianResult').remove();
  120.             wikipediaContinueString = "";
  121.             wikipediaResultsArray =[];
  122.             $('.wikipediaResult').remove();
  123.             apiCallWikipedia();
  124.             apiCallNYT();
  125.             apiCallGaurdain();
  126.         }
  127.     });
  128.     $('#NYT-results-table-scrolable-container').bind('scroll', function() {
  129.         if($(this).scrollTop() + $(this).innerHeight() >= this.scrollHeight) {
  130.             guardianResultsArray = [];
  131.             apiCallNYT();
  132.         }
  133.     });
  134.     $('#guardian-results-table-scrolable-container').bind('scroll', function() {
  135.         if($(this).scrollTop() + $(this).innerHeight() >= this.scrollHeight) {
  136.             guardianResultsArray =[];
  137.             apiCallGaurdain();
  138.         }
  139.     });
  140.     $('#wikipedia-results-table-scrolable-container').bind('scroll', function() {
  141.         if($(this).scrollTop() + $(this).innerHeight() >= this.scrollHeight) {
  142.             wikipediaResultsArray =[];
  143.             apiCallWikipedia();
  144.         }
  145.     });
  146. });

Section 2 - Explanation

The index.html and index.css files combined with Bootstrap control how the app appears on the the screen. It was important to set the container divs for the API results to have a set height on the page but to become scrollable when the user’s mouse hovers over them. This way I could ask the API, the results from which the user was scrolling through, to produce the next set of results once the user had almost scrolled down to the bottom of the div. I achieved this with the following custom CSS class and jQuery event handlers.

CSS

pop-uptext
  1. .results-table-scrolable-container{
  2.     height: 500px;
  3.     overflow: hidden;
  4. }
  5. .results-table-scrolable-container:hover {
  6.     overflow: auto;
  7.     overflow-y: scroll;
  8. }

jQuery

pop-uptext
  1. $('#NYT-results-table-scrolable-container').bind('scroll', function() {
  2.     if($(this).scrollTop() + $(this).innerHeight() >= this.scrollHeight) {
  3.         guardianResultsArray = [];
  4.         apiCallNYT();
  5.     }
  6. });
  7. $('#guardian-results-table-scrolable-container').bind('scroll', function() {
  8.     if($(this).scrollTop() + $(this).innerHeight() >= this.scrollHeight) {
  9.         guardianResultsArray =[];
  10.         apiCallGaurdain();
  11.     }
  12. });
  13. $('#wikipedia-results-table-scrolable-container').bind('scroll', function() {
  14.     if($(this).scrollTop() + $(this).innerHeight() >= this.scrollHeight) {
  15.         wikipediaResultsArray =[];
  16.         apiCallWikipedia();
  17.     }
  18. });

In the CSS we can see that all my results container divs have the same set height and hidden overflow when they are not being hovered over. This will keep the page organized and prevent different results div from becoming different heights depending on how many results are sent back by their corresponding API. Then when the user hovers a mouse over them, their overflow is set to scroll, allowing for the user to scroll down through all the results.

In the jQuery we can see that when an API result container is scrolled down to the bottom it will trigger a resetting of the API results array (the data structure that hold all the API results before they are added to the DOM) and then calls the function that asks that specific API for the next batch of results. The result is three “infinitely” scrolling divs that ask for more information from their corresponding API when and only when the user has scrolled down through the current batch of results. It may seem odd to reset the array that holds the search results from that API before calling the function that asks the API for results. This happens because my app does not store old API information on the backend. Instead I just print the results to the DOM and insert new results below. I do this to demonstrate how you could parse the information returned by an API into objects that you could then store permanently in a database but there is no reason to store results for the purpose of my app.

Now let us examine the functions that actually ask for, and then parse into objects, information from the APIs. These are the functions that the three jQuery event handlers above call when a user has scrolled to the bottom of one of them.

pop-uptext
  1. var userSearchTerm = "",
  2.        
  3.        NYTapiKey = "--------------google New York Times API key to get your own free key and then paste it here as a string---------------------------",
  4.        NYTtimesShown = 0,
  5.        NYTresultsArray = [],
  6.        
  7.        guardianApiKey = "--------------google Guardian API key to get your own free key and then paste it here as a string---------------------------",
  8.        guardianTimesShown = 1,
  9.        guardianResultsArray = [],
  10.        wikipediaResultsArray =[],
  11.        wikipediaContinueString = "";
  12. var apiCallNYT = function(){
  13.        $.ajax({
  14.            url: "http://api.nytimes.com/svc/search/v2/articlesearch.json?q=" + userSearchTerm.replace(' ', '+') + "&page=" + NYTtimesShown + "&api-key=" + NYTapiKey,  
  15.            dataType: 'json',
  16.            success: function(results){
  17.                var resultsString = JSON.stringify(results),
  18.                    resultsObject = results,
  19.                    documentsArray = resultsObject['response']['docs'];
  20.               
  21.                documentsArray.forEach(function(element, index, array){
  22.                    if (element['web_url'] == null || element['headline']['main'] == null || element['lead_paragraph'] == null) {
  23.                    } else {
  24.                    article = new ResultsObject(element['web_url'], element['headline']['main'], element['lead_paragraph'],  NYTresultsArray.length);
  25.                    NYTresultsArray.push(article);
  26.                    }
  27.                });
  28.                resultsPrintNYT(NYTresultsArray);
  29.                NYTtimesShown++;
  30.            }
  31.        });
  32.    };
  33.    var apiCallGaurdain = function() {        
  34.        $.ajax({
  35.            url: "http://content.guardianapis.com/search?api-key="+guardianApiKey+"&show-fields=all&page="+ guardianTimesShown + "&q="+userSearchTerm.replace(' ', '%'),
  36.            dataType: 'jsonp',
  37.            success: function( results ) {
  38.            var results = results;
  39.            //console.log(results);
  40.            var documentsArray = results['response']['results'];
  41.            //console.log(documentsArray);
  42.            documentsArray.forEach(function(element, index, array){
  43.                    if (element['webUrl'] == null || element['webTitle'] == null || element['fields']['standfirst'] == null) {
  44.                    } else {
  45.                    article = new ResultsObject(element['webUrl'], element['webTitle'], element['fields']['standfirst'],  guardianResultsArray.length);
  46.                    guardianResultsArray.push(article);
  47.                    }
  48.                });
  49.                resultsPrintGuardian(guardianResultsArray);
  50.                guardianTimesShown++;
  51.            }
  52.        });
  53.    };
  54.    var apiCallWikipedia = function(){
  55.        $.ajax({
  56.            url: "http://en.wikipedia.org/w/api.php?format=json&action=query&generator=allpages&gaplimit=10&gapfrom="+ userSearchTerm.replace(' ', '+')+"&prop=pageimages|extracts&pilimit=max&exintro&explaintext&exsentences=1&exlimit=max&gapcontinue=" + wikipediaContinueString,
  57.            dataType: 'jsonp',
  58.            success: function(results) {
  59.            var results = results;
  60.            console.log(results);
  61.            var documentsArray = results['query']['pages'];
  62.            console.log(documentsArray);
  63.            console.log(typeof documentsArray);
  64.            wikipediaContinueString = results['query-continue']['allpages']['gapcontinue'];
  65.            for (var prop in documentsArray) {
  66.                if (documentsArray.hasOwnProperty(prop)){
  67.                    article = new ResultsObject( "https://en.wikipedia.org/wiki/"+ encodeURI(documentsArray[prop]['title']), documentsArray[prop]['title'], documentsArray[prop]['extract'],  wikipediaResultsArray.length)
  68.                    wikipediaResultsArray.push(article);
  69.                }
  70.            }
  71.                 resultsPrintWikipedia(wikipediaResultsArray);
  72.                 console.log(wikipediaResultsArray);
  73.            }
  74.        });
  75.    }

At the top of this snippet from my jquery.js file, I define a number of variables that are essential for getting our API querying functions to work properly. Let’s take a look at them —

pop-uptext
  1. var userSearchTerm = "",
  2.        
  3.        NYTapiKey = "--------------google New York Times API key to get your own free key and then paste it here as a string---------------------------",
  4.        NYTtimesShown = 0,
  5.        NYTresultsArray = [],
  6.        
  7.        guardianApiKey = "--------------google Guardian API key to get your own free key and then paste it here as a string---------------------------",
  8.        guardianTimesShown = 1,
  9.        guardianResultsArray = [],
  10.        
  11.        wikipediaResultsArray =[],
  12.        wikipediaContinueString = "";

userSearchTerm is exactly what it sounds like, it will eventually contain the search term that the user enters into the search bar at the top of the page. We need the search term so we can pass it into our API call and get only results that are relevant to that search. Most APIs requre you to pass a key along with your request for information. This allows the API to identify who is asking for information and how many times that person asks for information. APIs that require a key will not work without one so be sure to get your own API key and enter it as a string value for the NYTapiKey and guardianApiKey variables before trying to run the above code. When you ask for data from an API it will often return only the first n results that match your request. The reason for this is efficiency. If every time a program asked the New York Times API for results about ‘Obama’ the New York Times API returned all matching results it would be constructing and sending way more information that is necessary. All that response construction costs server processing power which in turn costs lots of money. So, though some APIs allow you to specify exactly how many matching results you would like most will return the first ten or so and then provide you with a way to ask for the next ten and the next ten when and if you need to. This is the reason for initializing the NYTtimesShown and guardianTimesShown variables at their respective values. These numbers will go into our API queries along with our API key and userSearchTerm to get not only results that match our query but correct packet of results that match our query. The MediaWiki Api does not require a key and has a different way of interating over its packets of results that has to to with the wikipediaContinueString variable. As the users make more API calls by scrolling through the results divs, variables will automatically increment so that our API requests update to ask for the next and then next packet of information. Every time the wikipedia API is called, the wikipediaContineString will be updated with a piece of information that wikipedia includes in its response that indicates how to get the next packet of results. The results array variables will eventually contain objects that are filled with the information that we want from the response we get back from our API calls. In most cases the API you query will respond with more information than you need. It is helpful to construct objects from the information provided by the API that include only the information you are interested in. You can then do whatever you want with these objects without having to worry about the irrelevant information provided by the API’s response.

Now let us look at how we use these variables to build our API query and the process the results.

pop-uptext
  1. var apiCallNYT = function(){
  2.        $.ajax({
  3.            url: "http://api.nytimes.com/svc/search/v2/articlesearch.json?q=" + userSearchTerm.replace(' ', '+') + "&page=" + NYTtimesShown + "&api-key=" + NYTapiKey,  
  4.            dataType: 'json',
  5.            success: function(results){
  6.                var resultsString = JSON.stringify(results),
  7.                    resultsObject = results,
  8.                    documentsArray = resultsObject['response']['docs'];
  9.               
  10.                documentsArray.forEach(function(element, index, array){
  11.                    if (element['web_url'] == null || element['headline']['main'] == null || element['lead_paragraph'] == null) {
  12.                    } else {
  13.                    article = new ResultsObject(element['web_url'], element['headline']['main'], element['lead_paragraph'],  NYTresultsArray.length);
  14.                    NYTresultsArray.push(article);
  15.                    }
  16.                });
  17.                resultsPrintNYT(NYTresultsArray);
  18.                NYTtimesShown++;
  19.            }
  20.        });
  21.    };
  22.    var apiCallGaurdain = function() {        
  23.        $.ajax({
  24.            url: "http://content.guardianapis.com/search?api-key="+guardianApiKey+"&show-fields=all&page="+ guardianTimesShown + "&q="+userSearchTerm.replace(' ', '%'),
  25.            dataType: 'jsonp',
  26.            success: function( results ) {
  27.            var results = results;
  28.            //console.log(results);
  29.            var documentsArray = results['response']['results'];
  30.            //console.log(documentsArray);
  31.            documentsArray.forEach(function(element, index, array){
  32.                    if (element['webUrl'] == null || element['webTitle'] == null || element['fields']['standfirst'] == null) {
  33.                    } else {
  34.                    article = new ResultsObject(element['webUrl'], element['webTitle'], element['fields']['standfirst'],  guardianResultsArray.length);
  35.                    guardianResultsArray.push(article);
  36.                    }
  37.                });
  38.                resultsPrintGuardian(guardianResultsArray);
  39.                guardianTimesShown++;
  40.            }
  41.        });
  42.    };
  43.    var apiCallWikipedia = function(){
  44.        $.ajax({
  45.            url: "http://en.wikipedia.org/w/api.php?format=json&action=query&generator=allpages&gaplimit=10&gapfrom="+ userSearchTerm.replace(' ', '+')+"&prop=pageimages|extracts&pilimit=max&exintro&explaintext&exsentences=1&exlimit=max&gapcontinue=" + wikipediaContinueString,
  46.            dataType: 'jsonp',
  47.            success: function(results) {
  48.            var results = results;
  49.            console.log(results);
  50.            var documentsArray = results['query']['pages'];
  51.            console.log(documentsArray);
  52.            console.log(typeof documentsArray);
  53.            wikipediaContinueString = results['query-continue']['allpages']['gapcontinue'];
  54.            for (var prop in documentsArray) {
  55.                if (documentsArray.hasOwnProperty(prop)){
  56.                    article = new ResultsObject( "https://en.wikipedia.org/wiki/"+ encodeURI(documentsArray[prop]['title']), documentsArray[prop]['title'], documentsArray[prop]['extract'],  wikipediaResultsArray.length)
  57.                    wikipediaResultsArray.push(article);
  58.                }
  59.            }
  60.                 resultsPrintWikipedia(wikipediaResultsArray);
  61.                 console.log(wikipediaResultsArray);
  62.            }
  63.        });
  64.    }

The three apiCall functions above comunicate with their respective APIs by creating a JavaScript XMLHttpRequest object. This allows our application to ask for and receive information from the API’s server without refresing our page. This technology is called AJAX which stands for Asynchronous JavaScript and XML. To learn more about AJAX, check out this AJAX introduction on w3 schools. Essentially what is happening in the three apiCall functions above is a request is constructed by the url: properties on lines three. twenty-five, and forty-eaight, and then the results of that request, a JSON object, are processed in the success functions that start on lines five, twenty-seven, and fifty. The results argument that is passed into these functions is the JSON object that in turn is the response sent back by the API. The url is the query which defines what the response will be. I have left some console.logs in the code so that if you open up a console while running the application you can see what the results object is before we process it into our objects. Let us first examine the url properties of our three JavaScript XMLHttpRequest objects.

pop-uptext
  1. url: "http://api.nytimes.com/svc/search/v2/articlesearch.json?q=" + userSearchTerm.replace(' ', '+') + "&page=" + NYTtimesShown + "&api-key=" + NYTapiKey,
  2. url: "http://content.guardianapis.com/search?api-key="+guardianApiKey+"&show-fields=all&page="+ guardianTimesShown + "&q="+userSearchTerm.replace(' ', '%'),
  3. url: "http://en.wikipedia.org/w/api.php?format=json&action=query&generator=allpages&gaplimit=10&gapfrom="+ userSearchTerm.replace(' ', '+')+"&prop=pageimages|extracts&pilimit=max&exintro&explaintext&exsentences=1&exlimit=max&gapcontinue=" + wikipediaContinueString,

The urls above are requests sent to the servers of the API we are querying. Notice that these queries are dynamic (they are the result of concatenating our variables into the generic url request used to talk to the APIs). You could replace userSearchTerm.replace(‘ ‘, ‘+’) above with your own hard coded search query, but then every time you called the API call function you would get results only for that query. As the user sets this variable by hitting enter on the user search bar our queries will ask for results that are relevant to that query. You need to call replace(‘ ‘, ‘+’) on the userSearch term to remove blank space from the query and instead use the recognized blank space substitute, a ‘+’’ sign. We also pass variables in for the apiKey and the number of times shown. The Api key will validate our request with the API and the times shown will tell the API which packet of results we want to see. Wikipedia wants you to set a string equal to a property of the object it sends back and then append that string the to the continue field of the request ignored to get the next packet of information.

Now let’s look at what what we do with the results sent back to us once we have successfully constructed and sent our queries.

pop-uptext
  1. success: function(results){
  2.                 var resultsString = JSON.stringify(results),
  3.                     resultsObject = results,
  4.                     documentsArray = resultsObject['response']['docs'];
  5.                
  6.                 documentsArray.forEach(function(element, index, array){
  7.                     if (element['web_url'] == null || element['headline']['main'] == null || element['lead_paragraph'] == null) {
  8.                     } else {
  9.                     article = new ResultsObject(element['web_url'], element['headline']['main'], element['lead_paragraph'],  NYTresultsArray.length);
  10.                     NYTresultsArray.push(article);
  11.                     }
  12.                 });
  13.                 resultsPrintNYT(NYTresultsArray);
  14.                 NYTtimesShown++;
  15.             }
  16. success: function(results) {
  17.             var results = results;
  18.             //console.log(results);
  19.             var documentsArray = results['response']['results'];
  20.             //console.log(documentsArray);
  21.             documentsArray.forEach(function(element, index, array){
  22.                     if (element['webUrl'] == null || element['webTitle'] == null || element['fields']['standfirst'] == null) {
  23.                     } else {
  24.                     article = new ResultsObject(element['webUrl'], element['webTitle'], element['fields']['standfirst'],  guardianResultsArray.length);
  25.                     guardianResultsArray.push(article);
  26.                     }
  27.                 });
  28.                 resultsPrintGuardian(guardianResultsArray);
  29.                 guardianTimesShown++;
  30.             }
  31.  success: function(results) {
  32.             var results = results;
  33.             console.log(results);
  34.             var documentsArray = results['query']['pages'];
  35.             console.log(documentsArray);
  36.             console.log(typeof documentsArray);
  37.             wikipediaContinueString = results['query-continue']['allpages']['gapcontinue'];
  38.             for (var prop in documentsArray) {
  39.                 if (documentsArray.hasOwnProperty(prop)){
  40.                     article = new ResultsObject( "https://en.wikipedia.org/wiki/"+ encodeURI(documentsArray[prop]['title']), documentsArray[prop]['title'], documentsArray[prop]['extract'],  wikipediaResultsArray.length)
  41.                     wikipediaResultsArray.push(article);
  42.                 }
  43.             }
  44.                  resultsPrintWikipedia(wikipediaResultsArray);
  45.                  console.log(wikipediaResultsArray);
  46.             }

Here, in all three of the success functions we create a documentsArray that is filled with only the parts of the response object that we are interested in. Note that while lines four, twenty, thirty-seven all accomplish this they do so in different ways. This is because the responses sent back from the three different APIs are formatted differently. You can console.log out the results object sent back to you as I do for the results from the wikipedia API on line thirty eight to inspect it and see what parts of it are relevant to what you want to display on the page. In my case, the responses all contained an array of the title text, web url, and text summary or lead paragraph that I wanted to display so I simply set this information equal to my own documentsArray. Then the success functions iterate over the documentsArray parsing the relevant information from those arrays into objects and then pushing those objects into my own array. While it may seem convoluted to parse an array of objects that is given to you into your own array of slightly different objects it would be very useful if you wanted to store these objects in your database or perform more operations on them than my app currently does. The constructor for the results object is below. You will see that that lines nine, twenty-six, and forty-four of the above code create new objects with only the properties we want from each of the objects in the documentsArray.

pop-uptext
  1. function ResultsObject(url, headline, leadParagraph, index) {
  2.         this.url = url;
  3.         this.headline = headline;
  4.         this.leadParagraph = leadParagraph;
  5.         this.index = index;
  6.     }

Now, every time we run our apiCall functions they will create objects from the responses that contain only the information we want from those responses and place those objects into arrays that correspond to the API that the responses are from. All that’s left is to iterate over our arrays of objects and append them to the correct place in the DOM. This operation is accomplished in the print results functions which are called by the apiCall functions after they have constructed their resultsArrays. The results print functions are below.

pop-uptext
  1. var resultsPrintNYT = function(resultsArray){
  2.         for (var i = 0; i < resultsArray.length; i++){
  3.             if (resultsArray[i] !== undefined) {
  4.                 $("<tr id='NYTresult"+resultsArray[i].index+"'><td class='text-left NYTresult'><a href='" + resultsArray[i].url + "'>" + resultsArray[i].headline +"</a><p>" + resultsArray[i].leadParagraph + "</p></td></tr>").insertBefore('#NYT-last-row');
  5.             } 
  6.         }
  7.     };
  8. var resultsPrintGuardian = function(resultsArray){
  9.         for (var i = 0; i < resultsArray.length; i++){
  10.             if (resultsArray[i] !== undefined) {
  11.                 $("<tr id='guardianResult"+resultsArray[i].index+"'><td class='text-left guardianResult'><a href='" + resultsArray[i].url + "'>" + resultsArray[i].headline +"</a><p>" + resultsArray[i].leadParagraph + "</p></td></tr>").insertBefore('#guardian-last-row');
  12.             } 
  13.         }
  14.     };
  15. var resultsPrintWikipedia = function(resultsArray){
  16.         for (var i = 0; i < resultsArray.length; i++){
  17.             if (resultsArray[i] !== undefined) {
  18.                 $("<tr id='wikipeidaResult"+resultsArray[i].index+"'><td class='text-left wikipediaResult'><a href='" + resultsArray[i].url + "'>" + resultsArray[i].headline +"</a><p>" + resultsArray[i].leadParagraph + "</p></td></tr>").insertBefore('#wikipedia-last-row');
  19.             } 
  20.         }
  21.     };

The results print functions above will take our resultsArrays and iterate over them creating new table rows for our results display divs and then inserting them. The result is that the relevant properties of all the objects in our results arrays are displayed in the correct position in the DOM. And we’re done! A fully functioning AJAX powered API search engine!

If you have reached this point and are looking to make improvements here are a few suggestions:

  • Right now the results from the different API calls display as soon as they are processed. As the different API calls take different amounts of time to complete it means that front end is not as clean as it could be. It could look much better if all the results displayed at the same time. Look into jQuery's When Then to accomplish this though there are a number of other ways!
  • Add another API… or ten other API’s … or look into YQL which might be able to let you return results not from a specific API but from YAHOO’s search engine.
  • Add a feature where the users can add and delete APIs from the results they see. Maybe some people want results from The New York Times but not from Wikipedia for example.