8 years ago in javascript, blog · 3 MIN READ

USGS's Hidden Interactive Deaggregations api

UPDATES:

I made a Python version of this downloader, running in parallel, would save a lot of time. Could be found in my github, here.

==========================================================================================

USGS has some very useful online seismic hazard analysis tools. But some apis are hidden. The one I am using is 2008 Interactive Deaggregations .

Basically, if you want to get the processed data for one single location, you need to input the (lat,lon) pair, exceedance probability, Vs30...by typing, to this tool and it will return some files to you.

The problem I meet is: I have 221x6 locations. For each location, I spend about 56 seconds (including typing the parameters and waiting for calculation and downloading files to my disk) to get the files I need. So I will use about 21 hours to finish it, without any faults, any interval. That's a lot of work. So to save time and labor I use the hidden api to grab data automatically.

The 2008 interactive deaggregations tool is actually a html form, when you click to submit this form, a javascript function will capture your action and send your query to USGS server. After calculation, links to the locations of the process data files will be shown to you for downloading.

So, I need to know 2 things:

1. where to send my query?

2. how to parse the data the server returned?

The answers are easy to find in:

http://geohazards.usgs.gov/deaggint/2008/js/index.js

The url to receive queries can be found at line 344

var url = BASE_URL + '/application.php?';

Which actually is

http://geohazards.usgs.gov/deaggint/2008//application.php?

Now I will write some js to send my query to the server and parse the data returned.

I open google chrome, open http://geohazards.usgs.gov/deaggint/2008/, press F12 on the keyboard to open the debug console.

Then I input two functions:

function hackFile(data) {

BASE_URL = 'http://geohazards.usgs.gov/deaggint/2008';

name = data.name;

latitude = data.latitude;

longitude = data.longitude;

percent = data.percent;

years = data.years;

sa = 1;

gmpe = 1;

vs30 = data.vs30;

geo = 1;

var url = BASE_URL + '/application.php?';

url += 'name=' + name;

url += '&latitude=' + latitude;

url += '&longitude=' + longitude;

url += '&percent=' + percent;

url += '&years=' + years;

url += '&sa=' + sa;

url += '&gmpe=' + gmpe;

url += '&geo=' + geo;

url += '&vs30=' + vs30;

$.get(url, function (_response) {

   if (status == 0) {

       var item = document.createElement('li');

       var downloads = '';

       var returnedFile = '';

       //alert($(this).attr('url'));

       var counter = 0;

       $('link', $(_response)).each(function () {

           if (counter < 1) {

               //alert($(this).attr('url'));

               returnedFile = $(this).attr('url');

               if (percent < 10) {

                   var writeOutName = name + '_0' + percent + '_' + years;

               } else {

                   var writeOutName = name + '_' + percent + '_' + years;

               }

               SaveToDisk(returnedFile, writeOutName);

           }

           counter += 1;

           downloads += '<a href="' + $(this).attr('url') + '" title="' +

           $(this).attr('title') + ' ' + $(this).attr('size') +

           '" target="appOutputWindow">' + $(this).attr('display') +

           '</a>';

       });

       downloads = downloads.replace(/></g, '> | <');

       var itemContentStr = '<span class="downloads">[ ' + downloads +

               ' ]</span>';

       itemContentStr += '<span class="name">' +

       $('name', $(_response)).text() + '</span>';

       itemContentStr += '<span class="meta">' +

       $('metadata', $(_response)).text() + '</span>';

       $(item).append($(itemContentStr));

       $('#appOutputList').prepend(item);

       window.IO.closeDialog();

   } else {

       // Error, just alert the error message.

       var ul = document.createElement('ul');

       addClass(ul, 'error');

       $(ul).append($($('metadata', $(_response)).text()).html());

       window.IO.closeDialog();

       window.IO.alert(ul, {ctype: 'text/html', title: 'Error'});

   }

   $('#btnCompute').attr('src', 'images/btnCompute.gif');

});

}

function SaveToDisk(fileURL, fileName) {

//I found this download function here:

//http://muaz-khan.blogspot.com/2012/10/save-files-on-disk-using-javascript-or.html

// for non-IE

if (!window.ActiveXObject) {

   var save = document.createElement('a');

   save.href = fileURL;

   save.target = '_blank';

   save.download = fileName || 'unknown';

   var event = document.createEvent('Event');

   event.initEvent('click', true, true);

   save.dispatchEvent(event);

   (window.URL || window.webkitURL).revokeObjectURL(save.href);

}

// for IE

else if (!!window.ActiveXObject && document.execCommand) {

   var _window = window.open(fileURL, '_blank');

   _window.document.close();

   _window.document.execCommand('SaveAs', true, fileName || fileURL)

   _window.close();

}

}

I use hackFile to send query and parse data, use SaveToDisk to download files to my disk. The SaveToDisk is function is originated from here:

http://muaz-khan.blogspot.com/2012/10/save-files-on-disk-using-javascript-or.html

Now I can input my parameters and start to communicate with the USGS server:

var nameData = ['Location 1', 'Location 2']

var LatData = [34, 35];

var LonData = [-121, -122];

var Vs30Data = [300, 400];

var data = new Array;

for (i = 0; i < nameData.length; i++) {

data[i] = {

   name: nameData[i],

   latitude: latData[i],

   longitude: LonData[i],

   percent: 50,

   years: 75,

   vs30: Vs30Data[i]

};

}

var j = 0;

setInterval(function () {

j += 1;

{

   hackFile(data[j - 1]);

}

}, 20000);

Now Chrome starts to download files to my disk and the job will be done in 70 mins.

···