DataGaga - A manager for web data sources :earth_africa:

All data sources unite! A uniform JavaScript API for processing multiple data sources across the web. The included data sources can be "fused" together to become one single mashup data source!

Note: This project is under active development, so stay tuned!

:gift: What is it?

Have you ever had too many web services and data sources all over the internet and found it too cumbersome to save every single one of them? You only care about data and not where they are stored and how they can be retrieved?

No worries no more! DataGaga is a library to manage and organize the data sources on the web in one place. We define a general concept to support almost all popular data sources online. Not only can such data sources be imported from different providers and places, they can be combined as one single mashup data source for your convenience. You just need to declare the data source(s) you'd like to include, we take care of the rest.

:globe_with_meridians: How it started

In the field of Geographic Information System (GIS), besides displaying geometric 3D models (like buildings, bridges and tunnels), it is often required to enrich these models with thematic information, such as heights, numbers of stories, construction years, etc. Such data are however often provided from various web services and providers, which introduces many problems:

• The sheer number of the resource URIs will only increase as more and more data sources are involved, and developer sometimes lose track of which URI belongs to which data source and for what purpose.
• Each web data service or provider has their own standard and syntax to access to the hosted data. Morover, the responses also often differ vastly between web services.
• Sometimes an object may have its data stored in different locations and it is difficult to handle all of these at once.

For example, you'd like to query your own tables hosted in the following services:

• Google Spreadsheets
• PostgreSQL
• Oracle

Normally you would have to treat them separately in your code because they have different API syntax structures and protocols. Using our tool:

• You only need to declare these data sources once;
• You can then query your data easily in one place regardless of the syntax and structure differences between the different APIs;
• You can even treat all of them as one single data source and query/display their data in one single call.

:electric_plug: Installation

npm

This is a NodeJS project and can therefore be easily installed using npm:

npm install datagaga

Have a look at the npm page for more details.

Web project

Alternatively, the entire source codes are built into one single JavaScript file DataGaga.js and can be imported to your web projects with ease. (We are working on providing an import link such as or similar to CDN.)

:hammer: Build manually

Simply compile the all the TypeScript files in folder src. This will generate one single JavaScript file DataGaga.js located in the folder build.

Note: You must use the option "outFile": "build/DataGaga.js" (instead of outDir) in tsconfig.json. Also use either "module": "amd" or "module": "system" instead.

:monkey: How do I use this?

1. :seedling: First declare an options object for your data source:

   var options = {
       name: "My Data Source 1",
       type: "Table",
       provider: "The Provider",
       uri: "https://link.to.your.datasource"
   };

   where:

   • name (optional): the name of your data source
   • type (optional): the type of your data source
   • provider (optional): the provider of your data source
   • uri (required): the URL to your data source

2. :herb: Initialize data source:

   var googleSheetsDataSource = new GoogleSheets(options);
   var postgreSQLDataSource = new PostgreSQL(options);
   var kmlDataSource = new KML(options);
      
   // alternatively, you could use the controller
   var googleSheetsDataSource = DataGaga.createDataSource("GoogleSheets", options);
   var postgreSQLDataSource = DataGaga.createDataSource("PostgreSQL", options);
   var kmlDataSource = DataGaga.createDataSource("KML", options);

3. :palm_tree: (Optional) If needed, the created data sources can be combined into one single mashup data source:

   var mashupDataSource = new MashupDataSource(options);
   mashupDataSource.addDataSource(googleSheetsDataSource);
   mashupDataSource.addDataSource(postgreSQLDataSource);
   mashupDataSource.addDataSource(kmlDataSource);
      
   // to remove KML data source
   mashupDataSource.removeDataSource(2);

4. :tada: Start querying, both individual and mashup data sources have the same methods:

   mashupDataSource.fetchAttributeValuesFromId(id)
   .then(function(result) {
       console.log(result);
   }.catch(function(error) {
       console.warn(error);
   }));

   where:

   • id required: the ID of the object to be queried

:muscle: How do I declare my own data sources?

For your convenience, we already include Google Spreadsheets and PostgreSQL (more to come!) both in JavaScript and TypeScript. But if you know a popular data source that is not included, please let us know in the issue section, and perhaps we could define it for you!

Otherwise if you are a power user, you can declare your own data source(s) in a TypeScript file. Have a look at our codes here. We are working on a more detailed documentation on this, so stay tuned!