Products

Nov 17 2020

INT’s GeoToolkit.JS 2020.1 Release Brings Major Features in 2D/3D Visualization and Optimized Rendering in Web Browser

The release also includes many new tutorials and demos for ES6, new features around WebGL, WebAssembly, and more.

Houston, TX —INT is pleased to announce the newest release of GeoToolkit.JS data visualization libraries and toolkit. This major release includes new 3D visualization capabilities—contour 3D, 3D shape highlighting, and better 3D seismic—and new rendering options.

“Visualizing seismic data in the cloud is a challenge for many oil and gas companies. We have been working closely with our developers and partners to find ways to optimize seismic visualization and workflows for our users,” said Dr. Dmitriy Lukyanenko, Director of Software Development at INT. “We’ve added server-side rendering and improved our decompression and compression techniques to speed up performance and have optimized our seismic reader. We think these improvements will result in significant performance gains for our clients.”

GeoToolkit.JS 2020.1 also includes improved capabilities such as combined Vertical and Deviated Schematics, more robust BHA editor, new heatmap layer, and seismic decompression and filters with WebAssembly.

RELEASE HIGHLIGHTS:

Server-Side Rendering Capabilities — To offer a complete set of options for great performance for interacting with seismic in the cloud, we added support for server-side rendering in addition to the existing widget for client-side rendering by just integrating new services
Improved ES6 Support — Full set of all-new tutorials and demos for ES6 with code snippets. Supports Angular, React, and Vue frameworks.
Redesigned the Graphical Annotation — A new version of annotation tool with layer support to add text and graphical annotations, plus a demo to demonstrate this functionality to facilitate adoption
New 3D Features and Faster 3D Rendering — New contour 3D (in addition to existing 2D), optimized 3D seismic visualization (same API but faster rendering), and new 3D shape highlighting, and multilateral trajectory
MultiWell Performance Enhancements — Improved performance of MultiWellWidget if the same template is used for all wells
WebGL — Optimized deviated image and gradients and array log (log2d) with WebGL.

Read the press release on PRWeb >

For more information about GeoToolkit.JS or INT’s other data visualization products, please visit www.int.com.

Nov 09 2020

Human Friendly Error Handling in the IVAAP Data Backend

As the use cases of IVAAP grow, the implementation of the data backend evolves. Past releases of IVAAP have been focused on providing data portals to our customers. Since then, a new use case has appeared where IVAAP is used to validate the injection of data in the cloud. Both use cases have a lot in common, but they differ in the way errors should be handled.

In a portal, when a dataset fails to load, the reason why needs to stay “hidden” to end-users. The inner workings of the portal and its data storage mechanisms should not be exposed as they are irrelevant to the user trying to open a new dataset. When IVAAP is used to validate the results of an injection workflow, many more details about where the data is and how it failed to load need to be communicated. And these details should be expressed in a human friendly way.

To illustrate the difference between a human-friendly message and a non-human friendly message, let’s take the hypothetical case where a fault file should have been posted as an object in Amazon S3,… but the upload part of the ingestion workflow failed for some reason. When trying to open that dataset, the Amazon SDK would report this low-level error: “The specified key does not exist. (Service S3, Status Code: 404, Request ID: XXXXXX)”. In the context of an ingestion workflow, a more human friendly message would be “This fault dataset is backed by a file that is either missing or inaccessible.”

The IVAAP Data Backend is written in Java. This language has a built-in way to handle errors, so a developer’s first instinct is to use this mechanism to pass human friendly messages back to end users. However, this approach is not as practical as it seems. The Java language doesn’t make a distinction between human-friendly error messages and low-level error messages such as the one sent by the Amazon SDK, meant to be read only by developers. Essentially, to differentiate them, we would need to create a HumanFriendlyException class, and use this class in all places where an error with a human-friendly explanation is available.

This approach is difficult to scale to a large body of code like IVAAP’s. And the IVAAP Data Backend is not just code, it also comes with a large set of third-party libraries that have their own idea of how to communicate errors. To make matters worse, It’s very common for developers to do this:

    try {

             // do something here

        } catch (Exception ex) {

            throw new RuntimeException(ex);

        }

This handling wraps the exception, making it difficult to catch by the caller. A “better” implementation would be:

try {

             // do something here

       } catch (HumanFriendlyException ex) {

            throw ex;

        } catch (Exception ex) {

            throw new RuntimeException(ex);

        }

While this is possible to enforce this style for the entirety of IVAAP’s code, you can’t do this for third party libraries calling IVAAP’s code.

Another issue with Java exceptions is that they tend to occur at a low-level, where very little context is known. If a service needs to read a local file, a message “Can’t read file abc.txt” will only be relevant to end users if the primary function of the service call was to read that file. If reading this file was only accessory to the service completion, bubbling up an exception about this file all the way to the end-user will not help.

To provide human-friendly error messages, IVAAP uses a layered approach instead:

High level code that catches exceptions reports these exceptions with a human friendly message to a specific logging system
When exceptions are thrown in low level code, issues that can expressed in a human friendly way are also reported to that same logging system

With this layered approach where there is a high-level “catch all”, IVAAP is likely to return relevant human friendly errors for most service calls. And the quality of the message improves as more low-level logging is added. This continuous improvement effort is more practical than a pure exception-based architecture because it can be done without having to refactor how/when Java exceptions are thrown or caught.

To summarize, the architecture of IVAAP avoids using Java exceptions when human-friendly error messages can be communicated. But this is not just an architecture where human-friendly errors use an alternate path to bubble up all the way to the user. It has some reactive elements to it.

For example, if a user calls a backend service to access a dataset, and this dataset fails to load, a 404/Not Found HTTP status code is sent by default with no further details. However, if a human friendly error was issued during the execution of this service, the status code changes to 500/Internal Server Error, and the content of the human friendly message is included in the JSON output of this service. This content is then picked up by the HTML5 client to show to the user. I call this approach “reactive” because unlike a classic logging system, the presence of logs modifies the visible behavior of the service.

With the 2.7 release of IVAAP, we created two categories of human friendly logs. One is connectivity. When a human friendly connectivity log is present, 404/Not Found errors and empty collections are reported with a 500/Internal Server Error HTTP status code. The other is entitlement. When a human friendly entitlement log is present, 404/Not Found errors and empty collections are reported with a 403/Forbidden HTTP status code.

The overall decision on which error message to show to users belongs to the front-end. Only the front-end knows the full context of the task a user is performing. The error handling in the IVAAP Data Backend provides a sane default that the viewer can elect to use, depending on context. OSDU is one of the deployments where the error handling of the data backend is key to the user experience. The OSDU platform has ingestion workflows outside of IVAAP, and with the error reporting capabilities introduced in 2.7, IVAAP becomes a much more effective tool to QA the results of these workflows.

For more information on INT’s newest platform, IVAAP, please visit www.int.com/products/ivaap/

Nov 06 2020

How to Get the Best Performance out of Your Seismic Web Applications

One of the most challenging data management problems faced in the industry is with seismic files. Some oil and gas companies estimate that they acquire a petabyte of data per day or more. Domain knowledge and specific approaches are required to move, access, and visualize that data.

In this blog post, we will dive deep into the details of modern technology that can be useful to achieve speed up. We will also cover: common challenges around seismic visualization, how INT helps solve these challenges with advanced compression and decompression techniques, how INT uses vectorization to speed up compression, and more.

What Is IVAAP?

IVAAP is a data visualization platform that accelerates the delivery of cloud-enabled geoscience, drilling, and production solutions.

IVAAP Client offers flexible dashboards, 2D & 3D widgets, sessions, and templates
IVAAP Server side connects to multiple data sources, integrates with your workflows, and offers real-time services
IVAAP Admin client manages user access and projects

Server – Client Interaction

Interaction occurs when the client requests a file to display from the server, the server returns the file lists, the user chooses a file to display, and then the server starts sending chunks of data while it displays this data.

Some issues encountered with this scheme include:

Seismic data files are huge in size — they can be hundreds of gigabytes or even terabytes.
Because of the file size, it takes too much time to transfer files via network.
The network can have too much bandwidth.

The goals of this scheme are to:

Speed up file transfer time
Reduce data size for transfer
Add user controls for different network bandwidth

And the solution:

We decided to implement server-side compression and client-side decompression. We also decided to provide the client parameter that we call acceptable error level after the seismic data file compression/decompression process.

By taking a closer look at compression and decompression data, we can see that the original seismic data goes through a set of five transformations — AGC, Normalization, Hear Wavelets, Quantization, and Huffman. As a result of this transformation, we get a compressed file that can be sent to clients via network. And on the client’s side, there is a decompression process that goes in different directions — from inverse Huffman to inverse AGC. This is the way that clients get original data. It does not get precise, original data. But it gets data after the compression and decompression process. That’s why we added an acceptable error level after the compression and decompression process. This is because we have different scenarios where clients don’t always require the full original data with the full level of precision. For example, sometimes the client only needs to review the seismic data. So using this acceptable error level, they can control how much data will be passed by a network and, of course, speed up this process.

The resulting scheme looks like this:

The client requests a file list from the server, the user chooses a file to display, and then the server starts sending the data and compresses it. The server then sends it to the client, the client decompresses, and finally, it displays the data. This is repeated for each tile to display.

So why not use any other existing compression, like GZIP, LZ Deflate, etc.? We tried these compressions, but we found out that this type of compression is not as effective as we’d like it to be on our seismic data.

Server-Side Interaction

The primary objective was to speed up the current implementation of compression and decompression on both the server and client side.

The proposal:

Server-side compression is implemented in Java, so we decided to create C++ implementation of compression sequence and use JNI layer to call native methods. For the client-side decompression, we implemented in JavaScript to create C++ implementation of decompression and use WebAssembly (WASM) proposal for integrating C++ code into JS.
We implemented both compression and decompression algorithms in C++, but after comparing the results and performance of C++ and Java, we discovered that C++ was just 1.5 times faster than “warmed up JVM”. That’s why we decided to move on and apply SIMD instructions for further speedup.

Single Instruction Multiple Data (SIMD)

SIMD architecture performs the same operation on multiple data elements in parallel. For Scalar operation, you have to perform four separate calculations to get the right result. For SIMD operations, you apply one vector value calculation to get the correct result.

SIMD benefits:

Allows processing of several data values with one single instruction.
Much faster computation on predefined computation patterns.

SIMD drawbacks:

SIMD operations cannot be used to process multiple data in different ways.
SIMD operations can only be applied to predefined processing patterns with independent data handling.

Normalization: C++ scalar implementation

Normalization: C++ SIMD SSE implementation

Server-Side Speedup Results

There are different types of speedup for different algorithms:

Normalization is 9 times faster than the scalar C++ version
Haar Wavelets is 6 times faster than the scalar C++ version
Huffman has no performance increase (not vectorizable algorithm)

Overall, the server-side compression performance improvement is around 3 times faster than the Java version. This is applying SIMD C++ code. This was good for us, so we decided to move on to the client-side speedup.

Client-Side Speedup

For the client-side speedup, we implemented decompression algorithms in C++ and used WASM to integrate the C++ code in JavaScript.

WebAssembly

WASM is:

A binary executable format that can run in browsers
A low-level virtual machine
A high-level language compile result

WASM is not:

A programming language
Connected to the web and cannot be run outside the web

Steps to get WASM working:

Compile C/C++ code with Emscripten to obtain a WASM binary
Bind WASM binary to the page using a JavaScript “glue code”
Run app and let the browser instantiate the WASM module, the memory, and the table of references. Once that is done, the WebApp is fully operative.

C++ Code to Integrate (TaperFilter.h/cpp)

Emscripten Bindings

WebAssembly Integration Example

Client-Side Speedup Takeaways:

Emscripten supports the WebAssembly SIMD proposal
Vectorized code will be executed by browsers
The results of vectorization for decompression algorithm are:
- Inv Normalization: 6 times speedup
- Inv Haar Wavelets: 10 times speedup
- Inv Huffman: no performance improvement (not vectorizable)

Overall, the client-side decompression performance improvement with vectorized C++ code was around 6 times faster than the JavaScript version.

For more information on GeoToolkit, please visit int.com/geotoolkit/ or check out our webinar, “How to Get the Best Performance of Your Seismic Web Applications.”

Oct 07 2020

Search, Access, and Visualize OSDU Data with IVAAP

INT and OSDU Collaboration

INT has been working with the Open Subsurface Data Universe™ (OSDU) consortium since the beginning of the OSDU forum in 2019. OSDU is a standard data platform for the oil and gas industry which will reduce silos and put data at the center of the subsurface community. INT made IVAAP available as part of the Demo release of the OSDU platform.

IVAAP: The Premier HTML5 Data Visualization Platform for OSDU

At INT, we worked closely with the OpenSDU team to assist with validating the platform and creating a robust framework.

Developers can deliver cloud-based digital solutions quicker than ever.
Using the powerful OSDU search and delivery API, IVAAP allows users to search, find, display, and analyze data on the fly, including log, trajectories, tops, seismic, horizon, and fault data.
Creating a common environment hosted in the cloud enables oil and gas companies to efficiently access massive amounts of data, reduce data silos, collaborate remotely, implement machine learning (ML), and lower the cost of operations.

Search and interact with different data types.

Access your data in the cloud from anywhere.

Visualize data aggregated from various data sources in one place.

With IVAAP on OSDU R2, you can:

Combine data on the fly from wells, seismic, and many other sources.
Use a microservices architecture, IVAAP supports industry data standards, including OSDU, PPDM, WITSML, and several commercial databases.
Enable user co-visualization.
Combine data types like real-time, historical, 2D/3D seismic, reservoir, and more in your web browser with IVAAP’s one-of-a-kind dashboard capabilities.

Request a demo for IVAAP today

What Is OSDU?

The OSDU Forum was created with the objective of enabling new cloud-native data-driven applications with seamless access to the full range of subsurface and wells data as well as supporting existing applications and data frameworks.

Separating data from applications is the core principle of the OSDU solution. The plan is to achieve this by developing a common data platform with standard public APIs and to involve global cloud hosting vendors to build working implementations that will:

Allow companies to focus on the truly differentiating part of business activities
Reduce the resource and cost burden of independently designing and validating a subsurface data platform
Influence and accelerate key reference components that will be the foundation of the upstream digital transformation.

The OSDU data platform will:

Enable secure, reliable, global, and performant access to all subsurface and wells data
Reduce current data silos to enable transformational workflows
Accelerate the deployment of emerging digital solutions for better decision-making
Create an open, standards-based ecosystem that drives innovation

View the official OSDU brochure.

An Industry Solution

The OSDU Forum will:

Publish a Reference Architecture: a cloud-native subsurface and wells data platform reference architecture with initial implementations by Microsoft^® Azure, Amazon Web Services^® and Google^® GCP, with others expected.
Define Application Standards to ensure applications (microservices), developed by various parties, can run on the same OSDU data platform
Leverage Industry Data Standards for frictionless integration and data access

Business Impact

Reducing data silos: subsurface and wells data integrated and accessible in one data platform
Workflows in the cloud: user workflows seamlessly executed across the OSDU data platform, taking advantage of powerful cloud-native solutions
Access to all metadata: users will have access to powerful search capabilities
Competition and cooperation: software vendors, oil companies, academia, and open community projects can develop software on the OSDU data platform

Scope

Current Scope

Coverage: any organization with a need to store, manage or analyze subsurface and wells data
Portfolios: subsurface including exploration, development, and wells
Support for Data Driven Applications: All data in one Data Platform will enable AI (such as Machine Learning) based applications to access this data

Current Problem Statement

Data is linked to applications: we are not data-driven but applications-driven and, therefore, we will have silos
Data is stored in silos: therefore, there is no lineage between data sources or from wells back to exploration
Metadata is not stored with the data: metadata is information about the data
Limited/no search capabilities: since we have no metadata, we have no real search capabilities
Data set-up: not suitable for data-driven applications

OSDU Scope

All public cloud-based: provides unlimited scalability
All data for the data platform services loaded via the data ingestion services: extract metadata, data quality, etc.
Well-defined (RestFUL) APIs defining the access to the data platform services
Applications: (micro) services-based
Information security: covering all elements with focus on data security
In-country solution: for those countries where we are unable to move data out of the country and where there are no cloud services
Support for legacy applications based on Microsoft^® Windows or Linux^® desktop

What Is the OSDU Value?

The OSDU data platform will:

Revolutionize the industry’s ability to deliver new subsurface capabilities
Create an open, standards-based ecosystem that drives innovation
Accelerate the deployment of emerging digital solutions for better decision-making
Reduce implementation and lifecycle costs
Eliminate current data silos to enable transformational workflows
Provide access to all metadata: users can access their data and applications wherever they are
Create new opportunities for monetizing intellectual property:
- Moving to subscription models for applications or services sold by independent software and services providers
- Opportunity for multiple competitors to sell specialized algorithms or workflows that are easily integrated because of compliance to OSDU standards

Membership

If you’re interested in becoming a member of OSDU today, click here, or for a full list of members, click here.

Ready to Get Started?

Request a demo of the platform chosen to visualize OSDU data.

Request a demo for IVAAP today

Sep 15 2020

Integrating Powerful Map Capabilities into Your Subsurface Web Applications

Map-based search is an integral part of subsurface data visualization. In order to meet usability expectations, developers of subsurface applications in the cloud must add powerful map and map-based search functionalities.

The GeoToolkit map widget simplifies the process, allowing users to get quick and clear insights using common web mapping services. In this blog post, we will cover how to access the map widget in GeoToolkit, how to integrate other Web Services, including ArcGIS, ESRI REST, OpenStreetMap, Google, Microsoft, etc., and more about GeoToolkit’s features, including layering, labeling, symbols, annotations, and more.

Features

With GeoToolkit.JS, there are many general functionalities that you can use in your web application. The map, which is based on the Core toolkit, provides two different versions of libraries: it has ECMAScript 6 or if you’re going for a more classical approach, ECMAScript 5. It can be used in part with different UI frameworks like React, Angular, or VUE. Moreover, it has user-friendly functionalities like symbol aggregation, label collision, export to PDF, and imaging formats.

There are also different functionalities that can be available such as axes, titles, and scrollbars. The predefined interaction tools allow you to display crosshair, panning tools, zoom to measure distance, tooltip, rubberband, and more. GeoToolkit.JS map supports a wide range of map formats and services like ArcGIS, GeoJSON, KML, Mapbox, Bing, and so on. It is also compatible with other GeoToolkitJS elements like charts, shapes, and widgets.

Getting Started

import {Map} from '@int/geotoolkit/map/Map';
import {Plot} from '@int/geotoolkit/plot/Plot';
import {Tile} from '@int/geotoolkit/map/layers/Tile';
const map = new Map();
map.addLayer(new Tile({
 'name': 'OpenStreetMap',
 'url': https://demo.int.com/osm_tiles/{z}/{x}/{y}.png
}))
const plot = new Plot({
  'canvaselement': canvas,
  'root': map
});

Create a map widget.
Connect DOM canvas with the widget by creating a Plot done like other widgets.
Add a layer (or layers) of interest to the widget.

Web Services: ESRI REST, OpenStreetMap, Google, Microsoft

It is important to consider which types of services can be supported since all of your data is received from different types of servers. Some services hosted include: ESRI Rest, ArcGIS, OpenStreetMap, Google Map, Microsoft, etc.

The ArcGIS Web Map protocol which is used by ArcGIS online is an easy and convenient way to build your map application. You can go to ArcGIS online, create your content, put necessary layers, combine them together, provide the link to our map widget and it will automatically be recognized. Most layers are supported. Image services (ArcGIS Image Service Layer, Image Services Vector Layers, and WMS) and Tile services (Image Service, ArcGIS Tiled Map Service Layer, Web Tiled Layer, OpenStreetMap) are supported. Feature Services include map service (ArcGIS Feature Layer), KML, WFS, and CSV. We also provide some real-time services support like stream services (ArcGIS Stream Layer), GeoRSS, Vector Tiles (VectorTileLayer), and Bing Maps services. Two extra formats that are supported are GeoJSON and GPX.

Map Services

Services cannot be used without visualization. Our part in maps products is to provide visualization for these services. To start visualization, you need to create your map widget and specify different properties. One example of a system is CRS, which is a common coordinate system of displayed data. You can specify map limits if you want to limit the visualization area of interest. You can also set different adornments to your maps like map scale. Zoom settings include min/max with a range of scales available and time/speed to customize the map management to your convenience.

Map Layers

There are four main types of layers supported:

Image type that displays a single image received from a server. For example, WMS for Web Maps Services and ArcGISImage supports ArcGIS MapServer and ImageServer services.
With Tile layers, the data consists of several images (tiles) painted next to each other and thus forming a complete picture. Tile can be used as a universal layer for any tile service. Bing can be used for all Microsoft Bing Services.
Vector layers draw not pictures but vector data (called “features”): points, polygons, and polylines, which depict cities, rivers, islands, and more. Support of different formats includes: GeoJSON, KML, CSV, GeoRSS, Lerc translate their format into the map objects, the ArcGISFeature supports ARCGIS FeatureServer services., WFS for Web Feature Services, and VectorTile.
Shape layer is used for compatibility with other GeoToolkitJS elements to display on the map charts, contours, and other complex shapes (or just trivial ones).

Layer Settings

General setting for layers include url to the server or file, data coordinate system (epsg codes are supported), alpha as the value of the layer transparency, layerfilter for setting visibility conditions, tooltip.visible to enable tooltip support by the layer, and tooltip.formatter to generate information (in HTML format) of the tooltip content (can be used in both the basic GeoToolkitJS formatters or a custom one).

Examples of different layers:

Geo-Reference Images

Geotoolkit provides powerful options like fast WebGL implementation and ImageTransform to help speed up the process to transform any image in your application.

Feature Annotations

In map displays, you can have a lot of different annotations and some of them can potentially overlap. To help filter out the overlapped annotations, you want to use some collision detection to remove unnecessary labels. We provide all of these functionalities which can be configured. The steps to display labels are:

Use annotations.visible layer property to include annotations (hidden by default)
To select annotation info, change the annotation.strategy parameter to the AnnotationByAttribute or AnnotationByRule instance with the attribute name or \expression to display.
Set the text shape for the template to customize annotations drawing styles and textSizeInfo option to dynamically resize annotations.
Annotation filters prevent some information from being written in order to save space, time, etc.

Robust Map Features for Your Subsurface Application

Overall, GeoToolkit’s map widget allows you to integrate many robust map features into your subsurface or exploration applications. We hope this helps you simplify the process, meet usability expectations, and get the insights you need.

For more information on GeoToolkit’s maps widget and its features, please visit our GeoToolkit page.

Want to know more? Check out our webinar: Integrating Powerful Map Capabilities into Your Subsurface Web Applications.