JSON Viewtool

The JSONTool allows you to call local or remote json-based RESTful web services and parse the results for inclusion in a velocity template, page or widget. The following examples show how to use the JSONTool and JSONObjects with JSON in dotCMS.

Methods

The JSONTool is accessed using the $json object. The following methods are supported:

Important Limitations

Remote URL Limitations

Performance

$json.fetch() is a synchronous call and is only as reliable and responsive as the remote URL accessed.

• If the remote URL has a latency of a few seconds, then every dotCMS Page that accesses the URL is going to hang for a few seconds before the Page can render.
• In some cases (such as a high traffic site that uses a $json.fetch call on every page, for example in a header or footer), this could lock up all threads on the server, causing the entire server to hang or go down.

Access Frequency

$json calls will be performed every time a Page which contains them is accessed.

• If the remote URL has limitations on the number of calls which can be made (due to rate limiting, performance considerations, limits to the number of calls within a given time period, etc.), placing a $json call directly on a page may cause the remote URL to be accessed too frequently.

Handling Remote URL Limitations

There are several ways to both reduce performance issues when accessing JSON data from a remote URL, and to limit the number of calls made to the remote URL. It is strongly recommend that, whenever possible, you use one or more of these methods when using the JSONTool:

• When using $json.fetch(), always supply a reasonable time out value.
  • What time out makes sense will depend on both the load on your dotCMS server, and the responsiveness of the remote URL.
  • However, time outs on production servers should almost never be more than one second.
• Wrap all $json calls in a dotCache directive.
  • This both limits the load on the dotCMS server, and limits the number of calls to the remote URL.
• Access the remote URL from the client (using Javascript) instead of the server (using Velocity).
  • Any performance issues with the remote URL will cause delays in the client's browser, but will not block any threads on the dotCMS server.
  • Note, however, that this does not reduce the number of calls made to the remote URL.
• Retrieve and store the output of the remote URL as regular content, then render the results directly using the content (instead of using a $json call).
  • The URL can be accessed and content can be created via a cron job or other methods.
  • This allows the content to be updated asynchronously, ensuring server and page performance will not be impacted if the URL is not accessible for long periods of time.
• A hybrid approach.
  • For example, a robust solution might include a Scripted Custom Endpoint which fetches the results from the remote URL (using both a timeout on the $json.fetch() call and a dotCache block), and Javascript on the client to retrieve the results from the Scripted Endpoint.

Do Not Perform $json Calls to the dotCMS Server

For both performance and reliability reasons, it is strongly recommended that you never perform $json calls to the dotCMS server from within the dotCMS server's own Velocity code.

• A single request of this type causes several internal requests, so a single call multiplies the load on the server.
• Domain names and ports that dotCMS runs on can change.
• A dotCMS server running behind load balancers or firewalls might not be able resolve an external domain name (https://server.domain.com/endpoint) as an internal resource (/endpoint).
• The ports that dotCMS server listens on (internally) might change.

Retrieve and Parse External JSON ($json.fetch and $json.post)

The $json.fetch(), $json.put(), $json.post() methods both retrieve and parse JSON from a URL you supply. The methods return a JSON object whose properties can be traversed via the . (dot) operator.

These two methods behave very similar in most respects. For example, both methods allow you to retrieve JSON data from an external web API, both allow you to send headers with the request, and both return a traversible JSON object. There are 2 main differences between the fetch() and post() methods:

• The fetch() method uses an HTTP GET to submit the request, while the post() method uses an HTTP POST.
• Since it uses a POST instead of a GET, the post() method allows you to send data with the request.

Usage

The $json.fetch() method calls HTTP GET and can be called in any of the following ways:

$json.fetch( URL )
$json.fetch( URL, Timeout )
$json.fetch( URL, Headers )
$json.fetch( URL, Timeout, headerMap )

The $json.post() method calls HTTP POST and can be called in any of the following ways:

$json.post( URL, headerMap, rawData )
$json.post( URL, Timeout, headerMap, rawData )
$json.post( URL, headerMap, paramMap )
$json.post( URL, Timeout, headerMap, paramMap )

The $json.put() method calls HTTP PUT and can be called in any of the following ways:

$json.put( URL, headerMap, rawData )
$json.put( URL, Timeout, headerMap, rawData )
$json.put( URL, headerMap, paramMap )
$json.put( URL, Timeout, headerMap, paramMap )

Timeout Parameter

The timeout specifies the total time to wait to both make the connection and perform the read. The Import Tool will automatically kill the connection after the specified time, regardless of what state the import is in (even if it is in the middle of a read). Therefore the timeout should be greater than the time it takes for both the connection and read of the JSON resource.

Note:

• The timeout is specified in milliseconds (not seconds).
• If no timeout value is specified, the default timeout is used.
  • The default value may be changed by modifying the URL_CONNECTION_TIMEOUT property in the dotmarketing-config.properties file.
  • Important: It is strongly recommended that all changes to the dotmarketing-config.properties file be made through a properties extension file.

Headers Parameter

You may pass headers to be sent with the JSON request. This may be necessary, for example, to perform a fetch from a server that restricts access to certain types of clients (such as regular browsers) by checking the User Agent header.

The headers must be in a map, with each header to be passed consisting of both a header label and a header value.

The following code demonstrates how to generate and use the headers map:

#set($headers = {})
#set($dummy = $headers.put("User-Agent", "Mozilla/5.0"))
#set($results = $json.fetch("http://www.omdbapi.com/?t=Battlestar%20Galactica&y=&plot=short&r=json", $headers))

Form Parameters

You may pass form params to be sent with the request. This may be necessary, for example, to POST data to a server.

The form params must be in a map, with each param to be passed consisting of both a string key and a string value.

The following code demonstrates how to generate and use the param map:

#set($headers = {})
#set($params = {})
#set($dummy = $params.put("firstName", "Will"))
#set($dummy = $params.put("lastName", "Ezell"))
#set($results = $json.put("https://demo.dotcms.com/testing",$headers, $params))

Raw Data

You can send raw json data as well - just pass the json in as a fully formed json string

#set($headers = {})
#set($data = '{"firstName":"Will","lastName":"Ezell"}')
#set($results = $json.put("https://demo.dotcms.com/testing", $headers, $data))

Example

The following example fetches a JSON object from an external site and displays different properties of the object.

#set($myjson = $json.fetch("http://www.omdbapi.com/?t=Battlestar%20Galactica&y=&plot=short&r=json"))

<p>The title is: $myjson.Title</p>
<p>The year is: $myjson.Year</p>
<p>The rating is: $myjson.Rated</p>

The results of the code above are:

The title is: Battlestar Galactica
The year is: 2004–2009
The rating is: TV-14

Parsing Arrays and Loops

If the JSON property is an array, you can loop over the array using Velocity's #foreach directive:

#set($myjson = $json.fetch("http://imdbapi.poromenos.org/json/?name=battlestar%"))

#foreach($show in $myjson.shows)
    <p>$show.name ($show.year)</p>
#end

The code above lists every version of the Battlestar Galactica television show, including the year it was broadcast.

Limiting JSON Results

To limit the number of results returned in the JSON query, use Velocity $foreach.count and #break directives to exit the #foreach loop as soon as the count is met. The following example limits the previous code to only display 3 results:

#set($myjson = $json.fetch("http://imdbapi.poromenos.org/json/?name=battlestar%"))

#foreach($show in $myjson.shows)
   #if( $foreach.count > 3 )
      #break
   #end
   <p>$show.name ($show.year)</p>
#end

Handling Alternate JSON Array Formats

The JSONTool handles JSON arrays in certain specific formats automatically; however JSON arrays can be formatted in different ways, and for some JSON array formats, additional processing may be required to be able to iterate through the array results. If you can not iterate over a JSON array using the standard loop syntax (above), you must first read the JSON array and create a JSON object from it, and then iterate over the JSON object, as in the following example:

#set($jsonData = $import.read("http://imdbapi.poromenos.org/json/?name=battlestar%"))
#set($jsonData = "{data:${jsonData}}")
#set($jsonObject = $json.generate($jsonData))
#foreach($show in $jsonObject.data)
   <p>$show.name ($show.year)</p>
#end

Passing Data to a Post

The format and contents of the Data object passed with the post() method must match the requirements of the API/URL the post is being sent to. The Data object is created in the same way as the Headers object used in both the fetch() and post() methods.

#set($data    = $contents.getEmptyMap())
#set($dummy   = $data.put("Query", "+contentType:news"))
#set($results = $json.post($url, $data))

Create JSON Objects ($json.generate)

The $json.generate() method generates a velocity object or a JSON Object from Java Objects, Strings or Maps. The JSON object values can be populated with static or dynamic content from your dotCMS instance. The $json.generate() method by default generates a velocity object. To have the default back to returning a JSON object.

1. Navigate to the file dotmarketing-config.properties in webapps/ROOT/WEB-INF/classes
2. Open the file and find jsontool.generate.jackson=true
3. Edit the file and change the line above to jsontool.generate.jackson=false
4. Save and publish file.

The default output of $json.generate() is now a JSON object

Example

The following code generates a JSONObject from a java.collections.Map, demonstrating how to serve a JSON object from your site.

#set($mymap = $contents.getEmptyMap())
#set($dummy = $mymap.put("one","value 1"))
#set($dummy = $mymap.put("two","value 2"))
#set($dummy = $mymap.put("three","value 3"))
#set($myjson = $json.generate($mymap))

<p>$myjson</p>

<p>The value of key "one" is: $myjson.one</p>

The code generates the following results:

{"two":"value 2","one":"value 1","three":"value 3"}
The value of key "one" is: value 1

Parse Internal JSON Files

The following example code will allow you to parse JSON files that are stored on the dotCMS file system. Since the #include directive is only aware of files on the system and not files in dotCMS's virtual file system, use $webapi.getAssetPath to convert the virtual path to the system path, and then $json.generate to convert the file contents into a variable.

#define($file)#include($webapi.getAssetPath("/your/dotCMS/file/path/data.json", $host.identifier))#end
#set($data = $json.generate($file.toString()))  

-Thanks to Leonardo Bell for providing this example :-)

Toolbox.xml Configuration

The following example shows how the JSONTool is mapped in the toolbox.xml file:

    <tool>
        <key>json</key>
        <scope>application</scope>
        <class>com.dotcms.rendering.velocity.viewtools.JSONTool</class>
    </tool>