6.7. Advanced Features
**********************


6.7.1. Dynamic substitutions
============================

Dynamic substitution are mark-up placed in element of the scenario.
For HTTP, this mark-up can be placed in basic authentication
(www_authenticate tag: userid and passwd attributes), URL (to change
GET parameter) and POST content.

Those mark-up are of the form "%%Module:Function%%". Substitutions are
executed on a request-by-request basis, only if the request tag has
the attribute "subst="true"".

When a substitution is requested, the substitution mark-up is replaced
by the result of the call to the Erlang function:
"Module:Function({Pid, DynData})" where "Pid" is the Erlang process id
of the current virtual user and DynData the list of all Dynamic
variables (**Warn: before version 1.1.0, the argument was just the
Pid!**).

Here is an example of use of substitution in a Tsung scenario:

   <session name="rec20040316-08:47" probability="100" type="ts_http">
     <request subst="true">
       <http url="/echo?symbol=%%symbol:new%%" method="GET"></http>
     </request>
   </session>

For the http plugin, and since version 1.5.1, you can use the special
value "subst='all_except_body'" instead of "'true'" to skip the
substitutions in the body part of the HTTP response.

Here is the Erlang code of the module used for dynamic substitution:

   -module(symbol).
   -export([new/1]).

   new({Pid, DynData}) ->
      case random:uniform(3) of
          1 -> "IBM";
          2 -> "MSFT";
          3 -> "RHAT"
      end.

Use **erlc** to compiled the code, and put the resulting .beam file in
"$PREFIX/lib/erlang/lib/tsung-X.X.X/ebin/" on all client machines.

As you can see, writing scenario with dynamic substitution is simple.
It can be even simpler using dynamic variables (see later).

If you want to set unique id, you can use the built-in function
**ts_user_server:get_unique_id**.

   <session name="rec20040316-08:47" probability="100" type="ts_http">
     <request subst="true">
       <http url="/echo?id=%%ts_user_server:get_unique_id%%" method="GET" />
     </request>
   </session>


6.7.2. Reading external file
============================

**New in 1.0.3**: A new module "ts_file_server" is available. You can
use it to read external files. For example, if you need to read user
names and passwd from a CSV file, you can do it with it (currently,
you can read only a single file).

You have to add this in the XML configuration file:

   <option name="file_server"  value="/tmp/userlist.csv"></option>

**New in 1.2.2**: You can read several files, using the **id**
attribute to identify each file:

   <option name="file_server" value="/tmp/userlist.csv"></option>
   <option name="file_server" id='random' value="/tmp/randomnumbers.csv"></option>

Now you can build your own function to use it, for example, create a
file called "readcsv.erl":

   -module(readcsv).
   -export([user/1]).

   user({Pid,DynVar})->
      {ok,Line} = ts_file_server:get_next_line(),
      [Username, Passwd] = string:tokens(Line,";"),
      "username=" ++ Username ++"&password=" ++ Passwd.

The output of the function will be a string
"username=USER&password=PASSWORD"

Then compile it with **erlc readcsv.erl** and put "readcsv.beam" in
"$prefix/lib/erlang/lib/tsung-VERSION/ebin" directory (if the file has
an id set to "random", change the call to
"ts_file_server:get_next_line(random)").

Then use something like this in your session:

   <request subst="true">
     </http>
   </request>

Two functions are available: "ts_file_server:get_next_line" and
"ts_file_server:get_random_line". For the "get_next_line" function,
when the end of file is reached, the first line of the file will be
the next line.

**New in 1.3.0**: you no longer have to create an external function to
parse a simple csv file: you can use "setdynvars" (see next section
for detailed documentation):

   <setdynvars sourcetype="file" fileid="userlist.csv" delimiter=";" order="iter">
    <var name="username" />
    <var name="user_password" />
   </setdynvars>

This defines two dynamic variables **username** and **user_password**
filled with the next entry from the csv file. Using the previous
example, the request is now:

   <request subst="true">
     <http url='/login.cgi' version='1.0'
       contents='username=%%_username%%&amp;password=%%_user_password%%&amp;op=login'
     content_type='application/x-www-form-urlencoded' method='POST'>
     </http>
   </request>

Much simpler than the old method!

In case you have several arrival phases programmed and if you use file
with "order="iter"" the position in the file will not be reset between
different arrival phase. You will not be returned to the first line
when changing phase.

   <arrivalphase phase="1" duration="10" unit="minute">
     <users maxnumber="10" arrivalrate="100" unit="second" />
   </arrivalphase>
   <arrivalphase phase="2" duration="10" unit="minute">
     <users maxnumber="20" arrivalrate="100" unit="second"></users>
   </arrivalphase>

In this example phase 1 will read about 10 lines and phase 2 will read
the next 20 lines.


6.7.3. Dynamic variables
========================

In some cases, you may want to use a value given by the server in a
response later in the session, and this value is **dynamically
generated** by the server for each user. For this, you can use
"<dyn_variable>" in the scenario

Let’s take an example with HTTP. You can easily grab a value in a HTML
form like:

   <form action="go.cgi" method="POST">
     <hidden name="random_num" value="42"></form>
   </form>

with:

   <request>
     <dyn_variable name="random_num"></dyn_variable>
     <http url="/testtsung.html" method="GET" version="1.0"></http>
   </request>

Now "random_num" will be set to 42 during the users session. Its value
will be replace in all mark-up of the form "%%_random_num%%" if and
only if the "request" tag has the attribute "subst="true"", like:

   <request subst="true">
     <http url="/go.cgi" version="1.0"
       contents="username=nic&amp;random_num=%%_random_num%%&amp;op=login"
       content_type="application/x-www-form-urlencoded" method="POST">
     </http>
   </request>


6.7.3.1. Regexp
---------------

If the dynamic value is not a form variable, you can set a regexp by
hand, for example to get the title of a HTML page: the regexp engine
uses the "re" module, a Perl like regular expressions module for
Erlang.

   <request>
     <dyn_variable name="mytitlevar"
                   re="&lt;title&gt;(.*)&lt;/title&gt;"/>
     <http url="/testtsung.html" method="GET" version="1.0"></http>
   </request>

Previously (before 1.4.0), Tsung uses the old "regexp" module from
Erlang. This is now deprecated. The syntax was:

   <request>
     <dyn_variable name="mytitlevar"
                   regexp="&lt;title&gt;\(.*\)&lt;/title&gt;"/>
     <http url="/testtsung.html" method="GET" version="1.0"></http>
   </request>


6.7.3.2. XPath
--------------

A new way to analyze the server response has been introduced in the
release **1.3.0**. It is available only for the HTTP and XMPP plugin
since it is based on XML/HTML parsing. This feature uses the mochiweb
library and **only works with Erlang R12B and newer version**.

This give us some benefices:

* XPath is simple to write and to read, and match very well with
  HTML/XML pages

* The parser works on "binaries()", and doesn’t create any
  "string()".

* The cost of parsing the HTML/XML and build the tree is amortized
  between all the dyn_variables defined for a given request

To utilize XPath expression, use a "xpath" attribute when defining the
"dyn_variable", instead of "re", like:

   <dyn_variable name="field1_value" xpath="//input[@name='field1']/@value"/>
   <dyn_variable name="title" xpath="/html/head/title/text()"/>

There is a bug in the XPath engine, result nodes from “descendant-or-
self” aren’t returned in document order. This isn’t a problem for the
most common cases.

However, queries like "//img[1]/@src" are not recommended, as the
order of the "<img>" elements returned from "//img" is not the
expected.

The order is respected for paths without “descendant-or-self” axis, so
this: "/html/body/div[2]/img[3]/@src" is interpreted as expected and
can be safely used.

It is possible to use XPath to get a list of elements from an html
page, allowing dynamic retrieval of objects. You can either create
embedded Erlang code to parse the list produced, or use foreach that
was introduced in release **1.4.0**.

For XMPP, you can get all the contacts in a dynamic variable:

   <request subst="true">
      <dyn_variable name="contactJids"
        xpath="//iq[@type='result']/query[@xmlns='jabber:iq:roster']//item[string-length(@wr:type)=0]/@jid" />
      <jabber type="iq:roster:get" ack="local"/>
   </request>


6.7.3.3. JSONPath
-----------------

Another way to analyze the server response has been introduced in the
release **1.3.2** when the server is sending JSON data. It is only for
the HTTP plugin. This feature uses the mochiweb library and **only
works with Erlang R13B and newer version**.

Tsung implements a (very) limited subset of JSONPath as defined here
http://goessner.net/articles/JsonPath/

To utilize JSONPath expression, use a **jsonpath** attribute when
defining the "<dyn_variable>>", instead of "re", like:

   <dyn_variable name="array3_value" jsonpath="field.array[3].value"/>

You can also use expressions "Key=Val", e.g.:

   <dyn_variable name="myvar" jsonpath="field.array[?name=bar].value"/>

Starting with version **1.8.0** you can use variables to be
substituted in your JSONPath expression. This requires the surrounding
"<request>" to have set "subst="true"". Otherwise the JSONPath
expression will be taken literally.


6.7.3.4. PostgreSQL
-------------------

New in version 1.3.2.

Since the  PostgreSQL protocol is binary, regexp are not useful to
parse the output of the server. Instead, a specific parsing can be
done to extract content from the server’s response; to do this, use
the "pgsql_expr" attribute. Use "data_row[L][C]" to extract the
column C of the  line L of the data output. You can also use the
literal name of the column (ie. the field name of the table). This
example extract 3 dynamic variables from the server’s response:

First one, extract the 3rd column of the fourth row, then the "mtime"
field from the second row, and then it extract some data of the
"row_description".

   <request>
     <dyn_variable name="myvar" pgsql_expr="data_row[4][3]"/>
     <dyn_variable name="mtime" pgsql_expr="data_row[2].mtime"/>
     <dyn_variable name="row" pgsql_expr="row_description[1][3][1]"/>
     <pgsql type="sql">SELECT * from pgbench_history LIMIT 20;</pgsql>
   </request>

A row description looks like this:

   | =INFO REPORT==== 14-Apr-2010::11:03:22 ===
   |            ts_pgsql:(7:<0.102.0>) PGSQL: Pair={row_description,
   |                                                [{"tid",text,1,23,4,-1,16395},
   |                                                 {"bid",text,2,23,4,-1,16395},
   |                                                 {"aid",text,3,23,4,-1,16395},
   |                                                 {"delta",text,4,23,4,-1,16395},
   |                                                 {"mtime",text,5,1114,8,-1,16395},
   |                                                 {"filler",text,6,1042,-1,26,16395}]}

So in the example, the **row** variable equals “aid”.


6.7.3.5. Decoding variables
---------------------------

It’s possible to decode variable that contains html entities encoded,
this is done with **decode** attribute set to **html_entities**.

   <request>
     <dyn_variable name="mytitlevar"
                   re="&lt;title&gt;(.*)&lt;/title&gt;"
                   decode="html_entities"/>
     <http url="/testtsung.html" method="GET" version="1.0"></http>
   </request>


6.7.3.6. set_dynvars
--------------------

**Since version 1.3.0**, more powerful dynamic variables are
implemented.

You can set dynamic variables not only while parsing server data, but
you can build them using external files or generate them with a
function or generate random numbers/strings:

Several types of dynamic variables are implemented ("sourcetype"
attribute):

* Dynamic variables defined by calling an Erlang function:

     <setdynvars sourcetype="erlang" callback="ts_user_server:get_unique_id">
        <var name="id1" />

* Dynamic variables defined by parsing an external file:

     <setdynvars sourcetype="file" fileid="userdb" delimiter=";" order="iter">
       <var name="user" />
       <var name="user_password" />
     </setdynvars>

  *delimiter* can be any string, and *order* can be **iter** or
  **random**

* A dynamic variable can be a random number (uniform distribution)

     <setdynvars sourcetype="random_number" start="3" end="32">
       <var name="rndint" />
     </setdynvars>

* A dynamic variable can be a random string

     <setdynvars sourcetype="random_string" length="13">
        <var name="rndstring1" />
     </setdynvars>

* A dynamic variable can be a urandom string: this is much faster
  than the random string, but the string is not really random: the
  same set of characters is always used.

* A dynamic variable can be generated by dynamic evaluation of
  erlang code:

     <setdynvars sourcetype="eval"
                 code="fun({Pid,DynVars})->
                           {ok,Val}=ts_dynvars:lookup(md5data,DynVars),
                           ts_digest:md5hex(Val) end.">
       <var name="md5sum" />
     </setdynvars>

  In this case, we use tsung function "ts_dynvars:lookup" to retrieve
  the dynamic variable named "md5data". This dyn_variable "md5data"
  can be set in any of the ways described in the Dynamic variables
  section Dynamic variables.

* A dynamic variable can be generated by applying a JSONPath
  specification (see JSONPath) to an existing dynamic variable:

     <setdynvars sourcetype="jsonpath" from="notification" jsonpath="result[?state=OK].node">
       <var name="deployed" />
     </setdynvars>

* You can create dynamic variables to get the hostname and port of
  the current server

     <setdynvars sourcetype="server">
       <var name="host" />
       <var name="port" />
     </setdynvars>

* You can define a dynamic variable as constant value to use it in a
  plugin (since version **1.5.0**)

     <setdynvars sourcetype="value" value="foobar">
       <var name="constant" />
     </setdynvars>

A **setdynvars** can be defined anywhere in a session.


6.7.4. Checking the server’s response
=====================================

With the tag "match" in a "<request>" tag, you can check the server’s
response against a given string, and do some actions depending on the
result. In any case, if it matches, this will increment the "match"
counter, if it does not match, the "nomatch" counter will be
incremented.

For example, let’s say you want to test a login page. If the login is
ok, the server will respond with "Welcome !" in the HTML body,
otherwise not. To check that:

   <request>
      <match do="continue" when="match">Welcome !</match>
      <http url="/login.php" version="1.0" method="POST"
            contents="username=nic&amp;user_password=sesame"
            content_type="application/x-www-form-urlencoded" >
   </request>

You can use a regexp instead of a simple string.

The list of available actions to do is:

* **continue**: do nothing, continue (only update match or nomatch
  counters)

* **log**: log the request id, userid, sessionid, name in a file (in
  "match.log")

* **abort**: abort the session

* **abort_test**: abort the whole test

* **restart**: restart the session. The maximum number of restarts
  is 3 by default.

* **loop**: repeat the request, after 5 seconds. The maximum number
  of loops is 20 by default.

* **dump**: dump the content of the response in a file. The filename
  is "match-<userid>-<sessionid>-<requestid>-<dumpid>.dump"

You can mixed several match tag in a single request:

   <request>
     <match do="loop" sleep_loop="5" max_loop="10" when="match">Retry</match>
     <match do="abort" when="match">Error</match>
     <http url='/index.php' method=GET'>
   </request>

You can also do the action on **nomatch** instead of **match**.

If you want to skip the HTTP headers, and match only on the body, you
can use **skip_headers=’http’**. Also, you can apply a function to the
content before matching; for example the following example use both
features to compute the md5sum on the body of a HTTP response, and
compares it to a given value:

   <match do='log' when='nomatch' skip_headers='http' apply_to_content='ts_digest:md5hex'>01441debe3d7cc65ba843eee1acff89d</match>
   <http url="/" method="GET" version="1.1"/>

You can also use dynamic variables, using the **subst** attribute:

   <match do='log' when='nomatch' subst='true' >%%_myvar%%</match>
   <http url="/" method="GET"/>

**Since 1.5.0**, it’s now possible to add **name** attribute in
**match** tag to name a record printed in match.log as follow:

   <match do='log' when='match' name='http_match_200ok'>200OK</match>
   <http url="/" method="GET" version="1.1"/>


6.7.5. Loops, If, Foreach
=========================

**Since 1.3.0**, it’s now possible to add conditional/unconditional
loops in a session.

**Since 1.4.0**, it is possible to loop through a list of dynamic
variables thanks to foreach.


6.7.5.1. <for>
--------------

Repeat the enclosing actions a fixed number of times. A dynamic
variable is used as counter, so the current iteration could be used in
requests. List of attributes:

"from"
   Initial value

"to"
   Last value

"incr"
   Amount to increment in each iteration

"var"
   Name of the variable to hold the counter

   <for from="1" to="10" incr="1" var="counter">
     ...
     <request subst="true"> <http url="/page?id=%%_counter%%"></http> </request>
     ...
   </for>


6.7.5.2. <repeat>
-----------------

Repeat the enclosing action (while or until) some condition. This is
intended to be used together with "<dyn_variable>" declarations. List
of attributes:

"name"
   Name of the repeat

"max_repeat"
   Max number of loops (default value is 20)

The last element of repeat must be either "<while>" or "<until>"
example:

   <repeat name="myloop" max_repeat="40">
     ...
     <request>
       <dyn_variable name="result" re="Result: (.*)"/>
       <http url="/random" method="GET" version="1.1"></http>
     </request>
     ...
     <until var="result" eq="5"/>
   </repeat>

**Since 1.3.1**, it’s also possible to add if statements based on
dynamic variables:


6.7.5.3. <if>
-------------

   <if var="tsung_userid" eq="3">
     <request> <http url="/foo"/> </request>
     <request> <http url="/bar"/> </request>
   </if>

You can use "eq" or "neq" to check the variable.

**Since 1.5.1** you can also use the comparison operators "gt", "gte",
"lt" and "lte" to do respectively "greater than", "greater than or
equal to", "less than" and "less than or equal to".

If the dynamic variable is a list (output from XPath for example), you
can access to the n-th element of a list like this:

   <if var="myvar[1]" eq="3">

Here we compare the first element of the list to 3.


6.7.5.4. <abort>
----------------

**Since 1.7.0** you can abort the session or the whole test by using
an "<abort/>" element in a session (can be used inside an <if>
statement for example). By default it will abort the current user
session, but you can abort the whole test by setting the *type*
attribute to *all*  "<abort type='all'/>"


6.7.5.5. <foreach>
------------------

Repeat the enclosing actions for all the elements contained in the
list specified. The basic syntax is as follows:

   <foreach name="element" in="list">
     <request subst="true">
      <http url="%%_element%%" method="GET" version="1.1"/>
     </request>
   </foreach>

It is possible to limit the list of elements you’re looping through,
thanks to the use of the "include" or "exclude" attributes inside the
foreach statement.

As an example, if you want to include only elements with a local path
you can write:

   <foreach name="element" in="list" include="^/.*$">

If you want to exclude all the elements from a specific URI, you would
write:

   <foreach name="element" in="list" exclude="http:\/\/.*\.tld\.com\/.*$">

You can combine this with a XPath query. For instance the following
scenario will retrieve all the images specified on a web page:

   <request subst="true">
     <dyn_variable name="img_list" xpath="//img/@src"/>
     <http url="/mypage.html" method="GET" version="1.1"/>
   </request>
   <foreach name="img" in="img_list">
     <request subst="true">
       <http url="%%_img%%" method="GET" version="1.1"/>
     </request>
   </foreach>


6.7.6. Rate limiting
====================

Since version **1.4.0**, rate limiting can be enabled, either globally
(see Setting options), or for each session separately.

For example, to limit the rate to 64KB/sec for a given session:

   <session name="http-example" probability="70" type="ts_http">
     <set_option name="rate_limit" value="64" />
     ...
   </session>

Only the incoming traffic is rate limited currently.


6.7.7. Requests exclusion
=========================

New in version 1.5.1.

It is possible to exclude some request for a special run. To do this
you have to tag them and use the option "-x" when launching the run.

For example, to exclude the GET of foo.png, add a "tag" to the
respective request:

   <request>
     <http url="/" method="GET"></http>
   </request>
   <request tag="image">
     <http url="/foo.png" method="GET"></http>
   </request>

Then launch the run with:

   tsung -f SCENARIO.xml -x image start

Only the GET to "/" will be performed.

Note that request tags also get logged on **dumptraffic=”protocol”**
(see File structure)


6.7.8. Client certificate
=========================

New in version 1.5.1.

It is possible to use a client certificate for ssl authentication. You
can use dynamic variables to set some parameters of the certificate
(and the key password is optional).

   <session name="https-with-cert" probability="70" type="ts_http">

     <set_option name="certificate">
       <certificate cacertfile="/etc/ssl/ca.pem"
                    keyfile="%%_keyfile%%" keypass="%%_keypass%%" certfile="/home/nobody/.tsung/client.pem"/>
     </set_option>
