ActionGenerator - Part Two

In the previous part of the two – parts series about ActionGenerator we showed how to develop and run your own ActionGenerator. Today, we want to show you what action generators are there for you to use out-of-the-box and we want to share some insights about the future of this project.

ActionGenerator for ElasticSearch

The ag-player-es project contain all the code specific to action generators for ElasticSearch. You can find two sinks implemented – one for sending simple queries to ElasticSearch and the other for indexing data. They both use ElasticSearch REST API, so no dependencies are needed for those to work. The other piece available in ag-player-es are three players configured and ready to use:

SimpleEsPlayerMain – ActionGenerator for running random queries to ElasticSearch.
DictionaryEsPlayerMain – ActionGenerator for running queries to ElasticSearch. Queries are generated using a provided dictionary.
DictionaryDataEsPlayerMain – ActionGenerator for indexing data to ElasticSearch. Fields content is generated using provided dictionary.

ActionGenerator for Solr

Similar to ag-player-es one can expect that ag-player-solr will contain all the code specific to action generators for Apache Solr. In this project you can find two sinks implementation – one for sending queries to ElasticSearch and one for indexing data. The first one uses Solr HTTP API and the other one uses XML to index data to Solr. Similar to ag-player-es, no dependencies are needed, so you should be able to use action generator for Solr with all recent Solr versions. Apart from that, there are three players configured and ready for use:

DictionaryDataSolrPlayerMain – ActionGenerator for indexing data to Apache Solr. Fields content is generated using provided dictionary.
DictionarySolrPlayerMain – ActionGenerator for running queries to Apache Solr. Queries are generated using provided dictionary.
RandomQueriesSolrPlayerMain – ActionGenerator for running random queries to Apache Solr.

Using ActionGenerator for ElasticSearch

Lets concentrate on players available for ElasticSearch in ActionGenerator. As we wrote above, you have three main players for ElasticSearch that can be used out of the box.

SimpleEsPlayerMain

The simplest of the three generators available. It lets you generate random queries to a given index and with the use of the given field name. In order to use that player, you need to provide the following parameters:

ElasticSearch base URL
ElasticSearch index name
Name of the field queries should be run against
Number of events that should be generated

For example, you could run the following command and have 1000 queries sent to name field of the documents index on your local ElasticSearch instance:

java -cp ag-player-es-0.1.0-withdeps.jar 
com.sematext.ag.es.SimpleEsPlayerMain http://localhost:9200/ documents text 1000

DictionaryEsPlayerMain

The second generator that enables you to run queries uses a dictionary to generate text of your queries. It is similar to the SimpleEsPlayerMain, except for dictionary usage. In order to use that player, you need to provide the same parameters as to the SimpleEsPlayerMain and one additional parameters:

Dictionary path

For example, you could run the following command and have 1000 queries sent to name field of the documents index on your local ElasticSearch instance. Queries would be generated using dict.txt dictionary (each line containing different query string):

java -cp ag-player-es-0.1.0-withdeps.jar com.sematext.ag.es.DictionaryEsPlayerMain http://localhost:9200/ documents text 1000 dict.txt

DictionaryDataEsPlayerMain

The one and only player that enables you to index data to your ElasticSearch instance. Just as the player discussed above, DictionaryDataEsPlayerMain also works with the help of a dictionary. You need to provide the following parameters in order to use this player:

ElasticSearch base URL
ElasticSearch index name
ElasticSearch type name
Number of events that should be generated
Dictionary path
One or more fields and their types

So, for example if you would like to index 100.000 documents to documents index under document type to your local ElasticSearch instance you could run the following

java -cp ag-player-es-0.1.0-withdeps.jar com.sematext.ag.es.DictionaryDataEsPlayerMain http://localhost:9200/ documents document 100000 dict.txt id:numeric title:text likes:numeric

Right now, the following field types are available:

numeric
text

We plan to add more types in the future. Pull requests with patches are welcome!

Using ActionGenerator for Apache Solr

Players available for Apache Solr are similar to the ones available for ElasticSearch, but lets quickly look at them, too.

RandomQueriesSolrPlayerMain

This player is similar to the SimpleEsPlayerMain – it also generates random queries, but to Apache Solr. In order to use it, you need to provide the following parameters:

Apache Solr core search handler URL
The field queries should be run against
Number of queries to be generated

For example, you could run the following command and have 1000 queries sent to name field of the documents core of your local Apache Solr instance:

java -cp ag-player-solr-0.1.0-withdeps.jar com.sematext.ag.solr.RandomQueriesSolrPlayerMain http://localhost:8983/solr/documents name 1000

DictionarySolrPlayerMain

DictonarySolrPlayerMain is similar to RandomQueriesSolrPlayerMain except it uses dictionary to generate queries. In order to use this player you need to provide one additional parameter compared to the ones you provided to DictionarySolrPlayerMain:

Dictionary path

For example, you could run the following command and have 1000 queries sent to name field of the documents core of your local Apache Solr instance. Queries would be generated using dict.txt dictionary (each line containing different query string):

java -cp ag-player-solr-0.1.0-withdeps.jar com.sematext.ag.solr.DictionarySolrPlayerMain http://localhost:8983/solr/documents name 1000 dict.txt

DictionaryDataSolrPlayerMain

The last player I’d like to tell you about is the one that enables data indexation to your Apache Solr instance. DictionaryDataSolrPlayerMain is similar to its ElasticSearch counterpart. You need to provide the following parameters in order to use this player:

Apache Solr core update handler URL
Number of events that should be generated
Dictionary path
One or more fields and their types

For example, if you would like to index 100000 documents to documents core of your local Apache Solr instance you could run the following

java -cp ag-player-solr-0.1.0-withdeps.jar com.sematext.ag.solr.DictionaryDataSolrPlayerMain http://localhost:8983/solr/documents/update/ 100000 dict.txt id:numeric title:text likes:numeric

Lets omit the field types description as it was provided during DictionaryDataEsPlayerMain description above.

Calculating Metrics

Currently, the AbstractHttpSink class has the ability to gather metrics about how the system to which you are sending events is behaving. You can choose between two methods of metrics output – to the standard output or to a file. To enable metrics tracking you need to pass the -DenableMetrics=true parameter when running ActionGenerator. This parameter enables metrics gathering and outputs those to standard output. In order to change that behavior and output the metrics to a file you need to pass the -DmetricsType=file parameter. In addition, you need to specify which directory the metrics should be written to – you do that by passing -DmetricsDir=/path/to/output/dir/ parameter with the value of the directory. Please remember that the directory needs to be created before running your action generator.

Maven Artifacts

The libraries for all the projects creating ActionGenerator can be found in the Sonatype maven repository (https://oss.sonatype.org/content/repositories/releases/) under the following dependencies: