Fields
Every log event shipped to Sematext Logs has its structure - it is divided into fields. Each field has a type, for example string, date, or integer. It can even be an object holding structured data. We do everything we can to ensure that log event field types are inferred correctly. However, you may also want to set field types explicitly. This can be done using the Field Editor accessible via a Logs App settings or using the Templates and Mappings APIs.
Common Schema¶
Fields in log events are also referred to as Tags in Sematext. They are used for searching and filtering, but also for pivoting from Logs to other observability data in Sematext, such as performance metrics in Monitoring. The Common Schema for Logs lists special fields and their meaning.
Fields Structure¶
The structure of your logs - their fields and types - is automatically created when you ship your logs to a Logs App in Sematext. There are two places where you can easily see your fields. The first one is the fields and filters section:
The second one is the Field Editor:
Note:
Fields shown in the fields and filters panel and fields shown in the Field Editor may differ. The Field Editor shows only fields that are included in your current Logs App Mapping. On the other hand, fields and filters panel shows all fields that are still present in your Logs App. For example, if you used to ship logs with a foo field and then you deleted it, the Field Editor will not show it, while fields and filters panel will show it as long as there are log events that still contain it.
Field Types¶
Each field has a field type. The following field types are supported:
- Integer/Long - numerical data without floating points
- Float/Double - numerical, floating point data
- Boolean - Boolean field
- Keyword/Not analyzed string - not analyzed text
- Text/Analyzed string - analyzed, full text searchable data
- Date - date based data
- Geo - field type dedicated for spatial data
- Object - nested structure holding structured data
Modifying Fields¶
You can add, remove or edit existing fields by using the Field Editor accessible via a Logs App settings, by using the Templates and Mappings APIs or via "Edit Fields" in fields and filters:
Keep in mind that the modification is applied only to the data that is shipped after the adjustments, so you may need to re-index the data if you want the old data to be adjusted.
Field Editor¶
The Field Editor functionality allows adding, editing, and removing fields present in your logs mappings.
Adding Fields¶
Field Editor lets you add a new field by providing its name and type.
Editing Fields¶
Field Editor allows modifying a field type by changing its type.
The changes done to a field are only applied to the logs shipped after the change was done. Once the changes are applied, the Field Editor will give you an option to re-index your old logs - learn more.
Removing Fields¶
Field Editor lets you remove fields that are no longer present in your logs. You can mark multiple fields for deletion and apply the changes once everything is ready.
Deleting a field removes it from the logs mappings and the logs already shipped to your Logs App will not be affected. If you continue to ship logs with such a field it will appear again. To fully delete the field from your Logs App first make sure the deleted field is no longer present in logs you ship to Sematext and then delete it with Field Editor.
Excluding a field makes the specific field inaccessible for search and visualizing operations. When a field is excluded, it is disabled in the logs mappings and effectively excluded from the index's search capabilities. This means that queries and aggregations will not consider or return results based on the excluded field.
Re-indexing data¶
If you want your changes to apply to old data you can do that by re-indexing it. As you change fields, Field Editor will prompt you with the option to do that. If you start the indexing process all your old logs will be re-indexed in the background and the progress will be displayed on the Field Editor screen.
Note that re-indexing counts towards your Logs App daily volume. Consider double-checking your usage data and temporarily increasing the Daily Volume Limit to avoiding hitting that limit during re-indexing.
