Data Storage

UltiTools encapsulates a data storage API that supports MySQL database, SQLite database (since 6.1.0), and JSON file storage. Data storage is transparent to developers, and UltiTools will determine which storage method to use based on the server owner's configuration.

All you need is an entity class. CRUD operations will be done automatically by UltiTools.

Try not to nest objects

The storage API maps each @Column field to one column (one field in a JSON file). A field of a non-simple type is stored there as JSON text and cannot be queried by its inner fields, so try to use simple values.

SQLite on Paper with spark (all UltiTools versions)

On Paper, spark's background profiler is on by default, and with SQLite storage it shortens how long a server-thread database call waits for a locked database file from about 3 s to about 0.4 s, so the call fails only when another thread or process holds the write lock longer than that (#645). MySQL is not affected. Servers whose writes hold the SQLite lock for long should use MySQL.

Create Entity Class

BaseDataEntity

Create a class that extends BaseDataEntity<String>, and use the @Table and @Column annotations to mark your entity class.

package com.ultikits.docs.data;

import com.ultikits.ultitools.abstracts.data.BaseDataEntity;
import com.ultikits.ultitools.annotations.Column;
import com.ultikits.ultitools.annotations.Table;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import lombok.EqualsAndHashCode;
import lombok.NoArgsConstructor;

@Data
@Builder
@NoArgsConstructor
@AllArgsConstructor
@EqualsAndHashCode(callSuper = true)
@Table("user_data")
public class UserData extends BaseDataEntity<String> {
    @Column("player_name")
    private String playerName;

    @Column(value = "balance", type = "FLOAT")
    private double balance;
}

@Table is used to mark the data set corresponding to the class, and @Column is used to mark the field corresponding to the field of the data set of the class.

@Data, @Builder, @NoArgsConstructor, @AllArgsConstructor, @EqualsAndHashCode are Lombok annotations, which are used to automatically generate getter, setter, builder, equals, hashCode methods.

Migration from AbstractDataEntity

Starting from v6.2.0, DataOperator, Query, and UltiToolsPlugin.getDataOperator() require entities to extend BaseDataEntity<String> instead of AbstractDataEntity. If your entity still extends AbstractDataEntity, change it to BaseDataEntity<String>.

BaseDataEntity<String> provides lifecycle hooks for insert/update/delete/load events:

Method	Description
onCreate()	Called before the entity is first persisted
onUpdate()	Called before the entity is updated
onDelete()	Called before the entity is deleted
onLoad()	Called after the entity is loaded from the data store
validate()	Returns true if the entity is valid
isNew()	Returns true if the entity has no ID
copyWithoutId()	Creates a copy of the entity without the ID. The entity class must implement Cloneable.

As of v6.3.0, the data operators call these hooks themselves, on the JSON, MySQL and SQLite backends alike:

Hook	Called by
onCreate()	insert and insertAll, on each entity, before its fields are written
onUpdate()	update(entity), updateAll, updateCounted and updateIf, on the entity passed in, before its fields are written, whether or not a row is then written
onDelete()	delById and the Query DSL's delete(), on the stored entity, before it is removed; not called when no row has the id
onLoad()	getById, getAll, page, getLike and the Query DSL reads built on them, once per entity returned

update(column, value, id), del(conditions) and exist(...) do not read an entity and call no hook. Before v6.3.0 no operator called any of the four hooks.

AuditableDataEntity

For entities that require audit tracking of creation and modification, use AuditableDataEntity:

package com.ultikits.docs.data;

import com.ultikits.ultitools.abstracts.UltiToolsPlugin;
import com.ultikits.ultitools.abstracts.data.AuditableDataEntity;
import com.ultikits.ultitools.abstracts.data.BaseDataEntity;
import com.ultikits.ultitools.annotations.*;
import com.ultikits.ultitools.interfaces.DataOperator;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import lombok.EqualsAndHashCode;
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.Setter;
import org.bukkit.entity.Player;

import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.UUID;

@Data
@Builder
@NoArgsConstructor
@AllArgsConstructor
@EqualsAndHashCode(callSuper = true)
@Table("audit_log")
public class AuditEntry extends AuditableDataEntity<String> {
    @Column("action")
    private String action;
    @Column("details")
    private String details;
}

AuditableDataEntity<String> extends BaseDataEntity<String> and automatically manages:

Field	Type	Description
createdAt	LocalDateTime	Entity creation timestamp (auto-set in onCreate())
updatedAt	LocalDateTime	Last modification timestamp (updated in onUpdate())
createdBy	UUID	User ID who created the entity (from thread-local context)
updatedBy	UUID	User ID who last modified the entity (from thread-local context)

All four fields are pre-configured with @Column annotations and do not need to be declared in subclasses.

As of v6.3.0 the operators fill the four columns through these hooks: insert sets created_at and updated_at, and created_by and updated_by when a current user is set; an update sets updated_at, and updated_by when a current user is set, and leaves created_at and created_by unchanged. A command handled by BaseCommandExecutor for a player sets that player as the current user while the command body runs, so writes made there record the player. Elsewhere, set the context yourself as shown below. Before v6.3.0 the operators did not call the hooks, and all four columns stayed NULL.

User Context Management

To track which user performed operations, set the current user before database operations:

import com.ultikits.ultitools.abstracts.data.AuditableDataEntity;

UUID currentUserId = player.getUniqueId();
AuditableDataEntity.setCurrentUser(currentUserId);

try {
    DataOperator<AuditEntry> op = plugin.getDataOperator(AuditEntry.class);
    AuditEntry entry = AuditEntry.builder()
        .action("login")
        .details("Player logged in from 192.168.1.1")
        .build();
    op.insert(entry);  // createdBy and updatedBy automatically set
} finally {
    AuditableDataEntity.clearCurrentUser();
}

Always clear the context

Use a try-finally block to ensure clearCurrentUser() is called, otherwise the ThreadLocal context persists across requests and may leak user identity.

Utility Methods

AuditableDataEntity provides convenience methods for time-based queries:

Method	Returns	Description
getAge()	Duration or null	Time elapsed since entity creation
getTimeSinceUpdate()	Duration or null	Time elapsed since last modification
wasModified()	boolean	Whether entity was modified after creation

Example usage:

AuditEntry entry = op.getById("some-id");
if (entry.wasModified()) {
    System.out.println("Modified " + entry.getTimeSinceUpdate().getSeconds() + " seconds ago");
}

Null-safety

getAge() and getTimeSinceUpdate() return null if the entity has not been persisted (missing createdAt or updatedAt). Always check for null before calling methods on the returned Duration.

@Table

@Table annotation has a value attribute, which is used to specify the name of the data set corresponding to the class.

@Column

@Column annotation has two attributes, value attribute is used to specify the column of the data set corresponding to the field, type attribute is used to specify the type of the column of the data set corresponding to the field.

The default value of the type attribute is VARCHAR(255).

Available types can be found in MySQL Data Types.

CRUD Operations

UltiTools encapsulates a semantic CRUD operation API. You only need to call the corresponding method to complete the addition, deletion, modification and query of the data.

DataOperator

DataOperator is used for data operations.

In the main class that inherits UltiToolsPlugin, there is a getDataOperator method to get the data operator.

You need to get the instance of the module main class, and then call the getDataOperator method.

package com.ultikits.docs.data;

import com.ultikits.ultitools.abstracts.UltiToolsPlugin;
import com.ultikits.ultitools.entities.WhereCondition;
import com.ultikits.ultitools.interfaces.DataOperator;

import java.util.List;

public class UserDataService {

    public void save(UltiToolsPlugin plugin, UserData data) {
        DataOperator<UserData> operator = plugin.getDataOperator(UserData.class);
        operator.insert(data);
    }

    public List<UserData> findByName(UltiToolsPlugin plugin, String name) {
        DataOperator<UserData> operator = plugin.getDataOperator(UserData.class);
        return operator.getAll(
                WhereCondition.builder().column("player_name").value(name).build()
        );
    }
}

Obtain DataOperator when it is needed

DataOperator is not thread-safe. Please get DataOperator when you need it, and do not try to save DataOperator object.

Insert

SomeEntity entity = SomeEntity.builder()
    .name("test")
    .something(42.0)
    .build();
dataOperator.insert(entity);

Query

Using WhereCondition:

List<SomeEntity> list = dataOperator.getAll(
    WhereCondition.builder()
        .column("name")
        .value("test")
        .build()
);

Or get a single entity by ID:

SomeEntity entity = dataOperator.getById("some-id");

Get all entities:

List<SomeEntity> all = dataOperator.getAll();

Pagination:

List<SomeEntity> page = dataOperator.page(1, 10); // page 1, 10 per page

page() returns an empty list on the JSON backend

On the JSON backend page(int, int) forwards to getAll(WhereCondition...), whose zero-length branch returns an empty list, while the same call on MySQL or SQLite builds a plain LIMIT ? OFFSET ? and returns the rows, so one module gives two different results depending on the configured backend. Take getAll() and slice the result with subList when you need a page that behaves the same on every backend: page(1, 10, WhereCondition.empty()) is not a substitute, because the relational operators do not filter empty conditions and would emit WHERE null = ?. Aligning page and exist with getAll on empty conditions is tracked in issue #193.

Query DSL

Starting from v6.2.0, you can use the fluent Query DSL for more readable queries:

SomeEntity entity = dataOperator.query()
    .where("name").eq("test")
    .first();

Update

Update a single field:

dataOperator.update("name", "newName", entityId);

Update by entity object:

try {
    entity.setName("newName");
    dataOperator.update(entity);
} catch (IllegalAccessException e) {
    // handle or rethrow
}

This overload declares throws IllegalAccessException, so the calling method must declare or catch it.

As of v6.3.0, every entity a read returns (getById, getAll, page, getLike and the Query DSL) is a copy, and insert stores a copy of the entity you pass. Changing an entity has no effect on the stored data until you pass it to update(...), on every backend. Before v6.3.0 the JSON backend returned the instances it kept in memory, so there a change without update(...) was saved at the next flush, while MySQL and SQLite never saved it.

As of v6.3.0, update(T), update(column, value, id), delById and updateAll throw DataAccessException when the id is null, because no row can be addressed by it; updateAll checks every entity before it writes any. Rows that UltiTools-API 6.2.0 stored on SQLite without an id are given one when the table is initialised: the id the entity reports through getId(), or a new UUID when it reports none, as long as the entity then reports that id. One console line names the table and the count; a row that no id would make addressable is left as it is and counted in a warning. Every write stores getId() in the id column, so an entity that overrides getId() onto another field is addressable by the value it reports.

As of v6.3.0, an update by an id that no row has writes nothing and logs one warning naming the table and the id, on every backend, and returns normally. To learn whether the update wrote, call updateCounted(entity), which returns 1 for a written row and 0 when no row has the id:

if (dataOperator.updateCounted(entity) == 0) {
    // The row was deleted by another writer: nothing was written.
}

A DataOperator implementation outside the framework that does not override updateCounted is counted by whether a row with the id exists before its update.

Conditional Update

updateIf(entity, expected...) writes the entity only while the stored row still matches every expected condition, and returns whether it wrote. Use it to update a value you read earlier without overwriting a change another writer made in between:

Account read = dataOperator.getById(accountId);
double seen = read.getBalance();
read.setBalance(seen + amount);
boolean written = dataOperator.updateIf(read,
    WhereCondition.builder().column("balance").value(seen).build());
if (!written) {
    // Another writer changed the row first: read it again and decide again.
}

On MySQL and SQLite the check and the write are one UPDATE ... WHERE id = ? AND <conditions> statement, so the result holds across servers that share one database. On the JSON backend the check and the write run together under the table lock; a JSON store belongs to one server. The conditions mean what they mean in getAll(WhereCondition...).

updateIf returns false, and writes nothing, when no row with the entity's id matches every condition. It throws DataAccessException when the id is null, when a condition is null, when a condition names a column the entity does not map with @Column, or when a condition's value is null under a comparison other than the default EQUAL, on every backend. As of v6.3.0 a null expected value under the default comparison means IS NULL (JSON: the field is absent or JSON null), so a compare-and-set against a column that was still unset when you read it misses when another writer has filled it in the meantime; this meaning belongs to updateIf alone (#640). A DataOperator implementation outside the framework that does not implement it throws UnsupportedOperationException.

Delete

Delete by ID:

dataOperator.delById(entityId);

Delete by condition:

dataOperator.del(
    WhereCondition.builder()
        .column("name")
        .value("test")
        .build()
);

WhereCondition

WhereCondition is used to specify the query condition.

WhereCondition.builder().column("somecol").value(someval).build();

column is used to specify the column to be queried, and value is used to specify the value to be queried.

Comparisons

As of v6.3.0 a comparison other than EQUAL means the same in getAll, exist, page, del and updateIf on SQLite, MySQL and JSON, and no table is changed. 6.2.5's SQLite and MySQL operators matched every condition by equality.

• INCLUDE, STARTSWITH and ENDSWITH match the value literally as text on every backend: %, _, ! and \ are ordinary characters, not wildcards or escapes. getLike and Query#like keep pattern semantics.
• Letter case and accents follow the backend: SQLite ignores the case of ASCII letters only; MySQL compares under the utf8mb3_general_ci collation of the tables the framework creates, which ignores case and accents (é matches e); JSON compares exactly.
• GREATER and LESS on a byte, short, int, long, float or double field (or its wrapper) compare numbers of the field's type on SQLite and JSON. Pass the value as a number of that type (10, not "10"): on MySQL a string value compares as text, so "10" sorts below "2". On SQLite the typed comparison holds also in a column left on the default VARCHAR(255) type, with 64-bit integer precision, and a stored value compares the way it reads back: a float 0.1f is neither greater nor less than 0.1f. That read-back comparison applies to a value that is a finite number; an infinite value, a number beyond double range and any text are compared as bound, so GREATER Float.POSITIVE_INFINITY matches no row, and NaN matches no row. JSON does not support an infinite or NaN value: a condition holding one throws IllegalArgumentException when it is evaluated.
• On MySQL a numeric field left in a default text column compares in double precision, exact up to ±9,007,199,254,740,992 (2^53) and approximate beyond. Declare a numeric column type such as BIGINT where exact results matter. On JSON the values are compared as double, also approximate beyond 2^53.
• exist(...) works on SQLite; in 6.2.x it threw ClassCastException there.
• Known limitations: on SQLite a double in a default column is stored with 15 significant digits, so 1/3.0 reads back and compares as 0.333333333333333; declare type = "DOUBLE" to keep every digit. Non-numeric text written by hand into a numeric field's column compares on SQLite as its leading number ('abc' as 0, '12abc' as 12), while JSON compares it as text. On SQLite a number that agrees with a stored float to 15 significant digits but differs after them (for example a whole number of 10^15 or more such as (1L << 53) + 1, or a decimal with 16 or more significant digits) compares as equal to it, in a default and in a REAL column; pass a float value for a float field. On SQLite a NaN comparison value is bound as NULL, so a condition holding it matches no row and an updateIf with it never applies; do not retry updateIf on a NaN threshold.

Transactions

For operations that need to succeed or fail together, see the Transactions guide.

MySQL and SQLite run this block in one JDBC transaction

As of v6.3.0 the MySQL and SQLite stores give every operator they hand out a transaction manager (#307), so transaction(...) runs in a JDBC transaction and the two inserts below are committed or rolled back together. As of v6.3.0 the JSON backend stages the callback's writes and commits them together or not at all.

dataOperator.transaction(() -> {
    dataOperator.insert(entity1);
    dataOperator.insert(entity2);
    // Both inserted or none
});

On the JSON backend, a transaction begun by transaction(...) locks each table at its first read or write, as 6.2.5 serialized each table, so a read-modify-write inside it cannot lose an update; a read inside it can wait, at most 5 seconds, for another transaction that holds the table. In a @Transactional method at the default isolation, read-modify-write uses updateIf or isolation = REPEATABLE_READ instead; see JSON Transaction Behavior.

As of v6.3.0 a transaction whose own failure path fails never ends in an implicit commit (#634): when its rollback or its commit fails, the connection is discarded (a HikariCP pool evicts it) instead of being switched back to auto-commit, which would commit it, and the framework never retries the commit. What is stored afterwards depends on where the failure happened:

• When the action fails and the rollback succeeds, nothing is stored.
• When a SQLite or MySQL rollback fails, including the real rollback an outer commit performs for a rollback-only transaction, the framework issues no commit.
• When the commit throws UnexpectedRollbackException because the transaction was marked rollback-only, the transaction was rolled back and none of its writes is stored, on every backend.
• When a SQLite or MySQL commit itself fails (DataAccessException, TRANSACTION_FAILED), the outcome is unknown: the database may have applied the commit before its acknowledgement was lost, for example on a dropped MySQL connection, and discarding the connection cannot undo that. Code that must not apply a write twice, such as a payment or a reward, re-reads the data or uses updateIf before trying again.
• On JSON, a commit that fails commits nothing.

The caller gets the original exception, or UnexpectedRollbackException for a rollback-only transaction. An Error thrown inside transaction(...) rolls the transaction back (the JSON engine discards the staged writes) and is rethrown as the same instance.

Operating JSON Storage

As of v6.3.0, every plugin's JSON tables have a hidden .ultitools-journal folder beside them. Committed transactions are written there and forced to disk before commit returns, and the next start replays them into the table files. In a server it is plugins/UltiTools/data/<plugin name>/.ultitools-journal. See the Transactions guide for what a transaction guarantees.

• Run one server process per data folder. Two servers must never use the same plugins data folder at the same time.
• Back up the .ultitools-journal folder together with the table folders, or back up with the server stopped. A copy such as cp dir/* skips folders whose name starts with a dot.
• Never delete .ultitools-journal, not even after a crash: it holds commits that are not yet in the table files.
• After a crash, start the server once before you edit any JSON data file by hand: the start replays the journal and would overwrite the edit.
• A damaged newest journal record after a power cut usually means the disk or its controller did not honour flush requests. The plugin's JSON data stays closed until you move the named file out of the journal folder and restart the server. A module that obtains its data operators while it loads, as every shipped module does, is refused at that start. Only a plugin that first requests its data after it has loaded recovers at its next request, without a restart.
• One entity file that keeps failing to move (for example, a file another program keeps locked) stops every flush of that plugin from emptying the journal: records accumulate, start-up recovery takes longer, and each failed flush is reported in the server log.
• On Windows, a journal file move that another program (an antivirus scanner, an indexer, a backup tool) briefly holds is retried for at most 150 ms before the commit fails.
• A JSON data folder whose ids contain <, >, :, ", |, ? or * (UltiRemoteBag's edit claims use :) cannot be copied from Linux to Windows: Windows cannot create those files. Data whose ids differ only in letter case cannot be moved to Windows or to a default macOS volume either: the files collide. If a journal record holding such an id reaches a Windows server, start-up keeps that plugin's JSON data closed and names the record: apply it on the Linux server that wrote it, or move it out of the journal folder and lose its changes.
• New ids are checked when they are created, and existing files are never renamed; see JSON Entity Ids. A UUID string always works as an id; lowercase letters, digits, - and _ are safe only if the id is not a Windows device name (con, nul, aux, prn, com1–com9, lpt1–lpt9). Do not key entities by player names (Con and Nul are valid names). An existing id the rules refuse keeps working for reads, update and delete, but deleting it and inserting it again is refused: replace it with update.

Contributors

Ling Bao