Working with Simulation Directories

Analyzing simulation data is as important as running the simulation itself. In this notebook, we will see how we easily access all the data using the SimDir module.

(This notebook is meant to be converted in Sphinx documentation and not used directly.)

from kuibit import simdir as sd

Loading simulation data is as easy as generating SimDir objects. Just provide the path, and kuibit will do the rest.

sim = sd.SimDir("../../tests/tov")

We can get an overview of what is available by printing sim:

print(sim)

Indexed 450 files and 5 subdirectories
Folder /home/runner/work/kuibit/kuibit/tests/tov
/home/runner/work/kuibit/kuibit/tests/tov
Available scalar timeseries:
['physical_time_per_hour', 'current_physical_time_per_hour', 'time_total', 'time_evolution', 'time_computing', 'time_communicating', 'time_io', 'evolution_steps_count', 'local_grid_points_per_second', 'total_grid_points_per_second', 'local_grid_point_updates_count', 'total_grid_point_updates_count', 'local_interior_points_per_second', 'total_interior_points_per_second', 'local_interior_point_updates_count', 'total_interior_point_updates_count', 'io_per_second', 'io_bytes_per_second', 'io_bytes_ascii_per_second', 'io_bytes_binary_per_second', 'io_count', 'io_bytes_count', 'io_bytes_ascii_count', 'io_bytes_binary_count', 'comm_per_second', 'comm_bytes_per_second', 'comm_count', 'comm_bytes_count', 'time_levels']

Available minimum timeseries:
['eps', 'press', 'rho', 'vel[0]', 'vel[1]', 'vel[2]', 'H', 'kxx', 'kxy', 'kxz', 'kyy', 'kyz', 'kzz', 'M1', 'M2', 'M3', 'alp', 'gxx', 'gxy', 'gxz', 'gyy', 'gyz', 'gzz']

Available maximum timeseries:
['press', 'vel[0]', 'alp', 'rho', 'gxx', 'gxy', 'gxz', 'gyy', 'gyz', 'gzz', 'vel[1]', 'vel[2]', 'M1', 'M2', 'M3', 'kxx', 'kxy', 'kxz', 'kyy', 'kyz', 'kzz', 'H', 'eps']

Available norm1 timeseries:
['M1', 'M2', 'M3', 'gxx', 'gxy', 'gxz', 'gyy', 'gyz', 'gzz', 'press', 'rho', 'H', 'vel[0]', 'vel[1]', 'vel[2]', 'eps', 'kxx', 'kxy', 'kxz', 'kyy', 'kyz', 'kzz', 'alp']

Available norm2 timeseries:
['gxx', 'gxy', 'gxz', 'gyy', 'gyz', 'gzz', 'M1', 'M2', 'M3', 'H', 'press', 'eps', 'kxx', 'kxy', 'kxz', 'kyy', 'kyz', 'kzz', 'rho', 'vel[0]', 'vel[1]', 'vel[2]', 'alp']

Available average timeseries:
['kxx', 'kxy', 'kxz', 'kyy', 'kyz', 'kzz', 'rho', 'gxx', 'gxy', 'gxz', 'gyy', 'gyz', 'gzz', 'eps', 'alp', 'M1', 'M2', 'M3', 'press', 'H', 'vel[0]', 'vel[1]', 'vel[2]']
Variables available: dict_keys(['harmonic', 'phi2', 'psi4'])
For variable harmonic:

Avilable radii: [4.0, 8.0]

At radius 4.0, (l, m) available: [(0, 0), (1, -1), (1, 0), (1, 1), (2, -1), (2, -2), (2, 0), (2, 1), (2, 2)]
At radius 8.0, (l, m) available: [(0, 0), (1, -1), (1, 0), (1, 1), (2, -1), (2, -2), (2, 0), (2, 1), (2, 2)]

For variable phi2:

Avilable radii: [44.78, 53.96, 191.0, 60.13, 91.46, 77.93, 48.95, 140.16, 110.69, 67.88]

At radius 44.78, (l, m) available: [(2, -2), (2, -1), (2, 1), (2, 0), (2, 2)] (missing: [(1, 0), (1, 1), (1, -1), (0, 0)])
At radius 48.95, (l, m) available: [(2, -1), (2, 2), (2, 1), (2, -2), (2, 0)] (missing: [(1, 0), (1, 1), (1, -1), (0, 0)])
At radius 53.96, (l, m) available: [(2, -2), (2, 1), (2, 2), (2, -1), (2, 0)] (missing: [(1, 0), (1, 1), (1, -1), (0, 0)])
At radius 60.13, (l, m) available: [(2, -1), (2, -2), (2, 0), (2, 1), (2, 2)] (missing: [(1, 0), (1, 1), (1, -1), (0, 0)])
At radius 67.88, (l, m) available: [(2, -2), (2, 0), (2, -1), (2, 2), (2, 1)] (missing: [(1, 0), (1, 1), (1, -1), (0, 0)])
At radius 77.93, (l, m) available: [(2, -2), (2, 0), (2, -1), (2, 1), (2, 2)] (missing: [(1, 0), (1, 1), (1, -1), (0, 0)])
At radius 91.46, (l, m) available: [(2, 1), (2, 2), (2, -1), (2, 0), (2, -2)] (missing: [(1, 0), (1, 1), (1, -1), (0, 0)])
At radius 110.69, (l, m) available: [(2, -2), (2, 2), (2, -1), (2, 0), (2, 1)] (missing: [(1, 0), (1, 1), (1, -1), (0, 0)])
At radius 140.16, (l, m) available: [(2, -2), (2, -1), (2, 2), (2, 0), (2, 1)] (missing: [(1, 0), (1, 1), (1, -1), (0, 0)])
At radius 191.0, (l, m) available: [(2, 0), (2, 2), (2, 1), (2, -2), (2, -1)] (missing: [(1, 0), (1, 1), (1, -1), (0, 0)])

For variable psi4:

Avilable radii: [110.69, 77.93, 191.0, 140.16, 53.96, 91.46, 67.88, 48.95, 44.78, 60.13]

At radius 44.78, (l, m) available: [(2, -2), (2, -1), (2, 2), (2, 0), (2, 1)] (missing: [(1, 0), (1, 1), (1, -1), (0, 0)])
At radius 48.95, (l, m) available: [(2, -2), (2, 2), (2, 1), (2, 0), (2, -1)] (missing: [(1, 0), (1, 1), (1, -1), (0, 0)])
At radius 53.96, (l, m) available: [(2, -1), (2, -2), (2, 2), (2, 0), (2, 1)] (missing: [(1, 0), (1, 1), (1, -1), (0, 0)])
At radius 60.13, (l, m) available: [(2, 1), (2, -2), (2, -1), (2, 2), (2, 0)] (missing: [(1, 0), (1, 1), (1, -1), (0, 0)])
At radius 67.88, (l, m) available: [(2, 0), (2, 1), (2, -2), (2, 2), (2, -1)] (missing: [(1, 0), (1, 1), (1, -1), (0, 0)])
At radius 77.93, (l, m) available: [(2, -1), (2, 2), (2, 1), (2, -2), (2, 0)] (missing: [(1, 0), (1, 1), (1, -1), (0, 0)])
At radius 91.46, (l, m) available: [(2, -2), (2, 2), (2, 0), (2, -1), (2, 1)] (missing: [(1, 0), (1, 1), (1, -1), (0, 0)])
At radius 110.69, (l, m) available: [(2, -1), (2, 2), (2, -2), (2, 0), (2, 1)] (missing: [(1, 0), (1, 1), (1, -1), (0, 0)])
At radius 140.16, (l, m) available: [(2, -1), (2, 2), (2, 1), (2, 0), (2, -2)] (missing: [(1, 0), (1, 1), (1, -1), (0, 0)])
At radius 191.0, (l, m) available: [(2, 1), (2, 2), (2, -1), (2, 0), (2, -2)] (missing: [(1, 0), (1, 1), (1, -1), (0, 0)])

Available gravitational wave dataAvailable electromagnetic wave data
Available grid data of dimension 1D (x):
['kxx', 'kxy', 'kxz', 'kyy', 'kyz', 'kzz', 'gxx', 'gxy', 'gxz', 'gyy', 'gyz', 'gzz', 'H', 'press', 'alp', 'M1', 'M2', 'M3', 'vel[0]', 'vel[1]', 'vel[2]', 'eps', 'rho']


Available grid data of dimension 1D (y):
['vel[0]', 'vel[1]', 'vel[2]', 'eps', 'alp', 'H', 'press', 'M1', 'M2', 'M3', 'kxx', 'kxy', 'kxz', 'kyy', 'kyz', 'kzz', 'gxx', 'gxy', 'gxz', 'gyy', 'gyz', 'gzz', 'rho']


Available grid data of dimension 1D (z):
['alp', 'press', 'H', 'eps', 'kxx', 'kxy', 'kxz', 'kyy', 'kyz', 'kzz', 'rho', 'M1', 'M2', 'M3', 'vel[0]', 'vel[1]', 'vel[2]', 'gxx', 'gxy', 'gxz', 'gyy', 'gyz', 'gzz']


Available grid data of dimension 2D (xy):
['vel[0]', 'rho', 'vel[1]', 'vel[2]']


Available grid data of dimension 2D (xz):
['vel[0]', 'vel[1]', 'rho', 'vel[2]']


Available grid data of dimension 2D (yz):
['vel[0]', 'vel[2]', 'vel[1]', 'rho']


Available grid data of dimension 3D (xyz):
['vel[2]', 'rho', 'vel[0]', 'vel[1]', 'mp_harmonic']
No horizon found
Timers available for processes [0, 1]

This is a long output because we have a lot of available data! Everything that is printed is the output is easily accessible through SimDir. Moreover, the loading of data is lazy and the result is cached: computations are done only when needed, and all the subsequent calls will be immediate.

If you have thorns with custom output, you have to extend kuibit to analyze them. SimDir has multiple submodules to handle the different kind of data. Let us explore them.

SimDir and pickles

SimDir performs some operations only when they needed. This work has be done every time the Python interpreter is killed. In some cases, it is useful to save the progress to files, resulting in faster execution times. This can be done with pickles. The simplest way to do so is as follows.

with sd.SimDir("../../tests/tov", pickle_file="/tmp/sim.pickle") as sim2:
    print(sim2)

Indexed 450 files and 5 subdirectories
Folder /home/runner/work/kuibit/kuibit/tests/tov
/home/runner/work/kuibit/kuibit/tests/tov
Available scalar timeseries:
['physical_time_per_hour', 'current_physical_time_per_hour', 'time_total', 'time_evolution', 'time_computing', 'time_communicating', 'time_io', 'evolution_steps_count', 'local_grid_points_per_second', 'total_grid_points_per_second', 'local_grid_point_updates_count', 'total_grid_point_updates_count', 'local_interior_points_per_second', 'total_interior_points_per_second', 'local_interior_point_updates_count', 'total_interior_point_updates_count', 'io_per_second', 'io_bytes_per_second', 'io_bytes_ascii_per_second', 'io_bytes_binary_per_second', 'io_count', 'io_bytes_count', 'io_bytes_ascii_count', 'io_bytes_binary_count', 'comm_per_second', 'comm_bytes_per_second', 'comm_count', 'comm_bytes_count', 'time_levels']

Available minimum timeseries:
['eps', 'press', 'rho', 'vel[0]', 'vel[1]', 'vel[2]', 'H', 'kxx', 'kxy', 'kxz', 'kyy', 'kyz', 'kzz', 'M1', 'M2', 'M3', 'alp', 'gxx', 'gxy', 'gxz', 'gyy', 'gyz', 'gzz']

Available maximum timeseries:
['press', 'vel[0]', 'alp', 'rho', 'gxx', 'gxy', 'gxz', 'gyy', 'gyz', 'gzz', 'vel[1]', 'vel[2]', 'M1', 'M2', 'M3', 'kxx', 'kxy', 'kxz', 'kyy', 'kyz', 'kzz', 'H', 'eps']

Available norm1 timeseries:
['M1', 'M2', 'M3', 'gxx', 'gxy', 'gxz', 'gyy', 'gyz', 'gzz', 'press', 'rho', 'H', 'vel[0]', 'vel[1]', 'vel[2]', 'eps', 'kxx', 'kxy', 'kxz', 'kyy', 'kyz', 'kzz', 'alp']

Available norm2 timeseries:
['gxx', 'gxy', 'gxz', 'gyy', 'gyz', 'gzz', 'M1', 'M2', 'M3', 'H', 'press', 'eps', 'kxx', 'kxy', 'kxz', 'kyy', 'kyz', 'kzz', 'rho', 'vel[0]', 'vel[1]', 'vel[2]', 'alp']

Available average timeseries:
['kxx', 'kxy', 'kxz', 'kyy', 'kyz', 'kzz', 'rho', 'gxx', 'gxy', 'gxz', 'gyy', 'gyz', 'gzz', 'eps', 'alp', 'M1', 'M2', 'M3', 'press', 'H', 'vel[0]', 'vel[1]', 'vel[2]']
Variables available: dict_keys(['harmonic', 'phi2', 'psi4'])
For variable harmonic:

Avilable radii: [4.0, 8.0]

At radius 4.0, (l, m) available: [(0, 0), (1, -1), (1, 0), (1, 1), (2, -1), (2, -2), (2, 0), (2, 1), (2, 2)]
At radius 8.0, (l, m) available: [(0, 0), (1, -1), (1, 0), (1, 1), (2, -1), (2, -2), (2, 0), (2, 1), (2, 2)]

For variable phi2:

Avilable radii: [44.78, 53.96, 191.0, 60.13, 91.46, 77.93, 48.95, 140.16, 110.69, 67.88]

At radius 44.78, (l, m) available: [(2, -2), (2, -1), (2, 1), (2, 0), (2, 2)] (missing: [(1, 0), (1, 1), (1, -1), (0, 0)])
At radius 48.95, (l, m) available: [(2, -1), (2, 2), (2, 1), (2, -2), (2, 0)] (missing: [(1, 0), (1, 1), (1, -1), (0, 0)])
At radius 53.96, (l, m) available: [(2, -2), (2, 1), (2, 2), (2, -1), (2, 0)] (missing: [(1, 0), (1, 1), (1, -1), (0, 0)])
At radius 60.13, (l, m) available: [(2, -1), (2, -2), (2, 0), (2, 1), (2, 2)] (missing: [(1, 0), (1, 1), (1, -1), (0, 0)])
At radius 67.88, (l, m) available: [(2, -2), (2, 0), (2, -1), (2, 2), (2, 1)] (missing: [(1, 0), (1, 1), (1, -1), (0, 0)])
At radius 77.93, (l, m) available: [(2, -2), (2, 0), (2, -1), (2, 1), (2, 2)] (missing: [(1, 0), (1, 1), (1, -1), (0, 0)])
At radius 91.46, (l, m) available: [(2, 1), (2, 2), (2, -1), (2, 0), (2, -2)] (missing: [(1, 0), (1, 1), (1, -1), (0, 0)])
At radius 110.69, (l, m) available: [(2, -2), (2, 2), (2, -1), (2, 0), (2, 1)] (missing: [(1, 0), (1, 1), (1, -1), (0, 0)])
At radius 140.16, (l, m) available: [(2, -2), (2, -1), (2, 2), (2, 0), (2, 1)] (missing: [(1, 0), (1, 1), (1, -1), (0, 0)])
At radius 191.0, (l, m) available: [(2, 0), (2, 2), (2, 1), (2, -2), (2, -1)] (missing: [(1, 0), (1, 1), (1, -1), (0, 0)])

For variable psi4:

Avilable radii: [110.69, 77.93, 191.0, 140.16, 53.96, 91.46, 67.88, 48.95, 44.78, 60.13]

At radius 44.78, (l, m) available: [(2, -2), (2, -1), (2, 2), (2, 0), (2, 1)] (missing: [(1, 0), (1, 1), (1, -1), (0, 0)])
At radius 48.95, (l, m) available: [(2, -2), (2, 2), (2, 1), (2, 0), (2, -1)] (missing: [(1, 0), (1, 1), (1, -1), (0, 0)])
At radius 53.96, (l, m) available: [(2, -1), (2, -2), (2, 2), (2, 0), (2, 1)] (missing: [(1, 0), (1, 1), (1, -1), (0, 0)])
At radius 60.13, (l, m) available: [(2, 1), (2, -2), (2, -1), (2, 2), (2, 0)] (missing: [(1, 0), (1, 1), (1, -1), (0, 0)])
At radius 67.88, (l, m) available: [(2, 0), (2, 1), (2, -2), (2, 2), (2, -1)] (missing: [(1, 0), (1, 1), (1, -1), (0, 0)])
At radius 77.93, (l, m) available: [(2, -1), (2, 2), (2, 1), (2, -2), (2, 0)] (missing: [(1, 0), (1, 1), (1, -1), (0, 0)])
At radius 91.46, (l, m) available: [(2, -2), (2, 2), (2, 0), (2, -1), (2, 1)] (missing: [(1, 0), (1, 1), (1, -1), (0, 0)])
At radius 110.69, (l, m) available: [(2, -1), (2, 2), (2, -2), (2, 0), (2, 1)] (missing: [(1, 0), (1, 1), (1, -1), (0, 0)])
At radius 140.16, (l, m) available: [(2, -1), (2, 2), (2, 1), (2, 0), (2, -2)] (missing: [(1, 0), (1, 1), (1, -1), (0, 0)])
At radius 191.0, (l, m) available: [(2, 1), (2, 2), (2, -1), (2, 0), (2, -2)] (missing: [(1, 0), (1, 1), (1, -1), (0, 0)])

Available gravitational wave dataAvailable electromagnetic wave data
Available grid data of dimension 1D (x):
['kxx', 'kxy', 'kxz', 'kyy', 'kyz', 'kzz', 'gxx', 'gxy', 'gxz', 'gyy', 'gyz', 'gzz', 'H', 'press', 'alp', 'M1', 'M2', 'M3', 'vel[0]', 'vel[1]', 'vel[2]', 'eps', 'rho']


Available grid data of dimension 1D (y):
['vel[0]', 'vel[1]', 'vel[2]', 'eps', 'alp', 'H', 'press', 'M1', 'M2', 'M3', 'kxx', 'kxy', 'kxz', 'kyy', 'kyz', 'kzz', 'gxx', 'gxy', 'gxz', 'gyy', 'gyz', 'gzz', 'rho']


Available grid data of dimension 1D (z):
['alp', 'press', 'H', 'eps', 'kxx', 'kxy', 'kxz', 'kyy', 'kyz', 'kzz', 'rho', 'M1', 'M2', 'M3', 'vel[0]', 'vel[1]', 'vel[2]', 'gxx', 'gxy', 'gxz', 'gyy', 'gyz', 'gzz']


Available grid data of dimension 2D (xy):
['vel[0]', 'rho', 'vel[1]', 'vel[2]']


Available grid data of dimension 2D (xz):
['vel[0]', 'vel[1]', 'rho', 'vel[2]']


Available grid data of dimension 2D (yz):
['vel[0]', 'vel[2]', 'vel[1]', 'rho']


Available grid data of dimension 3D (xyz):
['vel[2]', 'rho', 'vel[0]', 'vel[1]', 'mp_harmonic']
No horizon found
Timers available for processes [0, 1]

When this code is run, kuibit checks if a pickle file exists in /tmp/sim.pickle. If yes, it will load it, effectively “resuming a previous session”. At the end of the with block, the file is updated with the new work done. If the file did not exist initially, it will be created.

It is important to be careful when working with pickles as the risk of stale objects is high. If the data is changing (e.g., new restarts are added), it is likely that it will result in errors.

It is always possible to save the state of a SimDir using the save method and reads back the data with the load_SimDir function

sim.save("/tmp/saved.pickle") # The filename can be anything
sim3 = sd.load_SimDir("/tmp/saved.pickle")

cactus_scalars

The cactus_scalars module understands ASCII files as output by CarpetASCII. It supports all the reductions (maximum, minimum, …), 0d files, as well as scalar data. Using regular expression on the header of the file, cactus_scalars can process one variable per file and multiple variables per file. In SimDir, you can access scalars with the ts or timeseries attributes.

timeseries = sim.ts
# equivalent: timeseries = sim.timeseries

If we printed timeseries, we would find part of the output from SimDir. This is because the printed output from SimDir is obtained printing all the different submodules. Another way to see the available variable for a secific reduction is with the keys method:

timeseries.maximum.keys()

dict_keys(['press', 'vel[0]', 'alp', 'rho', 'gxx', 'gxy', 'gxz', 'gyy', 'gyz', 'gzz', 'vel[1]', 'vel[2]', 'M1', 'M2', 'M3', 'kxx', 'kxy', 'kxz', 'kyy', 'kyz', 'kzz', 'H', 'eps'])

There are equivalent three different way to access those variables as timeseries. The different methods are nearly equivalent.

rho_max1 = timeseries.maximum.fields.rho
rho_max2 = timeseries.maximum['rho']
rho_max3 = timeseries.maximum.get('rho')

These objects are timeseries, which are extremely convienent representation of data evolving in time. These series span multiple simulation restarts and have files in different folders: kuibit does all the hard work to find the files and combine them. Moreover, they timeseries have a lot of features, as you can see in the timeseries tutorial.

cactus_multipoles and cactus_waves

Gravitational waves are one of the most important quantities in numerical relativity simulations. Typically, these are computed using the Newman-Penrose formalism and extracted in terms of their multipolar components. kuibit fully supports the Multipole thorn with the cactus_multipoles module. The sim.multipoles attribute contains multipolar data. This is a dictionary-like object with keys the various variable for which multipolar data is available.

print(sim.multipoles.keys())

dict_keys(['harmonic', 'phi2', 'psi4'])

Once a variable is selected, the returned object is once again a dictionary with keys the various extracion radii.

print(sim.multipoles["phi2"].keys())
phi2_0 = sim.multipoles["phi2"][91.46]

print(phi2_0.available_lm)

dict_keys([44.78, 53.96, 191.0, 60.13, 91.46, 77.93, 48.95, 140.16, 110.69, 67.88])
{(2, -1), (2, -2), (2, 1), (2, 0), (2, 2)}

Selected the extraction radius, the variable contains all the various multipole l and m available. (HDF5 data is preferred in this.) Each component is then a timeseries.

phi2_0_22 = phi2_0[2,2]
print(type(phi2_0_22))

<class 'kuibit.timeseries.TimeSeries'>

Gravitational waves have a dedicated interface that is based on cactus_multipole and adds new specific features. Gravitational waves can be accessed with the .gravitationalwaves or .gws attributes.

psi4 = sim.gws

radius0 = psi4.radii[0]

# Example of specific function
print("Max GW power for l = 2, m = 2:", psi4[radius0].get_power_lm(2, 2, 100).max())

Max GW power for l = 2, m = 2: 1713.0809998152504

cactus_grid_functions

The .gridfunctions (or .gf) attribute is used to access grid data, using the cactus_grid_functions module. The structure is similar to time series.

print(sim.gf)

Available grid data of dimension 1D (x):
['kxx', 'kxy', 'kxz', 'kyy', 'kyz', 'kzz', 'gxx', 'gxy', 'gxz', 'gyy', 'gyz', 'gzz', 'H', 'press', 'alp', 'M1', 'M2', 'M3', 'vel[0]', 'vel[1]', 'vel[2]', 'eps', 'rho']


Available grid data of dimension 1D (y):
['vel[0]', 'vel[1]', 'vel[2]', 'eps', 'alp', 'H', 'press', 'M1', 'M2', 'M3', 'kxx', 'kxy', 'kxz', 'kyy', 'kyz', 'kzz', 'gxx', 'gxy', 'gxz', 'gyy', 'gyz', 'gzz', 'rho']


Available grid data of dimension 1D (z):
['alp', 'press', 'H', 'eps', 'kxx', 'kxy', 'kxz', 'kyy', 'kyz', 'kzz', 'rho', 'M1', 'M2', 'M3', 'vel[0]', 'vel[1]', 'vel[2]', 'gxx', 'gxy', 'gxz', 'gyy', 'gyz', 'gzz']


Available grid data of dimension 2D (xy):
['vel[0]', 'rho', 'vel[1]', 'vel[2]']


Available grid data of dimension 2D (xz):
['vel[0]', 'vel[1]', 'rho', 'vel[2]']


Available grid data of dimension 2D (yz):
['vel[0]', 'vel[2]', 'vel[1]', 'rho']


Available grid data of dimension 3D (xyz):
['vel[2]', 'rho', 'vel[0]', 'vel[1]', 'mp_harmonic']

# rest-mass density on the xy plane at iteration 0
rho = sim.gf.xy["rho"][0]

For more information of grid data, check out the grid_data tutorial.

cactus_horizons

kuibit can read horizon data from QuasiLocalMeasures and AHFinderDirect. The module that is responsible for this is cactus_horizons. Horizon information is in the .horizon attribute.

cactus_timers

If timing information is present, it can be accessed with the .timers attribute. This is provided by the cactus_timers modules.