dpemu.filters package

Submodules

dpemu.filters.common module

class dpemu.filters.common.ApplyToTuple(ftr, tuple_index)

Bases: dpemu.filters.filter.Filter

Applies the given filter to only some index in the data tuple.

Given a filter and an index as parameters, applies the given filter to the given index in the data tuple.

Inherits Filter class.

__init__(ftr, tuple_index)

Parameters

• ftr (dpemu.filters.Filter) – Filter to apply.

• tuple_index (int) – Index of the tuple to apply the filter to.

apply(node_data, random_state, named_dims)

Applies the filter to the data.

Parameters

• node_data (numpy.ndarray) – Data to be modified as a NumPy array.

• random_state (mtrand.RandomState) – An instance of numpy.random.RandomState() random number generator.

• named_dims (dict) – Named dimensions.

class dpemu.filters.common.ApplyWithProbability(ftr, probability_id)

Bases: dpemu.filters.filter.Filter

Applies the input filter to the data with the specified probability.

Inherits Filter class.

__init__(ftr, probability_id)

Parameters

• ftr_id (str) – The key mapping to the filter.

• probability_id (str) – The key mapping to the probability of the filter being applied.

apply(node_data, random_state, named_dims)

Applies the filter to the data.

Parameters

• node_data (numpy.ndarray) – Data to be modified as a NumPy array.

• random_state (mtrand.RandomState) – An instance of numpy.random.RandomState() random number generator.

• named_dims (dict) – Named dimensions.

class dpemu.filters.common.Clip(min_id, max_id)

Bases: dpemu.filters.filter.Filter

Clips values between minimum and maximum values provided by the user.

Sets values less than the minimum value to min, and values greater than the maximum value to max.

Inherits Filter class.

__init__(min_id, max_id)

Parameters

• min_id (str) – The key mapping to the minimum value.

• max_id (str) – The key mapping to the maximum value.

apply(node_data, random_state, named_dims)

Applies the filter to the data.

Parameters

• node_data (numpy.ndarray) – Data to be modified as a NumPy array.

• random_state (mtrand.RandomState) – An instance of numpy.random.RandomState() random number generator.

• named_dims (dict) – Named dimensions.

class dpemu.filters.common.GaussianNoise(mean_id, std_id)

Bases: dpemu.filters.filter.Filter

Adds normally distributed noise to data.

Adds random noise drawn from a Gaussian distribution with the provided mean and standard deviation to each element in the array.

Inherits Filter class.

__init__(mean_id, std_id)

Parameters

• mean_id (str) – The key mapping to the mean of the random noise.

• std_id (str) – The key mapping to the standard deviation of the random noise.

apply(node_data, random_state, named_dims)

Applies the filter to the data.

Parameters

• node_data (numpy.ndarray) – Data to be modified as a NumPy array.

• random_state (mtrand.RandomState) – An instance of numpy.random.RandomState() random number generator.

• named_dims (dict) – Named dimensions.

class dpemu.filters.common.GaussianNoiseTimeDependent(mean_id, std_id, mean_increase_id, std_increase_id)

Bases: dpemu.filters.filter.Filter

Adds normally distributed noise increasing in intensity with time to the data.

Adds random noise drawn from a Gaussian distribution with mean and standard deviation calculated from the initial mean and standard deviation, the elapsed time, and the increase to mean and standard deviation per unit of time.

Inherits Filter class.

__init__(mean_id, std_id, mean_increase_id, std_increase_id)

Parameters

• mean_id (str) – The key mapping to the initial mean of the random noise.

• std_id (str) – The key mapping to the initial standard deviation of the random noise.

• mean_increase_id (str) – The key mapping to the increase of the mean of the random noise per unit of time.

• std_increase_id (str) – The key mapping to the increase of the standard

deviation of the random noise per unit of time.

apply(node_data, random_state, named_dims)

Applies the filter to the data.

Parameters

• node_data (numpy.ndarray) – Data to be modified as a NumPy array.

• random_state (mtrand.RandomState) – An instance of numpy.random.RandomState() random number generator.

• named_dims (dict) – Named dimensions.

class dpemu.filters.common.Missing(probability_id, missing_value_id)

Bases: dpemu.filters.filter.Filter

Introduces missing values to data.

For each element in the array, makes that element go missing with the provided probability. Values that go missing are replaced with the provided value, which should usually be NaN.

Inherits Filter class.

__init__(probability_id, missing_value_id)

Parameters

• probability_id (str) – The key mapping to the probability any specific value goes missing.

• missing_value_id (str) – The key mapping to the value that represents missing values.

apply(node_data, random_state, named_dims)

Applies the filter to the data.

Parameters

• node_data (numpy.ndarray) – Data to be modified as a NumPy array.

• random_state (mtrand.RandomState) – An instance of numpy.random.RandomState() random number generator.

• named_dims (dict) – Named dimensions.

class dpemu.filters.common.ModifyAsDataType(dtype_id, ftr)

Bases: dpemu.filters.filter.Filter

Applies the input filter to the data casted to the specified type.

First casts the data into the specified type, then applies the filter, then returns the data to its original type.

Inherits Filter class.

__init__(dtype_id, ftr)

Parameters

• dtype_id (str) – The key mapping to the data type to cast the input to.

• ftr (dpemu.filters.Filter) – The filter to apply to the casted data.

apply(node_data, random_state, named_dims)

Applies the filter to the data.

Parameters

• node_data (numpy.ndarray) – Data to be modified as a NumPy array.

• random_state (mtrand.RandomState) – An instance of numpy.random.RandomState() random number generator.

• named_dims (dict) – Named dimensions.

class dpemu.filters.common.StrangeBehaviour(do_strange_behaviour_id)

Bases: dpemu.filters.filter.Filter

Emulates strange sensor values due to anomalous conditions around the sensor.

The function do_strange_behaviour given as a parameter is used to output strange sensor values into the data.

Inherits Filter class.

__init__(do_strange_behaviour_id)

Parameters

do_strange_behaviour_id (str) – The key mapping to the strange behaviour -function.

apply(node_data, random_state, named_dims)

Applies the filter to the data.

Parameters

• node_data (numpy.ndarray) – Data to be modified as a NumPy array.

• random_state (mtrand.RandomState) – An instance of numpy.random.RandomState() random number generator.

• named_dims (dict) – Named dimensions.

dpemu.filters.filter module

class dpemu.filters.filter.Addition(filter_a, filter_b)

Bases: dpemu.filters.filter.BinaryFilter

Combines results of the two child filters by adding them together.

Inherits BinaryFilter class.

operation(element_a, element_b)

The pairwise binary operation used to combine results from the two child filters.

Parameters

• element_a (object) – The element from the data filter_a operated on.

• element_b (object) – The element from the data filter_b operated on.

class dpemu.filters.filter.And(filter_a, filter_b)

Bases: dpemu.filters.filter.BinaryFilter

Combines results of the two child filters with bitwise AND.

Inherits BinaryFilter class.

operation(element_a, element_b)

The pairwise binary operation used to combine results from the two child filters.

Parameters

• element_a (object) – The element from the data filter_a operated on.

• element_b (object) – The element from the data filter_b operated on.

class dpemu.filters.filter.BinaryFilter(filter_a, filter_b)

Bases: dpemu.filters.filter.Filter

Abstract Filter applying two given filters to the data, combining the results with a pairwise binary operation.

The pairwise binary operation is specified by the inheriting class by overriding the operation-function.

Inherits Filter class.

__init__(filter_a, filter_b)

Parameters

• filter_a (str) – The first filter.

• filter_b (str) – The second filter.

apply(node_data, random_state, named_dims)

Applies the filter to the data.

Parameters

• node_data (numpy.ndarray) – Data to be modified as a NumPy array.

• random_state (mtrand.RandomState) – An instance of numpy.random.RandomState() random number generator.

• named_dims (dict) – Named dimensions.

abstract operation(element_a, element_b)

The pairwise binary operation used to combine results from the two child filters.

Parameters

• element_a (object) – The element from the data filter_a operated on.

• element_b (object) – The element from the data filter_b operated on.

class dpemu.filters.filter.Constant(value_id)

Bases: dpemu.filters.filter.Filter

Overwrites all values in the data with the given value.

Inherits Filter class.

__init__(value_id)

Parameters

value_id – The key mapping to the value to overwrite values in the data with.

apply(node_data, random_state, named_dims)

Applies the filter to the data.

Parameters

• node_data (numpy.ndarray) – Data to be modified as a NumPy array.

• random_state (mtrand.RandomState) – An instance of numpy.random.RandomState() random number generator.

• named_dims (dict) – Named dimensions.

class dpemu.filters.filter.Difference(ftr)

Bases: dpemu.filters.filter.Filter

Returns change to data from filter

Given a filter, applies the filter to the data, then subtracting the original. Functions identically to Subtraction(filter, Identity()).

Inherits BinaryFilter class.

apply(node_data, random_state, named_dims)

Applies the filter to the data.

Parameters

• node_data (numpy.ndarray) – Data to be modified as a NumPy array.

• random_state (mtrand.RandomState) – An instance of numpy.random.RandomState() random number generator.

• named_dims (dict) – Named dimensions.

class dpemu.filters.filter.Division(filter_a, filter_b)

Bases: dpemu.filters.filter.BinaryFilter

Combines results of the two child filters by dividing the results of the first by the seconds.

Inherits BinaryFilter class.

operation(element_a, element_b)

The pairwise binary operation used to combine results from the two child filters.

Parameters

• element_a (object) – The element from the data filter_a operated on.

• element_b (object) – The element from the data filter_b operated on.

class dpemu.filters.filter.Filter

Bases: abc.ABC

A Filter is an error source which can be attached to an Array node.

The apply method applies the filter to the data. A filter may always assume that it is acting upon a NumPy array. (if the underlying data object is not a NumPy array, the required conversions are performed by the Array node to which the Filter is attached.)

Parameters

ABC (object) – Helper class that provides a standard way to create

an abstract class using inheritance.

__init__()

Set the seeds for the RNG’s of NumPy and Python.

abstract apply(node_data, random_state, named_dims)

Applies the filter to the data.

Parameters

• node_data (numpy.ndarray) – Data to be modified as a NumPy array.

• random_state (mtrand.RandomState) – An instance of numpy.random.RandomState() random number generator.

• named_dims (dict) – Named dimensions.

set_params(params_dict)

Set parameters for error generation.

Parameters

params_dict (dict) – A dictionary containing key-value pairs of error parameters.

class dpemu.filters.filter.Identity

Bases: dpemu.filters.filter.Filter

Acts as the identity operator, thus doesn’t modify the data.

Inherits Filter class.

apply(node_data, random_state, named_dims)

Applies the filter to the data.

Parameters

• node_data (numpy.ndarray) – Data to be modified as a NumPy array.

• random_state (mtrand.RandomState) – An instance of numpy.random.RandomState() random number generator.

• named_dims (dict) – Named dimensions.

class dpemu.filters.filter.IntegerDivision(filter_a, filter_b)

Bases: dpemu.filters.filter.BinaryFilter

Combines results of the two child filters by perfoming integer division on the results of the first by the results of the second.

The division is done with python’s // operator.

Inherits BinaryFilter class.

operation(element_a, element_b)

The pairwise binary operation used to combine results from the two child filters.

Parameters

• element_a (object) – The element from the data filter_a operated on.

• element_b (object) – The element from the data filter_b operated on.

class dpemu.filters.filter.Max(filter_a, filter_b)

Bases: dpemu.filters.filter.BinaryFilter

Combines results of the two child filters by taking the pairwise maximum of the results of the first and second.

Inherits BinaryFilter class.

operation(element_a, element_b)

The pairwise binary operation used to combine results from the two child filters.

Parameters

• element_a (object) – The element from the data filter_a operated on.

• element_b (object) – The element from the data filter_b operated on.

class dpemu.filters.filter.Min(filter_a, filter_b)

Bases: dpemu.filters.filter.BinaryFilter

Combines results of the two child filters by taking the pairwise minimum of the results of the first and second.

Inherits BinaryFilter class.

operation(element_a, element_b)

The pairwise binary operation used to combine results from the two child filters.

Parameters

• element_a (object) – The element from the data filter_a operated on.

• element_b (object) – The element from the data filter_b operated on.

class dpemu.filters.filter.Modulo(filter_a, filter_b)

Bases: dpemu.filters.filter.BinaryFilter

Combines results of the two child filters by taking the results of the first modulo results of the second.

Inherits BinaryFilter class.

operation(element_a, element_b)

The pairwise binary operation used to combine results from the two child filters.

Parameters

• element_a (object) – The element from the data filter_a operated on.

• element_b (object) – The element from the data filter_b operated on.

class dpemu.filters.filter.Multiplication(filter_a, filter_b)

Bases: dpemu.filters.filter.BinaryFilter

Combines results of the two child filters by multiplying them together.

Inherits BinaryFilter class.

operation(element_a, element_b)

The pairwise binary operation used to combine results from the two child filters.

Parameters

• element_a (object) – The element from the data filter_a operated on.

• element_b (object) – The element from the data filter_b operated on.

class dpemu.filters.filter.Or(filter_a, filter_b)

Bases: dpemu.filters.filter.BinaryFilter

Combines results of the two child filters with bitwise OR.

Inherits BinaryFilter class.

operation(element_a, element_b)

The pairwise binary operation used to combine results from the two child filters.

Parameters

• element_a (object) – The element from the data filter_a operated on.

• element_b (object) – The element from the data filter_b operated on.

class dpemu.filters.filter.Subtraction(filter_a, filter_b)

Bases: dpemu.filters.filter.BinaryFilter

Combines results of the two child filters by subtracting the results of the second from the firsts.

Inherits BinaryFilter class.

operation(element_a, element_b)

The pairwise binary operation used to combine results from the two child filters.

Parameters

• element_a (object) – The element from the data filter_a operated on.

• element_b (object) – The element from the data filter_b operated on.

class dpemu.filters.filter.Xor(filter_a, filter_b)

Bases: dpemu.filters.filter.BinaryFilter

Combines results of the two child filters with bitwise XOR.

Inherits BinaryFilter class.

operation(element_a, element_b)

The pairwise binary operation used to combine results from the two child filters.

Parameters

• element_a (object) – The element from the data filter_a operated on.

• element_b (object) – The element from the data filter_b operated on.

dpemu.filters.image module

class dpemu.filters.image.Blur(repeats_id, radius_id=None)

Bases: dpemu.filters.filter.Filter

Replaces the values of each pixel with the average values within the specified radius of it, iterated a given number of times.

Inherits Filter class.

__init__(repeats_id, radius_id=None)

Parameters

• repeats_id (str) – The key mapping to the number of iterations done.

• radius_id (str) – The key mapping to the radius to average over.

apply(node_data, random_state, named_dims)

Applies the filter to the data.

Parameters

• node_data (numpy.ndarray) – Data to be modified as a NumPy array.

• random_state (mtrand.RandomState) – An instance of numpy.random.RandomState() random number generator.

• named_dims (dict) – Named dimensions.

class dpemu.filters.image.BlurGaussian(standard_dev_id)

Bases: dpemu.filters.filter.Filter

Blur image according to a zero-centered normal distribution.

Create blur in images by applying a Gaussian filter. The standard deviation of the Gaussian is taken as a parameter.

Inherits Filter class.

__init__(standard_dev_id)

Parameters

standard_dev_id (str) – The key mapping to the standard deviation of the distribution.

apply(node_data, random_state, named_dims)

Applies the filter to the data.

Parameters

• node_data (numpy.ndarray) – Data to be modified as a NumPy array.

• random_state (mtrand.RandomState) – An instance of numpy.random.RandomState() random number generator.

• named_dims (dict) – Named dimensions.

class dpemu.filters.image.Brightness(tar_id, rat_id, range_id)

Bases: dpemu.filters.filter.Filter

Increases or decreases brightness in the image.

scales the brightness of every pixel with the formula new = tar + (old - tar) e^(-2 rat old) where old is the old brightness value, new is the new brightness value, and rat and tar are parameters.

tar is the target brightness value. The brightness of every pixel will move towards this value. rat is the ratio that is replaced with the new brightness value. The higher the ratio, the closer the new brightness values will be to the target brightness value.

RGB values should be either reals in the range [0, 1] or integers in [0, 255]. The range-parameter should be set to 1 in the first case and 255 in the second.

Inherits Filter class.

__init__(tar_id, rat_id, range_id)

Parameters

• tar_id (str) – The key mapping to the target brightness value.

• rat_id (str) – The key mapping to the ratio used to replace old brightness.

• range_id (str) – The key mapping to the range of the RGB values.

apply(node_data, random_state, named_dims)

Applies the filter to the data.

Parameters

• node_data (numpy.ndarray) – Data to be modified as a NumPy array.

• random_state (mtrand.RandomState) – An instance of numpy.random.RandomState() random number generator.

• named_dims (dict) – Named dimensions.

class dpemu.filters.image.JPEG_Compression(quality_id)

Bases: dpemu.filters.filter.Filter

Applies JPEG compression to the image.

Compresses the image into a JPEG, then uncompresses it. Quality should be in range [1, 100], the bigger the less loss.

Inherits Filter class.

__init__(quality_id)

Parameters

quality_id (str) – The key mapping to the quality of the compression.

apply(node_data, random_state, named_dims)

Applies the filter to the data.

Parameters

• node_data (numpy.ndarray) – Data to be modified as a NumPy array.

• random_state (mtrand.RandomState) – An instance of numpy.random.RandomState() random number generator.

• named_dims (dict) – Named dimensions.

class dpemu.filters.image.LensFlare

Bases: dpemu.filters.filter.Filter

Adds a lens flare to the image.

Inherits Filter class.

apply(node_data, random_state, named_dims)

Applies the filter to the data.

Parameters

• node_data (numpy.ndarray) – Data to be modified as a NumPy array.

• random_state (mtrand.RandomState) – An instance of numpy.random.RandomState() random number generator.

• named_dims (dict) – Named dimensions.

class dpemu.filters.image.Rain(probability_id, range_id)

Bases: dpemu.filters.filter.Filter

Add rain to images.

For every position in the image, creates a raindrop there with the given probability.

RGB values should be either reals in the range [0, 1] or integers in [0, 255]. The range-parameter should be set to 1 in the first case and 255 in the second.

Inherits Filter class.

__init__(probability_id, range_id)

Parameters

• probability_id (str) – The key mapping to the probability a raindrop is created.

• range_id (str) – The key mapping to the range of the RGB values.

apply(node_data, random_state, named_dims)

Applies the filter to the data.

Parameters

• node_data (numpy.ndarray) – Data to be modified as a NumPy array.

• random_state (mtrand.RandomState) – An instance of numpy.random.RandomState() random number generator.

• named_dims (dict) – Named dimensions.

class dpemu.filters.image.Resolution(k_id)

Bases: dpemu.filters.filter.Filter

Makes resolution k times smaller.

Resolution is changed with the formula:

new_image[y][x] = image[k * (y // k)][k * (x // k)] for all y and x,

where // is Python’s integer division. K must be an integer.

Inherits Filter class.

__init__(k_id)

Parameters

k_id (str) – The key mapping to the value of k.

apply(node_data, random_state, named_dims)

Applies the filter to the data.

Parameters

• node_data (numpy.ndarray) – Data to be modified as a NumPy array.

• random_state (mtrand.RandomState) – An instance of numpy.random.RandomState() random number generator.

• named_dims (dict) – Named dimensions.

class dpemu.filters.image.Rotation(min_angle_id, max_angle_id=None)

Bases: dpemu.filters.filter.Filter

Rotates the image.

If only min_angle is provided, the the image is rotated according to the angle. If both min_angle and max_angle are provided, then the rotation angle is chosen randomly from the uniform distribution Uniform(min_angle, max_angle).

If the angle is positive, then the image is rotated counterclockwise. Otherwise, the image is rotated clockwise.

Inherits Filter class.

__init__(min_angle_id, max_angle_id=None)

Parameters

• min_angle_id (str) – The key mapping to the minimum angle to rotate by.

• max_angle_id (str) – The key mapping to the maximum angle to rotate by.

None by default. If None, the angle of rotation will always be min_angle.

apply(node_data, random_state, named_dims)

Applies the filter to the data.

Parameters

• node_data (numpy.ndarray) – Data to be modified as a NumPy array.

• random_state (mtrand.RandomState) – An instance of numpy.random.RandomState() random number generator.

• named_dims (dict) – Named dimensions.

class dpemu.filters.image.Saturation(tar_id, rat_id, range_id)

Bases: dpemu.filters.filter.Filter

Increases or decreases saturation in the image.

scales the saturation of every pixel with the formula new = tar + (old - tar) e^(-2 rat old) where old is the old saturation value, new is the new saturation value, and rat and tar are parameters.

tar is the target saturation value. The saturation of every pixel will move towards this value. rat is the ratio that is replaced with the new saturation value. The higher the ratio, the closer the new saturation values will be to the target saturation value.

RGB values should be either reals in the range [0, 1] or integers in [0, 255]. The range-parameter should be set to 1 in the first case and 255 in the second.

Inherits Filter class.

__init__(tar_id, rat_id, range_id)

Parameters

• tar_id (str) – The key mapping to the target saturation value.

• rat_id (str) – The key mapping to the ratio used to replace old saturation.

• range_id (str) – The key mapping to the range of the RGB values.

apply(node_data, random_state, named_dims)

Applies the filter to the data.

Parameters

• node_data (numpy.ndarray) – Data to be modified as a NumPy array.

• random_state (mtrand.RandomState) – An instance of numpy.random.RandomState() random number generator.

• named_dims (dict) – Named dimensions.

class dpemu.filters.image.Snow(snowflake_probability_id, snowflake_alpha_id, snowstorm_alpha_id)

Bases: dpemu.filters.filter.Filter

Add snow to images.

This filter adds snow to images, and it uses Pierrre Vigier’s implementation of 2d perlin noise.

Pierre Vigier’s implementation of 2d perlin noise with slight changes. https://github.com/pvigier/perlin-numpy

The original code is licensed under MIT License:

MIT License

Copyright (c) 2019 Pierre Vigier

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Inherits Filter class.

__init__(snowflake_probability_id, snowflake_alpha_id, snowstorm_alpha_id)

Parameters

• snowflake_probability_id (str) – The key mapping to the probability a snowflake is created at every position.

• snowflake_alpha_id (str) – The key mapping to the alpha-value of snowflakes.

• snowstorm_alpha_id (str) – The key mapping to the alpha-value of the background snow.

apply(node_data, random_state, named_dims)

Applies the filter to the data.

Parameters

• node_data (numpy.ndarray) – Data to be modified as a NumPy array.

• random_state (mtrand.RandomState) – An instance of numpy.random.RandomState() random number generator.

• named_dims (dict) – Named dimensions.

class dpemu.filters.image.StainArea(probability_id, radius_generator_id, transparency_percentage_id)

Bases: dpemu.filters.filter.Filter

Adds stains to images.

With the given probability at every pixel creates a black stain with the given transparency and radius generated by the radius_generator function given as a parameter.

Inherits Filter class.

__init__(probability_id, radius_generator_id, transparency_percentage_id)

Parameters

• probability_id (str) – The key mapping to the probability of creating a stain.

• radius_generator_id (str) – The key mapping to the radius generator.

• transparency_percentage_id (str) – The key mapping to the transparency of stains.

apply(node_data, random_state, named_dims)

Applies the filter to the data.

Parameters

• node_data (numpy.ndarray) – Data to be modified as a NumPy array.

• random_state (mtrand.RandomState) – An instance of numpy.random.RandomState() random number generator.

• named_dims (dict) – Named dimensions.

dpemu.filters.sound module

class dpemu.filters.sound.ClipWAV(dyn_range_id)

Bases: dpemu.filters.filter.Filter

Clips the WAV data to a scaled range

Scales the range of the data by the given parameter, then sets values smaller than the new lower bound to the lower bound and values larger than the new upper bound to the new upper bound.

Inherits Filter class.

__init__(dyn_range_id)

Parameters

dyn_range_id (str) – The key mapping to the value to scale the range of values by.

apply(node_data, random_state, named_dims)

Applies the filter to the data.

Parameters

• node_data (numpy.ndarray) – Data to be modified as a NumPy array.

• random_state (mtrand.RandomState) – An instance of numpy.random.RandomState() random number generator.

• named_dims (dict) – Named dimensions.

dpemu.filters.text module

class dpemu.filters.text.MissingArea(probability_id, radius_generator_id, missing_value_id)

Bases: dpemu.filters.filter.Filter

Emulates stains in text causing characters to not be readable.

At every position in the input text, with the given probability, generates a random radius with the radius_generator parameter function, and replaces all characters within that radius with the given missing value.

Inherits Filter class.

__init__(probability_id, radius_generator_id, missing_value_id)

Parameters

• probability_id (str) – The key mapping to the probability of creating a stain at every position.

• radius_generator_id (str) – The key mapping to the radius_generator function.

• missing_value_id (str) – The key mapping to the missing value to replace characters with.

apply(node_data, random_state, named_dims)

Applies the filter to the data.

Parameters

• node_data (numpy.ndarray) – Data to be modified as a NumPy array.

• random_state (mtrand.RandomState) – An instance of numpy.random.RandomState() random number generator.

• named_dims (dict) – Named dimensions.

class dpemu.filters.text.OCRError(normalized_params_id, p_id)

Bases: dpemu.filters.filter.Filter

Emulates optical character recognition (OCR) errors.

Provided a probability distribution specifying how probable it is to mistakenly read a given character as another, randomly replaces characters according to that distribution. Example weights for the distribution can be found in the data directory. These files are:

example_ocr_error_config.json

These weights can be loaded and normalized into a probability distribution using functions from dpemu/pg_utils.py.

Inherits Filter class.

__init__(normalized_params_id, p_id)

Parameters

• normalized_params_id (str) – The key mapping to the character replacement probability distribution.

• p_id (str) – The key mapping to the probability distribution of a character replacement being applied.

apply(node_data, random_state, named_dims)

Applies the filter to the data.

Parameters

• node_data (numpy.ndarray) – Data to be modified as a NumPy array.

• random_state (mtrand.RandomState) – An instance of numpy.random.RandomState() random number generator.

• named_dims (dict) – Named dimensions.

generate_ocr_errors(string_, random_state)

replace_char(c, random_state)

class dpemu.filters.text.Uppercase(probability_id)

Bases: dpemu.filters.filter.Filter

Randomly convert characters to uppercase.

For each character in the input text, with the given probability converts it to uppercase.

Inherits Filter class.

__init__(probability_id)

Parameters

probability_id (str) – The key mapping to the probability of characters being converted to uppercase.

apply(node_data, random_state, named_dims)

Applies the filter to the data.

Parameters

• node_data (numpy.ndarray) – Data to be modified as a NumPy array.

• random_state (mtrand.RandomState) – An instance of numpy.random.RandomState() random number generator.

• named_dims (dict) – Named dimensions.

dpemu.filters.time_series module

class dpemu.filters.time_series.Gap(prob_break_id, prob_recover_id, missing_value_id)

Bases: dpemu.filters.filter.Filter

Introduce gaps to time series data by simulating sensor failure.

Models the state of the sensor with a Markov chain. The sensor always starts in a working state. During every unit of time, if the sensor is working, it breaks with the first specified probability, and if it is currently broken, it starts working with the second specified probability.

While the sensor is broken, values produced by it will be replaced with the provided missing value. Otherwise the original data remains unchanged.

Inherits Filter class.

__init__(prob_break_id, prob_recover_id, missing_value_id)

Parameters

• prob_break_id (str) – The key mapping to the probability the working sensor breaks in one unit of time.

• prob_recover_id (str) – The key mapping to the probability of the sensor recovering in one unit of time.

• missing_value_id (str) – The key mapping to the value that the broken sensor produces.

apply(node_data, random_state, named_dims)

Applies the filter to the data.

Parameters

• node_data (numpy.ndarray) – Data to be modified as a NumPy array.

• random_state (mtrand.RandomState) – An instance of numpy.random.RandomState() random number generator.

• named_dims (dict) – Named dimensions.

class dpemu.filters.time_series.SensorDrift(magnitude_id)

Bases: dpemu.filters.filter.Filter

Emulates sensor values drifting due to a malfunction in the sensor.

Magnitude is the linear increase in drift during time period t_i -> t_i+1.

Inherits Filter class.

__init__(magnitude_id)

Parameters

magnitude_id (str) – The key mapping to the magnitude value.

apply(node_data, random_state, named_dims)

Applies the filter to the data.

Parameters

• node_data (numpy.ndarray) – Data to be modified as a NumPy array.

• random_state (mtrand.RandomState) – An instance of numpy.random.RandomState() random number generator.

• named_dims (dict) – Named dimensions.

Module contents

class dpemu.filters.Filter

Bases: abc.ABC

A Filter is an error source which can be attached to an Array node.

The apply method applies the filter to the data. A filter may always assume that it is acting upon a NumPy array. (if the underlying data object is not a NumPy array, the required conversions are performed by the Array node to which the Filter is attached.)

Parameters

ABC (object) – Helper class that provides a standard way to create

an abstract class using inheritance.

__init__()

Set the seeds for the RNG’s of NumPy and Python.

abstract apply(node_data, random_state, named_dims)

Applies the filter to the data.

Parameters

• node_data (numpy.ndarray) – Data to be modified as a NumPy array.

• random_state (mtrand.RandomState) – An instance of numpy.random.RandomState() random number generator.

• named_dims (dict) – Named dimensions.

set_params(params_dict)

Set parameters for error generation.

Parameters

params_dict (dict) – A dictionary containing key-value pairs of error parameters.

class dpemu.filters.Identity

Bases: dpemu.filters.filter.Filter

Acts as the identity operator, thus doesn’t modify the data.

Inherits Filter class.

apply(node_data, random_state, named_dims)

Applies the filter to the data.

Parameters

• node_data (numpy.ndarray) – Data to be modified as a NumPy array.

• random_state (mtrand.RandomState) – An instance of numpy.random.RandomState() random number generator.

• named_dims (dict) – Named dimensions.

class dpemu.filters.Addition(filter_a, filter_b)

Bases: dpemu.filters.filter.BinaryFilter

Combines results of the two child filters by adding them together.

Inherits BinaryFilter class.

operation(element_a, element_b)

The pairwise binary operation used to combine results from the two child filters.

Parameters

• element_a (object) – The element from the data filter_a operated on.

• element_b (object) – The element from the data filter_b operated on.

class dpemu.filters.Subtraction(filter_a, filter_b)

Bases: dpemu.filters.filter.BinaryFilter

Combines results of the two child filters by subtracting the results of the second from the firsts.

Inherits BinaryFilter class.

operation(element_a, element_b)

The pairwise binary operation used to combine results from the two child filters.

Parameters

• element_a (object) – The element from the data filter_a operated on.

• element_b (object) – The element from the data filter_b operated on.

class dpemu.filters.Multiplication(filter_a, filter_b)

Bases: dpemu.filters.filter.BinaryFilter

Combines results of the two child filters by multiplying them together.

Inherits BinaryFilter class.

operation(element_a, element_b)

The pairwise binary operation used to combine results from the two child filters.

Parameters

• element_a (object) – The element from the data filter_a operated on.

• element_b (object) – The element from the data filter_b operated on.

class dpemu.filters.Division(filter_a, filter_b)

Bases: dpemu.filters.filter.BinaryFilter

Combines results of the two child filters by dividing the results of the first by the seconds.

Inherits BinaryFilter class.

operation(element_a, element_b)

The pairwise binary operation used to combine results from the two child filters.

Parameters

• element_a (object) – The element from the data filter_a operated on.

• element_b (object) – The element from the data filter_b operated on.

class dpemu.filters.IntegerDivision(filter_a, filter_b)

Bases: dpemu.filters.filter.BinaryFilter

Combines results of the two child filters by perfoming integer division on the results of the first by the results of the second.

The division is done with python’s // operator.

Inherits BinaryFilter class.

operation(element_a, element_b)

The pairwise binary operation used to combine results from the two child filters.

Parameters

• element_a (object) – The element from the data filter_a operated on.

• element_b (object) – The element from the data filter_b operated on.

class dpemu.filters.Constant(value_id)

Bases: dpemu.filters.filter.Filter

Overwrites all values in the data with the given value.

Inherits Filter class.

__init__(value_id)

Parameters

value_id – The key mapping to the value to overwrite values in the data with.

apply(node_data, random_state, named_dims)

Applies the filter to the data.

Parameters

• node_data (numpy.ndarray) – Data to be modified as a NumPy array.

• random_state (mtrand.RandomState) – An instance of numpy.random.RandomState() random number generator.

• named_dims (dict) – Named dimensions.

class dpemu.filters.And(filter_a, filter_b)

Bases: dpemu.filters.filter.BinaryFilter

Combines results of the two child filters with bitwise AND.

Inherits BinaryFilter class.

operation(element_a, element_b)

The pairwise binary operation used to combine results from the two child filters.

Parameters

• element_a (object) – The element from the data filter_a operated on.

• element_b (object) – The element from the data filter_b operated on.

class dpemu.filters.Xor(filter_a, filter_b)

Bases: dpemu.filters.filter.BinaryFilter

Combines results of the two child filters with bitwise XOR.

Inherits BinaryFilter class.

operation(element_a, element_b)

The pairwise binary operation used to combine results from the two child filters.

Parameters

• element_a (object) – The element from the data filter_a operated on.

• element_b (object) – The element from the data filter_b operated on.

class dpemu.filters.Or(filter_a, filter_b)

Bases: dpemu.filters.filter.BinaryFilter

Combines results of the two child filters with bitwise OR.

Inherits BinaryFilter class.

operation(element_a, element_b)

The pairwise binary operation used to combine results from the two child filters.

Parameters

• element_a (object) – The element from the data filter_a operated on.

• element_b (object) – The element from the data filter_b operated on.

class dpemu.filters.Min(filter_a, filter_b)

Bases: dpemu.filters.filter.BinaryFilter

Combines results of the two child filters by taking the pairwise minimum of the results of the first and second.

Inherits BinaryFilter class.

operation(element_a, element_b)

The pairwise binary operation used to combine results from the two child filters.

Parameters

• element_a (object) – The element from the data filter_a operated on.

• element_b (object) – The element from the data filter_b operated on.

class dpemu.filters.Max(filter_a, filter_b)

Bases: dpemu.filters.filter.BinaryFilter

Combines results of the two child filters by taking the pairwise maximum of the results of the first and second.

Inherits BinaryFilter class.

operation(element_a, element_b)

The pairwise binary operation used to combine results from the two child filters.

Parameters

• element_a (object) – The element from the data filter_a operated on.

• element_b (object) – The element from the data filter_b operated on.

class dpemu.filters.BinaryFilter(filter_a, filter_b)

Bases: dpemu.filters.filter.Filter

Abstract Filter applying two given filters to the data, combining the results with a pairwise binary operation.

The pairwise binary operation is specified by the inheriting class by overriding the operation-function.

Inherits Filter class.

__init__(filter_a, filter_b)

Parameters

• filter_a (str) – The first filter.

• filter_b (str) – The second filter.

apply(node_data, random_state, named_dims)

Applies the filter to the data.

Parameters

• node_data (numpy.ndarray) – Data to be modified as a NumPy array.

• random_state (mtrand.RandomState) – An instance of numpy.random.RandomState() random number generator.

• named_dims (dict) – Named dimensions.

abstract operation(element_a, element_b)

The pairwise binary operation used to combine results from the two child filters.

Parameters

• element_a (object) – The element from the data filter_a operated on.

• element_b (object) – The element from the data filter_b operated on.

class dpemu.filters.Modulo(filter_a, filter_b)

Bases: dpemu.filters.filter.BinaryFilter

Combines results of the two child filters by taking the results of the first modulo results of the second.

Inherits BinaryFilter class.

operation(element_a, element_b)

The pairwise binary operation used to combine results from the two child filters.

Parameters

• element_a (object) – The element from the data filter_a operated on.

• element_b (object) – The element from the data filter_b operated on.

class dpemu.filters.Difference(ftr)

Bases: dpemu.filters.filter.Filter

Returns change to data from filter

Given a filter, applies the filter to the data, then subtracting the original. Functions identically to Subtraction(filter, Identity()).

Inherits BinaryFilter class.

apply(node_data, random_state, named_dims)

Applies the filter to the data.

Parameters

• node_data (numpy.ndarray) – Data to be modified as a NumPy array.

• random_state (mtrand.RandomState) – An instance of numpy.random.RandomState() random number generator.

• named_dims (dict) – Named dimensions.