API Reference

Utilities

Auto-Download Module

This function downloads Sunsmart PV Power Plant data from the OSF database.

dddt_gfm.util.io.OSF_download(file_number, output_file, output_dir)

Helper function that downloads CSV files from OSF Database. The urls are already in the function so whichever one can be downloaded

Parameters:

• file_number (int) – an integer (0-89) that specifies which csv file to download -> 0,1,2 for 2001; 3,4,5 for 2002; etc…

• output_file (string) – name for CSV file

• output_dir (string) – directory for which the file should be saved in

class dddt_gfm.util.callbacks.LossPlotCallback(*args: Any, **kwargs: Any)

Bases: Callback

on_train_epoch_end(trainer, pl_module)

on_validation_epoch_end(training, pl_module)

plot_losses()

Models

class dddt_gfm.models.stGAE.STConvAE(*args: Any, **kwargs: Any)

Bases: Module

forward(x, edge_index, edge_weight)

class dddt_gfm.models.stGAE.STConvDecoder(num_nodes, in_channels, hidden_channels, out_channels, kernel_size, kernel_size_de, stride, padding, K, normalization='sym', bias=True)

Bases: Module

forward(X, edge_index, edge_weight=None)

Forward pass. If edge weights are not present the forward pass defaults to an unweighted graph.

Return type:

FloatTensor

Arg types:

• X (PyTorch FloatTensor) - Sequence of node features of shape (Batch size X Input time steps X Num nodes X In channels).

• edge_index (PyTorch LongTensor) - Graph edge indices.

• edge_weight (PyTorch LongTensor, optional)- Edge weight vector.

Return types:

• T (PyTorch FloatTensor) - Sequence of node features.

class dddt_gfm.models.stGAE.STConvEncoder(num_nodes, in_channels, hidden_channels, out_channels, kernel_size, stride, padding, K, normalization='sym', bias=True)

Bases: Module

forward(X, edge_index, edge_weight=None)

Forward pass. If edge weights are not present the forward pass defaults to an unweighted graph.

Return type:

FloatTensor

Arg types:

• X (PyTorch FloatTensor) - Sequence of node features of shape (Batch size X Input time steps X Num nodes X In channels).

• edge_index (PyTorch LongTensor) - Graph edge indices.

• edge_weight (PyTorch LongTensor, optional)- Edge weight vector.

Return types:

• T (PyTorch FloatTensor) - Sequence of node features.

class dddt_gfm.models.stGAE.TemporalConv(in_channels, out_channels, kernel_size, stride, padding)

Bases: Module

Parameters:

• in_channels (int) – Number of input features.

• out_channels (int) – Number of output features.

• kernel_size (int) – Convolutional kernel size.

forward(X)

Forward pass through temporal convolution block.

Return type:

FloatTensor

Arg types:

• X (torch.FloatTensor) - Input data of shape

  (batch_size, input_time_steps, num_nodes, in_channels).

Return types:

• H (torch.FloatTensor) - Output data of shape

  (batch_size, in_channels, num_nodes, input_time_steps).

class dddt_gfm.models.stGAE.TemporalDeConv1(in_channels, out_channels, kernel_size, stride, padding)

Bases: Module

Parameters:

• in_channels (int) – Number of input features.

• out_channels (int) – Number of output features.

• kernel_size (int) – Convolutional kernel size.

forward(X)

Forward pass through temporal convolution block.

Return type:

FloatTensor

Arg types:

• X (torch.FloatTensor) - Input data of shape

  (batch_size, input_time_steps, num_nodes, in_channels).

Return types:

• H (torch.FloatTensor) - Output data of shape

  (batch_size, in_channels, num_nodes, input_time_steps).

class dddt_gfm.models.stGAE.TemporalDeConv2(in_channels, out_channels, kernel_size, stride)

Bases: Module

Parameters:

• in_channels (int) – Number of input features.

• out_channels (int) – Number of output features.

• kernel_size (int) – Convolutional kernel size.

forward(X)

Forward pass through temporal convolution block.

Return type:

FloatTensor

Arg types:

• X (torch.FloatTensor) - Input data of shape

  (batch_size, input_time_steps, num_nodes, in_channels).

Return types:

• H (torch.FloatTensor) - Output data of shape

  (batch_size, in_channels, num_nodes, input_time_steps).

Dataloaders

class dddt_gfm.dataloaders.dataloader_DIW.DIWDataset(target_path, connectivity=None, eval_mask=None, index=None, mask=None, covariates=None, input_map=None, target_map=None, auxiliary_map=None, scalers=None, trend=None, transform=None, window=12, stride=1, window_lag=1, precision=32, name=None)

Bases: ImputationDataset

dddt_gfm.dataloaders.dataloader_sunsmart.extract_numbers(file_paths)

class dddt_gfm.dataloaders.dataloader_sunsmart.stDataloader(*args: Any, **kwargs: Any)

Bases: Dataset

add_channel(df, df_name='new_df', write=False)

apply_conditions(conditions_list, write=False)

Apply a list of conditions to the specified DataFrame in a list using conditions based on other DataFrames.

Parameters:

conditions_list (list of tuples) – Each tuple contains (target_index, conditions, fill_df_index, fallback_index). - conditions: A list of tuples, each containing (channel_x, operator, channel_y, scalar). - conditions_op: Operator to act on conditional statments. - channel_if_true: Name of DataFrame from self.channel_names to get the value_if_true from. If None, use scalar_values. - channel_if_false: Name of DataFrame self.channel_names to return values from if conditions are not met. - scalar_if_true: Value used for filling if fill_df_index = None. - scalar_if_true: Value used for returning if df_df_index = None. - condition_name: Name corresponding to this condition.

Returns:

The modified DataFrame list after applying all conditions.

Return type:

pd.DataFrame

col_normalization(channel, norm_scale=100)

data_availibility(measurement_cols, list_dfs=None, time_var='tmst', write=False)

data_to_model_input_transform(normalize=True)

find_indices_within_bins(bins)

generate_datasetgrade_adjacency_matrix(grade_col='', min_grade_threshold=[0.7], max_diff_thresholds=[0.75], weighted=True, nodeID_col='nodeID', write=False)

generate_distance_adjacency_matrix(latitude='latd', longitude='longd', epsilons=[0.75], weighted=True, nodeID_col='nodeID', write=False)

generate_feature_adjacency_matrix(feature_cols=[], nodeID_col='nodeID', write=False)

get_channel_indices()

get_longest_timespan(list_dfs=None, freq='15min', time_var='tmst', tz='EST', set_params=True)

get_model_inputs(input_channels, output_channels, mask_channels)

get_model_parameters()

get_modelprediction_stats(model_list=[''], write_name='')

get_windows()

haversine(lon1, lat1, lon2, lat2)

Calculate the great circle distance between two points on the earth (specified in decimal degrees)

intialize_directories(parent_directory)

read_files_with_nodeID()

Reads a list of CSV files and adds a ‘nodeID’ column to each DataFrame with the name of the file (without .csv).

Parameters: filepaths (list): A list of file paths to the CSV files.

Returns: list: A list of DataFrames with the ‘nodeID’ column added.

read_from_tabular(raw_parquet_path=None, df_split_col=None, df_id_col=None, write=False)

set_date_range(start, stop, freq, tz='UTC')

set_model_inputs(input_channels, output_channels, mask_channels, adjacency_matrices)

set_train_test_splits(splits)

set_window_size(window_size)

split_columns_randomly()

split_nodes_from_rowwise(df, df_split_col, df_id_col=None, write=False)

split_windows_randomly()

subset(data, dim)

Splits the array along a specified dimension using predefined column indices.

Parameters:

• array (np.array) – The input array to be split.

• dim (int) – The dimension along which to split the array.

Returns:

A tuple containing the subsets of the array for training, testing, and validation.

Return type:

tuple

tabular_to_matrix(measurement_cols, list_dfs=None, time_var='tmst', write=False)

ts_reindex_with_interpolation(df_to_reindex, start_freq='1h', chunk_size=None, method='linear', order=3)

Reindex a DataFrame to a new index range and perform spline interpolation.

Parameters:

df_to_reindex (pd.DataFrame) – Input DataFrame.

Returns:

DataFrame with reindexed index and interpolated values.

Return type:

pd.DataFrame

weight_subset(data, dim)

Splits the data along a specified dimension using predefined column indices, while also dropping the corresponding row indices.

Parameters:

• data (np.array) – The input array to be split.

• dim (int) – The dimension along which to split the array.

Returns:

A tuple containing the subsets of the array for training, testing, and validation.

Return type:

tuple

weight_to_model_input_transform(datasplit='training')

dddt_gfm.dataloaders.dataloader_sunsmart.within_n_std_dev(mean_df, var_df, input_df, n=1)

dddt_gfm.dataloaders.rwb_dataloader_sunsmart.extract_numbers(file_paths)

class dddt_gfm.dataloaders.rwb_dataloader_sunsmart.stDataloader(*args: Any, **kwargs: Any)

Bases: Dataset

add_channel(df, df_name='new_df', write=False)

apply_conditions(conditions_list, write=False)

Apply a list of conditions to the specified DataFrame in a list using conditions based on other DataFrames.

Parameters:

conditions_list (list of tuples) – Each tuple contains (target_index, conditions, fill_df_index, fallback_index). - conditions: A list of tuples, each containing (channel_x, operator, channel_y, scalar). - conditions_op: Operator to act on conditional statments. - channel_if_true: Name of DataFrame from self.channel_names to get the value_if_true from. If None, use scalar_values. - channel_if_false: Name of DataFrame self.channel_names to return values from if conditions are not met. - scalar_if_true: Value used for filling if fill_df_index = None. - scalar_if_true: Value used for returning if df_df_index = None. - condition_name: Name corresponding to this condition.

Returns:

The modified DataFrame list after applying all conditions.

Return type:

pd.DataFrame

batch_plot(x, mask, y, y_pred, plot_nodes=5, num_plots_per_node=3, plot_range=200, plot_type='sunangle')

col_normalization(channel, norm_scale=100)

data_availibility(measurement_cols, list_dfs=None, time_var='tmst', write=False)

data_to_model_input_transform(normalize=True)

find_indices_within_bins(bins)

generate_datasetgrade_adjacency_matrix(grade_col='', min_grade_threshold=[0.7], max_diff_thresholds=[0.75], weighted=True, nodeID_col='nodeID', write=False)

generate_distance_adjacency_matrix(latitude='latd', longitude='longd', epsilons=[0.75], weighted=True, nodeID_col='nodeID', write=False)

generate_feature_adjacency_matrix(feature_cols=[], nodeID_col='nodeID', write=False)

get_channel_indices()

get_longest_timespan(list_dfs=None, freq='15min', time_var='tmst', tz='EST', set_params=True)

get_model_inputs(input_channels, output_channels, mask_channels)

get_model_parameters()

get_modelprediction_stats(model_list=[''], write_name='')

get_windows()

haversine(lon1, lat1, lon2, lat2)

Calculate the great circle distance between two points on the earth (specified in decimal degrees)

intialize_directories(parent_directory)

read_files_with_nodeID()

Reads a list of CSV files and adds a ‘nodeID’ column to each DataFrame with the name of the file (without .csv).

Parameters: filepaths (list): A list of file paths to the CSV files.

Returns: list: A list of DataFrames with the ‘nodeID’ column added.

read_from_tabular(raw_parquet_path=None, df_split_col=None, df_id_col=None, write=False)

set_date_range(start, stop, freq, tz='UTC')

set_model_inputs(input_channels, output_channels, mask_channels, adjacency_matrices)

set_train_test_splits(splits)

set_window_size(window_size)

split_columns_randomly()

split_nodes_from_rowwise(df, df_split_col, df_id_col=None, write=False)

split_windows_randomly()

subset(data, dim)

Splits the array along a specified dimension using predefined column indices.

Parameters:

• array (np.array) – The input array to be split.

• dim (int) – The dimension along which to split the array.

Returns:

A tuple containing the subsets of the array for training, testing, and validation.

Return type:

tuple

tabular_to_matrix(measurement_cols, list_dfs=None, time_var='tmst', write=False)

ts_reindex_with_interpolation(df_to_reindex, start_freq='1h', chunk_size=None, method='linear', order=3)

Reindex a DataFrame to a new index range and perform spline interpolation.

Parameters:

df_to_reindex (pd.DataFrame) – Input DataFrame.

Returns:

DataFrame with reindexed index and interpolated values.

Return type:

pd.DataFrame

weight_subset(data, dim)

Splits the data along a specified dimension using predefined column indices, while also dropping the corresponding row indices.

Parameters:

• data (np.array) – The input array to be split.

• dim (int) – The dimension along which to split the array.

Returns:

A tuple containing the subsets of the array for training, testing, and validation.

Return type:

tuple

weight_to_model_input_transform(datasplit='training')

dddt_gfm.dataloaders.rwb_dataloader_sunsmart.within_n_std_dev(mean_df, var_df, input_df, n=1)