.. This file is part of Sympathy for Data. .. Copyright (c) 2010-2019 Combine Control Systems AB .. .. SYMPATHY FOR DATA COMMERCIAL LICENSE .. You should have received a link to the License with Sympathy for Data. .. _`nodeoverview`: Node overview ============= Here follows lists of most of the nodes in the standard library of Sympathy for Data. Nodes originating from toolkits are not yet listed here. Analysis -------- The **Analysis** folder holds mathematical nodes for data analysis. Statistics and various algorithms for time-series analysis. More nodes are available in the Timeseries toolkit. Statistics ^^^^^^^^^^ * :ref:`Descriptive Statistics` * :ref:`Heatmap Calculation` * :ref:`Histogram Calculation` Data Processing --------------- In **Data Processing** you find nodes for formatting and structuring data, commonly used in a pre-processing step for clean up. The operations inculde: converting between data formats, naming, setting attributes, replacing values etc. Attributes ^^^^^^^^^^ * :ref:`Set `/:ref:`Get ` Table(s) Name * :ref:`Set `/:ref:`Get ` Table(s) Attributes * :ref:`Set `/:ref:`Get ` column attributes in Table Calculate ^^^^^^^^^ The **Calculate** tag holds special nodes which manipulate the data by applying functions. * :ref:`Calculator` Apply pre-defined Python functions (arithmetics, comparators, logics etc.) to any data type. * :ref:`Calculator List` * :ref:`F(x)` The most versitile node in the Library. This node applies any user-defined Python function to any datatype. * :ref:`F(x) List` * :ref:`Json Calculator` Apply functions to JSON files, keeping the JSON format. * :ref:`Matlab Table` Similar to :ref:`F(x)` but functions are written in Matlab instead of Python. Convert ^^^^^^^ Nodes that convert between different Sympathy data formats. * :ref:`ADAF to Table`/:ref:`Table to ADAF` * :ref:`Table to Datasources`/:ref:`Datasource to Table` * JSON to :ref:`Dict `/:ref:`List `/:ref:`Text `/:ref:`Table ` * :ref:`Table to Text`/:ref:`Text to Table` * etc.. Data ^^^^ Nodes that edit data. * :ref:`Convert columns in Table` Change the datatype of a table column, e.g integer to floating point. * :ref:`Hold value Table` Replace nan values with the previous non-nan. * :ref:`Replace values in Table` Find and replace any value in a Table. * :ref:`Replace values in Table with Table` Use a Table with find-replace pairs to replace several values in target Table. * :ref:`Rename datasource with regex` Use regular expression to edit the path of a Datasource. Index ^^^^^ * :ref:`ADAF structure to Table` * :ref:`Create Index Table` Adds an index column with new indices for unique values in the Table. Select ^^^^^^ Nodes that selects/discards parts of data. * :ref:`Drop NaN Table`/:ref:`Drop NaN Tables` * :ref:`Drop masked values in Table` * :ref:`Fill masked values in Table` * :ref:`Filter rows in Table`/:ref:`Filter rows in Tables` * :ref:`Mask values in Table` * :ref:`Remove Json` * :ref:`Select Json` * :ref:`Select categories in ADAFs` * :ref:`Select columns by type in Table`/:ref:`Select columns by type in Tables` * :ref:`Select columns in ADAF with structure Table` * :ref:`Select columns in ADAFs with structure Table`/:ref:`Select columns in ADAFs with structure Tables` * :ref:`Select columns in Table`/:ref:`Select columns in Tables` * :ref:`Select columns in Table with Regex` * :ref:`Select columns in Table with Table` * :ref:`Select columns in Tables with Table` * :ref:`Select keys in Json` * :ref:`Select rows in Table`/:ref:`Select rows in Tables` * :ref:`Select rows in Table with Table` * :ref:`Select rows in Tables with Table` * :ref:`Slice List` * :ref:`Slice columns in Table`/:ref:`Slice columns in Tables` * :ref:`Slice rows in Table`/:ref:`Slice rows in Tables` * :ref:`Unique Table`/:ref:`Unique Tables` Structure ^^^^^^^^^ Nodes that perform non-mutating transformations such as transposing, sorting, joining and splitting. * :ref:`Cartesian Product Table`/:ref:`Cartesian Product Tables` * :ref:`Ensure columns in Tables with Table` * :ref:`Get column names in Table` * :ref:`HJoin Table`/:ref:`HJoin Tables` * :ref:`HJoin Tables pairwise` * :ref:`HSplit Table`/:ref:`HSplit Tables` * :ref:`Index columns in Table`/:ref:`Index columns in Tables` * :ref:`Index rows in Table`/:ref:`Index rows in Tables` * :ref:`Index table` * :ref:`Lookup Table`/:ref:`Lookup Tables` * :ref:`Match Table lengths`/:ref:`Match Tables lengths` * :ref:`Merge Table`/:ref:`Merge Tables` * :ref:`Pivot Table`/:ref:`Pivot Tables` * :ref:`Rename columns in ADAF with Table`/:ref:`Rename columns in ADAFs with Table` * :ref:`Rename columns in Table`/:ref:`Rename columns in Tables` * :ref:`Rename columns in Tables with Table` * :ref:`Reverse columns in Table`/:ref:`Reverse columns in Tables` * :ref:`Reverse rows in Table`/:ref:`Reverse rows in Tables` * :ref:`Set column names in Table with Table`/:ref:`Set column names in Tables with Table` * :ref:`Sort columns in Table` * :ref:`Sort rows in Table`/:ref:`Sort rows in Tables` * :ref:`Transpose Table`/:ref:`Transpose Tables` * :ref:`Unpivot Table`/:ref:`Unpivot Tables` * :ref:`VJoin Table`/:ref:`VJoin Tables` * :ref:`VJoin Tables pairwise` * :ref:`VSplit Table`/:ref:`VSplit Tables` Text ^^^^ * :ref:`Concatenate text`/:ref:`Concatenate texts` * :ref:`Jinja2 template` * :ref:`Split text` Development ----------- Nodes with the main purpose of aiding Sympathy development and testing. Debug ^^^^^ Export/Import the internal datastructure of any datatype. * :ref:`Debug Export` * :ref:`Debug Import` Example ^^^^^^^ Various examples of nodes for help in node creation are found under the **Example** tag. Test ^^^^ * :ref:`Assert Equal ADAF` * :ref:`Assert Equal Table` * :ref:`Conditional error/warning` Raises errors/warnings based on custom lambda expression. Disk ---- Under the **Disk** tag are a collection of nodes that do system operations such as copying, moving and deleting files. * :ref:`Archive files` * :ref:`Copy file`/:ref:`Copy files` * :ref:`Delete file`/:ref:`Delete files` * :ref:`Download URL to file` * :ref:`Move file`/:ref:`Move files` Generic ------- Configuration ^^^^^^^^^^^^^ Create and edit JSONs for configuration of other nodes * :ref:`Configure Json Parameters` Configure a given set of Json parameters in this nodes configuration GUI * :ref:`Create Json` Manually create a Json object. * :ref:`Create Json Parameters` Create a Sympathy parameter structure with Python code. * :ref:`Update Configuration with Table` Update the configuration parameters with values from a Table. Control ^^^^^^^ Here you find nodes for controlling the execution of the flow. * :ref:`Conditional Propagate` Propagate one of two inputs based on custom `if` condition. * :ref:`Propagate First Input` Always propagate the first input, useful for waiting until different parts have finished before advancing. * :ref:`Propagate First Input (Same Type)` Dict ^^^^ Nodes that do operations on Sympathy dict datatype. * :ref:`Dict items` Get key value pairs as Sympathy list of Sympathy tuples * :ref:`Dict keys` Get keys in list * :ref:`Dict values` Get values in list * :ref:`Get Item Dict` Get value based on key * :ref:`Insert Dict` Insert key, value pair * :ref:`Items to Dict` Convert list of key value pairs to dict * :ref:`Update Dict` Combine two dicts Lambda ^^^^^^ Nodes that work with the special datatype **Lambda**. Which are unexecuted flows packaged for later use. * :ref:`Apply` Apply (execute) the lambda flow on an argument. * :ref:`Extract Flows as Lambdas` Extract all top-level flows as Lambdas from a .syx file. * :ref:`Extract Lambdas` Extract all Lambdas from a .syx file. * :ref:`Map` Map (execute) the lambda flow on each element in a list. * :ref:`Parallel Map 2` Map in 2 parallel processes. * :ref:`Parallel Map 4` Map in 4 parallel processes. * :ref:`Parallel Map 4 Configurable` Map in 1-4 parallel processes. List ^^^^ Nodes that do operations on Sympathy dict datatype. * :ref:`Append List` Append an item to a list. * :ref:`Bisect List` Split list into equally big parts. * :ref:`Chunk List` Split list into a defined number of parts. * :ref:`Extend List` Combine two lists. * :ref:`Filter List Predicate` Filter items that meets custom condition. * :ref:`Filter List with Table` Select items based on values (True/False) in a Table. * :ref:`Flatten List` Flattens nested list. * :ref:`Get Item List` Get item at index. * :ref:`Group List` Create groups of nested lists. * :ref:`Index List` Select items based on index. * :ref:`Insert List` Insert an item in a list. * :ref:`Item to List` Convert item to list of length 1 * :ref:`Pad List` Pad a list to match length of other list. * :ref:`Pad List with Item` Pad a list to match length of other list with given item. * :ref:`Partition List Predicate` Divide list into two based on custom condition. * :ref:`Reverse List` * :ref:`Sort List` * :ref:`Transpose List` Swap the first and second layer of a nested list. Tuple ^^^^^ Nodes that do operations on Sympathy tuple datatype. * :ref:`Cartesian Product Tuple` Create a list of tuples with all combinations of elements from multiple lists. * :ref:`First Tuple2` Get first item in tuple with two items. * :ref:`Second Tuple2` Get second item in tuple with two items. * :ref:`Tuple` Create a tuple from two or more items. * :ref:`Untuple` Get items from a tuple. * :ref:`Unzip Tuple` Get items in separate lists from list of tuples. * :ref:`Zip Tuple` Create a list of tuples from two or more lists. Image Processing ---------------- In the **Image Processing** folder are all nodes that handle the :ref:`image port type`. More nodes are available in the Image Analysis toolkit. Extract statistics ^^^^^^^^^^^^^^^^^^ * :ref:`Extract Image Data` Extract data from image from specified pixel regions in a Table. * :ref:`Image Statistics` Extracts statistics with one of a selection of algorithms. Image Manipulation ^^^^^^^^^^^^^^^^^^ Nodes that operate on images, drawing, editing filtering etc. * :ref:`Color range conversion` * :ref:`Color space conversion` * :ref:`Corner detection` * :ref:`Draw on Image` * :ref:`Edge detection` * :ref:`Filter Image, Dual Output` * :ref:`Filter image` * :ref:`Morphological Image Operations` * :ref:`Morphology (single input)` * :ref:`Transform image` Input/Output ^^^^^^^^^^^^ Nodes that create, import and export images. * :ref:`Convert 2D Table to Image` * :ref:`Convert Image to 2D Table` * :ref:`Convert Image to Table` * :ref:`Convert Table to Image` * :ref:`Example Image` * :ref:`Generate Image` * :ref:`Load Image` * :ref:`Load Image List` * :ref:`Save Image` Layer operations ^^^^^^^^^^^^^^^^ Nodes that operate on the different color channels of an image, overlaying, merging and splitting them. * :ref:`Image to List` * :ref:`List to Image` * :ref:`Merge Image Channels` * :ref:`Overlay Images` * :ref:`Overlay Images List` * :ref:`Split Image Channels` Segmentation ^^^^^^^^^^^^ * :ref:`Label image` Create integer image with labels for each connected region (same pixel values). * :ref:`Threshold image` Boolean output from thresholding the pixel values. Input ----- Environment ^^^^^^^^^^^ * :ref:`Get Environment to Table` Choose between a number of environment variables to display in a Table. Generate ^^^^^^^^ Under the **Generate** tag are nodes that can generate data in various Sympathy formats. * :ref:`Create Table` * :ref:`Create Text` * :ref:`Empty` * :ref:`Generate Signal Table` * :ref:`Generate Signal Tables` * :ref:`Random ADAF` * :ref:`Random ADAFs` * :ref:`Random Table` * :ref:`Random Tables` Import ^^^^^^ Nodes used for importing data into the Sympathy workflow. All import nodes are preceeded by the :ref:`Datasource` node that specify the location to import from. * :ref:`ADAF` * :ref:`ADAFs` * :ref:`Datasource` * :ref:`File Datasources` * :ref:`Json` * :ref:`Jsons` * :ref:`Table` * :ref:`Tables` * :ref:`Text` * :ref:`Texts` Machine Learning ---------------- The **Machine Learning** tag holds nodes that work with the Model datatype. The models include classifiers, regressors, dimensionality reduction etc. More models are available in the Avanced Machine Learning extension. Apply model ^^^^^^^^^^^ Nodes that apply, trained or un-trained Models to data. * :ref:`Decision Function` Apply a trained Models decision function (if avaliable). * :ref:`Extract Attributes` Extract Model attributes into Table. * :ref:`Fit` Fit a Model to Table data. * :ref:`Fit Texts` Fit a Model to Text data. * :ref:`Fit Transform` Fit a transformation Model and apply it to the data. * :ref:`Fit Transform Text` * :ref:`Inverse Transform` If possible, reverse data transformation from a Model. * :ref:`Pipeline` Create a pipline of sucessive Models. * :ref:`Pipeline decomposition` * :ref:`Predict` Use Model to predict Y given X. * :ref:`Predict Probabilities` Use Model to predict Y given X, and return estimated probabilities. * :ref:`Select Features from Model` * :ref:`Set Input and Output Names` * :ref:`Transform` Apply a transformation Model to data. * :ref:`Transform Text` Apply a transformation Model to text. * :ref:`Voting Classifier` Use voting by multiple classifier Models. Dimensionality reduction ^^^^^^^^^^^^^^^^^^^^^^^^ Models that transform data by projecting onto a space of lower dimension. * :ref:`Kernel Principal Component Analysis (KPCA)` * :ref:`Partial Least Squares cross-decomposition (PLS regression)` * :ref:`Principal Component Analysis (PCA)` IO ^^ Import/export Models and generate ML datasets. * :ref:`Example datasets` * :ref:`Export Model` * :ref:`Features to Images` * :ref:`Generate classification dataset` * :ref:`Generate dataset blobs` * :ref:`Generate dataset blobs from table` * :ref:`Images to Features` * :ref:`Import Model` Metrics ^^^^^^^ Nodes that produce quantitative measures used to evaluate and compare the performance of the **Machine Learning** Models. * :ref:`Conditional Probabilty from Categories` * :ref:`Confusion Matrix` * :ref:`Learning Curve` * :ref:`ROC from Probabilities` * :ref:`R² regression score (R2)` * :ref:`Score` Parameters ^^^^^^^^^^ Nodes used for setting and extracting Model hyperparameters. * :ref:`Extract Parameters` * :ref:`Grid Parameter Search` * :ref:`Parameter Distribution` * :ref:`Randomized Parameter Search` * :ref:`Set Parameters` * :ref:`Simulated Annealing Parameter Search` Partitioning and validation ^^^^^^^^^^^^^^^^^^^^^^^^^^^ Nodes used for partitioning data, cross-validataion and train-test split. * :ref:`Group K-fold Cross Validation` * :ref:`K-fold Cross Validation` * :ref:`Leave One Group out Cross Validation` * :ref:`Score Cross Validation` * :ref:`Simple Train-Test Split` * :ref:`Split Data for Cross Validation` * :ref:`Stratified K-fold cross validation` * :ref:`Time Series K-fold Based Cross Validation` Processing ^^^^^^^^^^ Nodes for pre-processing data for ML purposes. Scaling, normalizing, encoding labels etc. * :ref:`Binarizer` * :ref:`Categorical Encoder` * :ref:`Imputer` * :ref:`Label Binarizer` * :ref:`Label Encoder` * :ref:`Max Abs Scaler` * :ref:`Normalizer` * :ref:`One-Hot Encoder` * :ref:`Polynomial Features` * :ref:`Robust Scaler` * :ref:`Standard Scaler` * :ref:`Text Count Vectorizer` Regression ^^^^^^^^^^ Regressor Models * :ref:`Epsilon Support Vector Regression` * :ref:`Kernel Ridge Regression` * :ref:`Linear Regression` * :ref:`Multi-output Regressor` Supervised ^^^^^^^^^^ Models for supervised learning, that require labeled data. * :ref:`Decision Tree Classifier` * :ref:`K-Nearest Neighbors Classifier` * :ref:`Logistic Regression` * :ref:`Multi-Layer Perceptron Classifier` * :ref:`Multi-Layer Perceptron Regressor` * :ref:`Multi-output Classifier` * :ref:`Random Forest Classifier` * :ref:`Support Vector Classifier` Unsupervised ^^^^^^^^^^^^ Models that work unsupervised, without labels on the data. * :ref:`Isolation Forest` * :ref:`K-means Clustering` * :ref:`Mini-batch K-means Clustering` * :ref:`One Class Support Vector Machines` Output ------ Export ^^^^^^ Export data from Sympathy to external sources. * :ref:`Export ADAFs` * :ref:`Export Bokeh Figures` * :ref:`Export Figures` * :ref:`Export Tables` * :ref:`Export Texts` Visual ------ In the **Visual** folder you find nodes for visualization of the data. Figure ^^^^^^ Nodes that create and work with the Figure datatype. These are the recommended nodes for creating plots from your data. * :ref:`Bokeh Components` Export html components for embedding a Bokeh figure. * :ref:`Bokeh Figure` Create a Bokeh Figure * :ref:`Colormap lookup` Map input into colors. * :ref:`Figure` Create a Figure. * :ref:`Figure Compressor` Combine several Figures into one, overlaying the plots. * :ref:`Figures` Create a list of Figures. * :ref:`Layout Bokeh Figures in Subplots` * :ref:`Layout Figures in Subplots` Create a subplot layout with several figures. Html ^^^^ * :ref:`HTML to Text` Convert html code to text. * :ref:`Html Report` Import HTML code from datasource. Plot ^^^^ * :ref:`Scatter 3D Table` Create a 3D plot, saved to disk. Report ^^^^^^ Under the **Report** tag are nodes for creating reports in PDF format. The reports can be customized to include plots and data from the Sympathy flow. These nodes are not actively maintained. Consider using either :ref:`Figure` or :ref:`Bokeh Figure` instead. * :ref:`Merge Reports` * :ref:`Report Apply ADAFs` * :ref:`Report Apply ADAFs with Datasources` * :ref:`Report Apply Tables` * :ref:`Report Template ADAFs` * :ref:`Report Template Tables` * :ref:`Select Report Pages`