.. This file is part of Sympathy for Data. .. .. Copyright (c) 2010-2017 System Engineering Software Society .. .. Sympathy for Data is free software: you can redistribute it and/or modify .. it under the terms of the GNU General Public License as published by .. the Free Software Foundation, either version 3 of the License, or .. (at your option) any later version. .. .. Sympathy for Data is distributed in the hope that it will be useful, .. but WITHOUT ANY WARRANTY; without even the implied warranty of .. MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the .. GNU General Public License for more details. .. You should have received a copy of the GNU General Public License .. along with Sympathy for Data. If not, see . What's new ========== News in 1.5.0 ------------- Sympathy for Data version 1.5.0 offers several improvements and new features, such as workflows in the library and routing points for connections. Nodes and flows from 1.3 and 1.4 should for the most part be compatible with 1.5.0. .. * Connections are rendered as lines. This can be changed to *Spline* in .. *Preferences -> General: connection shape* if you prefer the original look. Platform changes: ^^^^^^^^^^^^^^^^^ * Markdown_ is now supported in text fields. * Mark unavailable items in list views. * list views and combobox views can be edited when customized with editor attribute edit=True. This makes it possible to create selections that includes columns that do not exist in the input data or without executing previous nodes. * Shorter tab labels for flows, only showing parent flow names as necessary. * Only nodes that are in the current libraries can be used. Current libraries depend on the configured libraries for the current flow and the global selection from preferences. * Allow subflows to specify custom svg icons. * Route points for connections, see :ref:`route_points`. * 0 based indexing is used more consistently, that is, 0 is the index of the first element, etc. This behavior is more consistent with python code. * English/US locale is used regardless of the host language settings. * Generalized text-output port on all nodes. * Flows can be added to library, see :ref:`flows_in_library`. .. _Markdown: http://daringfireball.com/projects/markdown API changes: ^^^^^^^^^^^^ * Improved backwards compatibility of ParameterLists during execution. * ParameterLists return copies of its internal state disallowing direct mutation. * Ability to get and set dask arrays in adaf and table APIs. See example nodes: :ref:`Dask max example`, :ref:`Dask stack example` and :ref:`Dask tail example`. * Made getitem, setitem usable as a shorthand way of working with arrays columns in tables. For example table1['x'] = table2['y']. Node/plugin changes: ^^^^^^^^^^^^^^^^^^^^ * Improved masked arrays support in input data to :ref:`Select rows nodes` get normal numpy arrays instead of pandas Series. * :ref:`Select rows in Table with Table` understands literal comparison operators such as '==' or '>' in addition to their old text representations. * :ref:`HJoin Tables` gives consistent names when renaming duplicate columns * :ref:`Figure from Table` allows you to rotate bin labels in bar plots. * New parameter in :ref:`Select rows with Table(s)` * :ref:`Select rows nodes` * Faster execution of :ref:`Table` and :ref:`Select category in ADAFs` * Responsive preview for :ref:`Calculator List` and :ref:`Calculator` API changes ^^^^^^^^^^^ * Added MATLAB API for writing scripts executed by the Matlab node * Added update method to Attributes class. (ADAF API) * Added support for placeholder text in :ref:`lineedit_editor` in parameter helper * Added visibility and enable/disable slots to ParameterValueWidget Bug fixes ^^^^^^^^^ * Fixed name and type of output port of :ref:`Report Apply` nodes * Fixed a bug where save file dialog wouldn’t show up at all when trying to save subflow on Windows, if the subflow label contained some specific unallowed characters * Made sure that aborting a subflow doesn't also abort nodes outside of the subflow * Fixed a bug where linked subflows were sometimes inserted with absolute path Stability ^^^^^^^^^ * Improved reliability when working with lambdas, maps and apply nodes Deprecated nodes ^^^^^^^^^^^^^^^^ Deprecated nodes don't show up in the library view, but can still be used in workflows. * Type specific versions of list operation nodes (such as Get Item Table and Append ADAF). * Old FilterADAFs node News in 1.3 series ------------------ Sympathy for Data version 1.3.0 offers several new and prominent features such as generic types, higher order functions and much improved support for linked subflows. Many small improvements were made to the standard node library. Nodes will often cope better with empty input data and deliver informative, but less detailed, feedback. Nodes from 1.2.x should be compatible with 1.3.0 but there are new, more succinct, ways of writing nodes for 1.3.x that are not backwards compatible with 1.2.x. When writing new nodes, consider which older versions of the platform that will be used. New features ^^^^^^^^^^^^ * :ref:`Generic types` * :ref:`Higher order functions`: Lambda, Map and Apply * Official, and much improved, support for :ref:`linked subflows` * Official support for :ref:`locked_subflows` * New library structure using tags New nodes ^^^^^^^^^ * New generic versions of all list operations * :ref:`Ensure columns in Tables with Table` * :ref:`Conditional Propagate` * :ref:`Extract Lambdas` builtin node for reading lambda functions from existing workflows User interface ^^^^^^^^^^^^^^ * Right-click on an empty part of the flow to insert :ref:`higher order functions`. * New command in context menu for inserting a subflow as a :ref:`link`. * Improved file dialogs in node configurations, by using native dialog when asking for an existing directory and starting file dialogs from currently selected file path. API changes ^^^^^^^^^^^ * Simpler APIs for writing nodes. See :ref:`nodewriting` * New method in :ref:`adafapi`: ``Group.number_of_rows`` * Configuration widgets can expose a method called save_parameters which is called before the gui is closed. See :ref:`custom_gui` * Added API (parameter helper): List parameter widgets emit ``valueChanged`` signal * Improved slicing of (sy)table with slice object with negative or undefined stride * Automatically update order, label, and description for parameters when the node’s definition changes * :ref:`NodeContext ` is no longer a named tuple * Added new method: :meth:`NodeContext.manage_input`. A managed input will have its lifetime decided outside of the node Linked/locked subflows ^^^^^^^^^^^^^^^^^^^^^^ * Include subflows relative to path of parent flow, not relative to root flow. This affects where sympathy searches for linked subflows inside linked subflows and should hopefully feel more natural than the old system * Allow opening of flows with broken links * Import and export nodes can now be used inside locked subflows and lambdas * Made it impossible for flows below a locked flow to themselves be locked * Improved abort for locked subflows Node changes ^^^^^^^^^^^^ * :ref:`Report` framework: histogram2d graph layer is now called heatmap and can handle different reduction functions (count, mean, median, max, min). * Improved XLS(X) :ref:`import`/:ref:`export`. Especially handling of dates, times, and mixed types. Cells formatted as Time are now imported as timedeltas. * Renamed Sort Table(s) to :ref:`Sort rows in Table(s)` * :ref:`Calculator List`: chooses columns case-sensitively on Windows too. * :ref:`Calculator List`: shows number of output rows in preview in calculator gui. * :ref:`VSplit Table`: Removed constraint that the index should be sorted. The elements will be grouped by the first occurrence of each unique value. * :ref:`Convert columns in Table`: Added conversion path between datetime and float. * :ref:`Select columns in ADAF with structure Table` now works as expected when *Remove selected* has been checked. * :ref:`Select rows in Table with Table` offers a choice of reduction function between rows in config table. Previously it only read first row of the config table. * "Slice List of ADAFs/Tables": Basic integer indexing now works as expected. * Improve handling of one sample signals in :ref:`Interpolate ADAF(s) ` * :ref:`Report Apply nodes ` output datasources to created files * Improved :ref:`CSV import
`. Can now handle empty input, input with only one row, with or without trailing newline, and files with errors towards the end. It also features a new option for how to handle errors when importing a file. Header row has been made independent of the other input boxes, and no longer affects the data row. When read to end of file is selected, the number of footer rows is ignored. Delimiter detection was improved * Fixed issues with nesting of higher order functions (:ref:`Map `, :ref:`Lambda` and :ref:`Apply`) * Improvements to :ref:`reporting `: Improved bin placement and x-axis extent of 1d histograms. Automatically set axes labels from data source if they are empty. Added option "Lift pen when x decreases" to line graph layer. Added vline layer in reporting tool. * Several nodes are better at forwarding attributes, table names, etc. to output :ref:`Slice data Table`, :ref:`Select columns in ADAF(s) with structure Table(s)`, :ref:`Interpolate ADAF(s) `, and :ref:`Rename columns nodes ` * Many nodes are better at handling missing or incomplete input data: :ref:`Filter Rows in Table`, :ref:`Replace values in Tables`, :ref:`Detrend ADAF(s) `, :ref:`ADAF(s) to Table(s) `, :ref:`Select Report Pages`, :ref:`Scatter nodes `. * Added 'calculation' attribute on all output columns from :ref:`Calculator List` node * :ref:`Export Tables` and :ref:`Archive files` create missing folders * Fixed :ref:`Export Texts` Other improvements ^^^^^^^^^^^^^^^^^^ * Added :ref:`default workflow environment variables ` ``SY_FLOW_FILEPATH``, ``SY_FLOW_DIR`` and ``SY_FLOW_AUTHOR``. All flows have these and they can't be set or deleted. * Subflows can define :ref:`workflow variables `. Each subflow specializes the variables of its parent flow, so that the parent flows vars are accessible in the subflow but not vice versa. * Improve performance by skipping validation of any nodes that don’t implement :meth:`verify_parameters` * Improve performance by changing compression settings for sydata files, compression is faster but compresses slightly less * Pretty print workflow xml files, making diffs possible New requirements: * Requiring pandas version 0.15 for the CSV import, for versions before 0.15 down to 0.13 it will still work but may behave slightly differently in edge cases with blank rows News in 1.2 series ------------------ Sympathy for Data version 1.2 is a significant minor release for Sympathy for Data. It features several prominent new features, improved stability and more. It is however not redesigned and with only a few small modifications, all existing nodes and flows should work as well as in 1.1. The bundled python installation has been upgraded with new versions of almost every package. Added to the packages is scikit-learn, used for machine learning. Our investigations suggest that the new package versions are reasonably compatible with old nodes and cause no significant differences for the standard library. New features ^^^^^^^^^^^^ * Added support for using environment variables, and per installation/workflow variables. The variables which can have a default value are used in string fields of configuration widgets to enable parametrization. See :ref:`env_vars`. * Added support for profiling, with the ability to produce graphs if Graphviz is available. See :ref:`profiling`. * Added support for debugging single nodes with data available from Sympathy using spyder. See :ref:`spyder_debug`. * Added new Node Wizard for generating new nodes. See :ref:`node_wizard`. * Added support for configuring subflows by aggregating selected node configurations. See :ref:`subflow_config`. * Improved support for plugins in third party libraries. It is no longer necessary to add the folder with the plugin to python path in preferences * Support for adding custom data types in third party libraries. See :ref:`create_type`. * Significantly improved handling of unicode paths including the ability to install Sympathy and third party libraries in a path with unicode characters Nodes and plugins ^^^^^^^^^^^^^^^^^ * Added CarMaker type 2 ERG ADAF importer plugin called “CM-ERG” * Plugins can now export to non-ascii filenames * Fixed MDF export of boolean signals * Added generating nodes for empty Table, Tables ADAF and ADAFs. * Convert column nodes can convert to datetime * Calculator node can produce compact output for length matched output * Lookup nodes handles both event column and other columns with datetimes * Time Sync nodes “SynchronizeLSF” strategy should work as expected again. The Vjoin index option is now only used for the ”Sync parts” strategy New command line options ^^^^^^^^^^^^^^^^^^^^^^^^ See :ref:`start_options` for more info. * Added new command line option, '--generate_documentation' for generating documentation from CLI * Added 'exit_after_exception' argument which is activated by default in CLI. It makes Sympathy exit with error status if an unhandled exception occurs in a signal handler. * Added separate flag: --node_loglevel, for controlling the log output from nodes. * Made it possible to set the number of workers using --num_worker_processes n. API changes ^^^^^^^^^^^ * Libraries must now have only a single python package in their Common folders. See :ref:`nodewriting`. In the Standard Library this package is called sylib * Removed ``has_parameter_view`` from node interface. See :ref:`custom_gui`. * Changed default unit for time series to empty string instead of ``'unknown'``. * Added ``has_column`` method in sytable and added corresponding method in ``table.File`` * Accessing an ADAF basis which does not exist will raise a KeyError * Improved node error handling, making it possible for nodes to issue user friendly error messages as well as warnings. See :ref:`node_errors`. * Expanded and improved documentation, including API references for all default data types, and documentation on how to create your own data type * Improved error handling in many data type API functions User interface ^^^^^^^^^^^^^^ * Improved selection and context menu handling * "Help" in node context menus will now also build documentation if necessary. * Allow connections to be made by dragging from an input to an output port * Added zoom with Ctrl/Cmd + scroll wheel * Added working stop button. * Improved the presentation of data in the viewer with a clearer font and better size handling as well as coloring of columns by data type * Improved undo/redo functionality, making more operations available in the undo history Stability ^^^^^^^^^ * Avoid hanging on Windows when too much output is produced during startup * Avoid infinite wait during node validation