.. This file is part of Sympathy for Data. .. Copyright (c) 2010-2017 Combine Control Systems AB .. .. Sympathy for Data is free software: you can redistribute it and/or modify .. it under the terms of the GNU General Public License as published by .. the Free Software Foundation, version 3 of the License. .. .. Sympathy for Data is distributed in the hope that it will be useful, .. but WITHOUT ANY WARRANTY; without even the implied warranty of .. MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the .. GNU General Public License for more details. .. .. You should have received a copy of the GNU General Public License .. along with Sympathy for Data. If not, see . What's new ========== News in 4.1.1 ------------- Node changes ^^^^^^^^^^^^ * Fixed pasting datetimes in :ref:`org.sysess.sympathy.create.createtable`. * :ref:`org.sysess.sympathy.visualize.figure`: Warn about errors in python properties (will become errors in Sympathy for Data 5.0.0). * Fix broken preview when long running calculations are canceled in the calculator node. Enterprise Edition ^^^^^^^^^^^^^^^^^^ * Improved ability to recover licensing after problems (Sympathy Enterprise). News in 4.1.0 ------------- User interface ^^^^^^^^^^^^^^ * Viewer improvements: * Open viewers and configuration windows no longer cause problems for node execution. * Reload viewer data when nodes finish executing. * Show label to inform user of empty dict or list viewer. * Fix confusing feedback for automatic binning in histograms. * Fix table viewer forgets selected line plots when switching to another table, for example in viewer for tables. * Improved handling of messages from flows and viewers: * Implement go to node. * Fix automatic archiving. * Code editor improvements: * Fix confirm not working with usual Ctrl+Enter, inside editor. * Fix indentation on line with only whitespace before cursor. * Subflow improvements: * Show subflow icon as window icon for subflows. * Show node icons in subflow settings and configuration. Support was fixed for subflow settings, tabbed configuration and added to wizard configuration. * Fix confirm (via Commit/OK) can cause changes to other subflow nodes resulting from automatic migrations. * Improve printing of exceptions from user code in :ref:`org.sysess.sympathy.data.fx`. * Data Binding GUI can be hidden via right-click context menu when `Advanced development options` are enabled. See :ref:`configuration_port_data_binding` for more info. * Make Library selection the first page of Node Wizard. Node/plugin changes ^^^^^^^^^^^^^^^^^^^ * Fix ADAF importer plugin for TDMS files. API changes ^^^^^^^^^^^ * Table API: * Consider all values when inferring type of object type array or series. Enterprise Edition ^^^^^^^^^^^^^^^^^^ * Improved MongoDB format support (Sympathy Enterprise): * New node: *Create MongoDB JSON*, used to in create MongoDB compatible JSON. * New constructor functions can be used to conveniently create data of different MongoDB types, in the configurations of MongoDB nodes, instead of using the corresponding Python literal encoding. For example, `ObjectId` and `ISODate`. News in 4.0.1 ------------- Bugfixes ^^^^^^^^ * Fix case where combo box editor for string parameters would insert None in place of an empty string value. * Fix nodes in locked flow continue executing even if they depend on other nodes that have failed. * Fix missing attribute which could prevent node wizard from showing. * Fix formatting of user traceback in F(x) nodes. * Subflow configuration: * Fix race condition which resulted in wizard configuration page showing before its input data was ready and therefore displaying old or empty data. * Fix page ordering for locked subflows in wizard configuration. * Show traceback information for exceptions that occur when building selected widgets (and their node contexts) as warning output. * Migrations: * Fix unable to add configuration ports on nodes after migration to a different node. * Fix migrating a node which is removed from library discards node label. News in 4.0.0 ------------- User interface ^^^^^^^^^^^^^^ * Improved high resolution handling and scaling behavior. * Trace level error details (tracebacks) are no longer shown by default, they can be toggled by clicking the link in the bottom of the message. * Limit number of characters for streamed messages in Messages window. * Grid align inserted elements. * Internal code editor now supports indentation and has been optimized to handle more data. * Drop of syx-file will either open the flow or insert it as a subflow depending on the location (tab bar or flow view). Dashed border indicator shows the current location. * Multiple subflow links can be inserted from context menu by selecting multiple files or by drag and drop in the flow view. * Improved view for environment variables distinguishes reserved, local, local override and inherited (from parent) variables. * Substitude node counts on subflows with a new status icon for partially executed subflow. * Allow force migrating past migrations that will change the node's behavior. * Migration icon is only shown if the node _must_ be migrated before it can be executed. In this case it will have a red migration icon. The green and yellow migration icons are no longer used. * Nodes that need to be manually migrated before being able to run, will now be in a permanent error state until migrated. See :ref:`Migrations ` for more info. Enterprise Edition ^^^^^^^^^^^^^^^^^^ * Sympathy Graph 365 client is added to the Azure toolkit (Sympathy Enterprise) and offers the ability to work with files in OneDrive or SharePoint on Microsoft 365. * Sympathy Graph 365 nodes included in Azure toolkit (Sympathy Enterprise): * Check in File * Check out File * Create File Link * Create Folder * Delete File * Delete Folder * Download File * Get OneDrive Drive * Get Drive Root Folder * Get Custom Resource * Graph 365 * List Drive Items in Folder * Search for Drive Items in Folder * Search for Drive Items * Select Item from Response List * Select Drive Item in Folder * Select SharePoint Drive * Select SharePoint Site * Upload File * Graph Data to JSON * Graph Data from JSON * Floating licenses with support for long time checkout (Sympathy Enterprise). * Improved license status feedback and recovery (Sympathy Enterprise). License checks that fail due to internet connection issues can be recovered. Message feedback is shown in both cases. Node/plugin changes ^^^^^^^^^^^^^^^^^^^ * Select columns in Table with Regex allows search pattern (new default) instead of full match. * Bug fixes: * Fix XLSX export failing due to duplicate defined names. Duplicate names will be enumerated: X, X2, X3. * Fix CSV import failing to import large data that starts with numbers and eventually contains strings. * Fix Cartesian product Table produce incorrect result with duplicate column names. * Remove six from available modules, in calculator, etc. * Rename sort order options from Ascending and Descending to Standard and Reverse. * F(x) nodes can use internal editor for editing external files. * Failing Extract node, shows user friendly message for known problems. * Calculator plugins are no longer allowed to use the ${} syntax in gui_dict(). Use simple placeholder variable names instead. New nodes/plugins ^^^^^^^^^^^^^^^^^ * :ref:`Parquet Table Importer ` * :ref:`org.sysess.sympathy.environment.get` * Bokeh Figure nodes: These are new, alternative Figure nodes that are built using `bokeh `_ (new port type) to create interactive, web enabled, plots. * :ref:`org.sysess.sympathy.visualize.bokehfigure` * :ref:`org.sysess.sympathy.visualize.bokehfiguresubplot` * :ref:`org.sysess.sympathy.export.exportbokehfigures` Deprecated nodes/plugins ^^^^^^^^^^^^^^^^^^^^^^^^^ * GeoJSON Removed nodes/plugins ^^^^^^^^^^^^^^^^^^^^^ * Figure(s) (depreacted) * Figure from Table with Table * Export Figures with Datasources * Calculator Table(s) * Jinja2 template (deprecated) (New migration to :ref:`org.sysess.sympathy.texts.generic_jinja2template` added) * Convert specific columns in Table(s) (New migration to :ref:`org.sysess.sympathy.data.table.convertcolumnstables` added) * TimeSync ADAF(s) * HJoin ADAF(s), HJoin ADAFs pairwise * VSplit ADAF * Select rows in ADAFs * ADAF(s) import plugin: ATF * ADAF(s) import plugin: ATFX Platform changes ^^^^^^^^^^^^^^^^ * Json configuration binding can specify dictionary element holding the data root. This makes it possible to apply different binding configuration for different nodes with the same json input. * Disable completion for datasource parameters to avoid delays and rendering problems. * Fix parameter integer change signals support for large values. * Support for creating empty Lambdas, which produce empty output. They can be used in error handling. * Automatic line wrapping of tooltips for parameter views. * Pre-calculate node plugins to avoid unnecessary file imports of unrelated plugins. * Close configuration dialogs and viewers when the nodes are removed. * Treat unavailable internal file as empty to avoid exceptions from appearing all over. API changes ^^^^^^^^^^^ * Table API: * Add `to_array` and `from_array`, for converting between table and 2D numpy array. * Deprecated `to_matrix` and `from_matrix`, since numpy intends to deprecate matrix. https://numpy.org/doc/stable/user/numpy-for-matlab-users.html#array-or-matrix-which-should-i-use. Use `to_array`, `from_array` instead. * Remove `input_index` argument from `vjoin`. * Improve behavior of `set_column_from_series` with object type data: * Consider all values when determining type for more consistent type conversion. * Deprecate conversion with mixed types involving strings * Deprecate conversion with mixed types involving datetime or timedelta * Add support for delete named columns using `del`. * Fix `vjoin` of non-empty table. * ADAF API: * Remove `File.vjoin`, `File.hjoin`, `SystemGroupContainer.hjoin`, `TimeseriesGroup.hjoin` and `File.vsplit`. * Remove `input_index` argument from `vjoin`. * Parameter API: * Remove default float editor precision limit * Removed deprecated ParameterEntity.parameter_dict. Nodes should rely on The ParameterEntity API to interact with parameters rather than their dictionary representation. * Add ParameterEntity.to_dict which returns a new dictionary, without shared internal state with the ParameterEntity. * Expose ParameterList.multiselect_mode to users. ParameterList._multiselect_mode has been deprecated. * Data API (Table API, ADAF API, etc.): * Made filename an explicit argument of constructors, first keyword argument. * Remove six package from use in python code written in node configurations, like the Calculator API. * Deprecated: * sympathy.api.qt * Removed deprecated: * node_helper.table_node_factory * node_helper.tables_node_factory * node_helper.adafs_node_factory * node_helper.TableOperation Python versions ^^^^^^^^^^^^^^^ * Support for Python 3.9. * Draft support for Python 3.10. * Removed support for Python 3.7 and older. We now target 3.9. * Switch to PySide 6 (Qt for Python). * Dropped compatibility with older python package versions. News in 3.2.0 ------------- Features ^^^^^^^^ * Support for logging in with :ref:`Azure credentials ` with an external web browser. This can be helpful for installations where specific browsers are required. * Support for :ref:`passing credentials via environment variables `. User interface ^^^^^^^^^^^^^^ * Improved feedback after problems: * Show dialog with traceback when problem occurs during launch. * Show message when switching to a workflow without usable folder. * Note when library creator loads library with unsatisfied requirements. * Show resample (downsampling) for histogram plots in the table viewer. * New item to show privacy notice from the help menu. Platform changes ^^^^^^^^^^^^^^^^ * Text port type: * Can be linked to avoid duplication. * Supports data sizes of 2 GiB or larger. * Locked subflows: * Locked subflows are shown as executing until fully done, including write of output data. * Ensure locked subflows are not nested. * Use OpenGLES on Windows to avoid problems with the internal web browser on installations without support for OpenGL. * Documentation: * Debugging instructions for :ref:`Visual Studio Code `. * Documentation for :ref:`Encodings `. * Bug fixes: * Fix marker choice ignored for line plots in the table viewer when x-axis data is in datetime format. * Fix problem using table config port with nested parameter groups. * Fix error message for parameters containing non-json serializable data. Enterprise Edition ^^^^^^^^^^^^^^^^^^ * Proxy handling for HTTP(s) requests (Sympathy Enterprise). * Improved license status (Sympathy Enterprise): * Preferences shows more details about the license. * License info shown in toolbar and on the splash screen. * User oriented feedback message after license problems. * Experimental Deep Learning nodes `Tabular Classifier`, `Tabular Regressor`, `Image Classifier` in Advanced Machine Learning toolkit (Sympathy Enterprise) now support: * Cross-validation * Early stopping * More feedback and increased consistency for empty inputs and other errors in toolkit nodes (Sympathy Enterprise). Node/plugin changes ^^^^^^^^^^^^^^^^^^^ * Fix color cycle choice ignored in Figure. * MDF3 Exporter: * Choose encoding from list of options. * Export encoding as code page number. * Exported version changed from 3.00 to 3.30 (includes code page number). * MDF3 Importer: * Choose encoding from list of options. * Option to use code page number stored in imported file. * System name is configurable. * Html Report: * Remove tempfile option. * Improved generation of standalone HTML on Windows. * Several nodes show user oriented feedback instead of generic error messages. API changes ^^^^^^^^^^^ * New helper API: `sympathy.api.masked` contains utilities for handling masked arrays. News in 3.1.0 ------------- Major features ^^^^^^^^^^^^^^ * Support for :ref:`Azure credentials ` for connecting to different Azure services by logging in through a web browser, this requires a client implemented in Sympathy: * Sympathy PowerBi client is included in new Azure toolkit (Sympathy Enterprise). Experimental feature ^^^^^^^^^^^^^^^^^^^^ * Support for lazy-loaded datasets and deep learning models (Sympathy Enterprise) Deprecated Nodes/Plugins ^^^^^^^^^^^^^^^^^^^^^^^^ * Slice Data Table(s) (deprecated) * Slice List (deprecated) New nodes ^^^^^^^^^ * More specific nodes which replace the deprecated Slice nodes: * :ref:`org.sysess.sympathy.index.columns.table` * :ref:`org.sysess.sympathy.index.rows.table` * :ref:`org.sysess.sympathy.list.index` * :ref:`org.sysess.sympathy.reverse.columns.table` * :ref:`org.sysess.sympathy.reverse.rows.table` * :ref:`org.sysess.sympathy.list.reverse` * :ref:`org.sysess.sympathy.slice.columns.table` * :ref:`org.sysess.sympathy.slice.rows.table` * :ref:`org.sysess.sympathy.slice.list` * PowerBi nodes included in Azure toolkit (Sympathy Enterprise): * Create Dataset * Create Workspace * Delete active Dataset * Delete active Workspace * Get Datasets as Table * Get Workspaces as Table * PowerBi * Push rows to Dataset Table * Set active Dataset * Set active Workspace * Dataset and Deep Learning nodes included in Advanced Machine Learning toolkit (Sympathy Enterprise): * Table Dataset * Image Dataset * Transform Table Dataset * Transform Image Dataset * Convert column type in Table Dataset * Split Dataset * Fit Dataset * Predict Dataset * Tabular Classifier * Tabular Regressor * Image Classifier * Dataset to tables * Tables to dataset * Dataset to dict Node changes ^^^^^^^^^^^^ * JSON nodes can operate on empty string keys: * :ref:`org.sysess.sympathy.data.json.selectkeyjson` * :ref:`org.sysess.sympathy.data.json.removekeyjson` * :ref:`org.sysess.sympathy.data.json.splitonkeyjson` * `Table(s)` imported with :ref:`SQL ` plugin gives more feedback after errors. Platform changes ^^^^^^^^^^^^^^^^ * Table binding is available for all nodes that have parameters, like JSON binding. It was previously unavailable for nodes with custom GUIs, like the calculator, etc. * Faster view for multi-select list parameters, especially when showing large numbers of items. Python versions ^^^^^^^^^^^^^^^ * Draft support for Python 3.9. Bugfixes ^^^^^^^^ * Tooltips for broken links and missing nodes mention the missing status instead of showing the generic tooltip for error. * Fix problem preventing startup on Mac OS Big Sur. API changes: ^^^^^^^^^^^^ * Deprecated `ParameterEntity.parameter_dict`. Use the parameter's normal API instead of its dict. * Table: `set_column_from_series` and `from_dataframe` supports object type series so long as the values are of a single supported type or markers for missing data. * Table: `to_dataframe` supports masked arrays by converting masked values to markers for missing data. * Combobox editor can have placeholder text. News in 3.0.1 ------------- Bugfixes ^^^^^^^^ * Improve Calculator Table migration's ability to decide if a column comes from ``arg`` or ``res`` in some simple cases. * Fix a bug where configuring a node with migrations would run yellow migrations which are intended to require manually triggering them. * Fix a bug where the properties dialog for nodes with overrides couldn't be opened. * Fix a situation where Sympathy would think that a flow had changes that needed saving, when there were no such changes. * Migrations implementation is more robust in case of missing nodes or corrupt parameters. * Fix preview not auto-updating after parameter change. * Fix bugs handling parameter visibility using controller. Labels were not affected and show could result in a floating window. * Fix exception refreshing viewer. * Clearer error when creating sydict with empty key. * Fix case where configuration of locked subflows using wizard could wait forever for previous nodes to become done. News in 3.0.0 ------------- Major features ^^^^^^^^^^^^^^ * :ref:`Credentials `: * Support for storing credentials. Used when requested by nodes. * Makes it more convenient to share workflows that access databases, etc. without also sharing your passwords. * Usable in :ref:`org.sysess.sympathy.files.downloadfile`, `Table(s)` by using :ref:`plugin_tabledataimporterbase.dataimportsql` and `Export Table(s)` by using :ref:`plugin_tabledataexporterbase.dataexportsql`. * Also usable in importer nodes (`Table(s)`, `ADAF(s)`, `Text(s)`) when input is an URL to download. * :ref:`Migrations ` system that lets you migrate old and deprecated nodes in your flow into new ones with a simple GUI action. This will automatically update any old configuration on the node. * A node can be migrated if it has a green/yellow icon above its top-right corner. * Migrate a node by right-clicking on it and choosing *Refactor* -> *Migrate node*. * Sympathy 3.0.0 includes migrations for updating the old `Calculator Table` node, the old `Figure` node, and the old `Export Figures with Datasources` node to more modern equivalents. * Currently, third-party nodes can not have migrations. User interface ^^^^^^^^^^^^^^ * Improved :ref:`messages view `: * Only show messages from the current flow. * Each severity level can be shown/hidden independently. * When clearing the messages view, messages are archived instead of deleted. To view an older message, just click the "Show archived" icon in side toolbar. * Node messages are automatically archived when the node is reloaded. * The number of archived messages that are retained before being deleted can be changed under *Preferences* -> *General* -> *Max archived messages*. * New, convenient, user interface for :ref:`auto-connecting ` node ports: * Hold *Shift* when dragging a node to get a suggested connection between the closest valid ports. Release mouse to confirm and create new connection! Pressing *c* also confirms without releasing the mouse, and is useful to connect multiple ports. * Project colors in flow tabs. When working on two or more different projects at the same time Sympathy will now color all the flow tabs to indicate which flows belong to the same project. * Reported issues are sent to our new service desk. Reporting issues now requires that you enter your email address. This gives us a chance to get back to you with status updates and questions about your issue. Please feel encouraged to report issues if you notice something wrong or something to be improved. * Highlighting flow elements under the cursor for better visual feedback and to help selection. * Improvements for configuration GUIs: * Better alignment of labels and their corresponding values. * Remove unnecessary groups inside of tabs for cleaner interfaces. * Some table views with fixed headers expand to full width. * Updates to Preferences: * New sections for User, System and Privacy -information. * :ref:`Python section ` lists and allows installation of new packages using pip. * Present profiling results even if any of the profiled nodes encountered errors. * Add grid toggle to view menu with a keyboard shortcut. * Nodes retains their state in some cases where they would previously loose executed state. This saves you from unnecessary re-execution. Platform changes ^^^^^^^^^^^^^^^^ * Overrides are always enabled on linked subflows. * More granular logging. Logging is configured via the cli flag --loglevel/-L which can now take two arguments: a logger name and a level and the flag can be repeated allowing for specialized logging configurations. * Better performance in large flows: * Avoid crashes when too many nodes change their state. * Faster state updates for operations that affect many nodes. * Cycle detection avoids creation of cycles in the flow. * Encrypted communication between workers and platform. * New connection parameter type which can use credentials to safely configure a connection to databases, webservers, etc. * Removed support for setting min_version for flows. * Documentation: * Updated instructions for debugging using PyCharm Node changes ^^^^^^^^^^^^ * Add sum operation to Descriptive Statistics. * Explicit handling of missing sheet in XLSX importer. * Renamed Manually Create Table to Create Table. * Renamed Manually Create JSON to Create JSON. * Unique table handles more data types * Credentials support: * Datasource * Import nodes * Download URL * SQL importer and exporter * MongoDB nodes (Sympathy Enterprise) Bug fixes ^^^^^^^^^ * :ref:`Figure`: Bar label font now correctly applies to histogram bar labels. * Fix event :ref:`lookup ` incorrectly matching rows if *Perfect match* was disabled and there were two or more rows before the first entry in the lookup table. * Make sure that some machine learning nodes don't give warnings with newer versions of scikit-learn. * Fix: Sympathy cannot start if session path is unavailable * Fix: MDF3 importer got stuck processing some corrupt input files New installer ^^^^^^^^^^^^^ * MSI installer * Installs for the current user. MSIEXEC /I ALLUSERS=1 can be used to install for all users. Python versions ^^^^^^^^^^^^^^^ * Support for Python 3.8. * Compatible with numpy >= 0.18.0. Examples ^^^^^^^^ New example for Outlier Removal (Sympathy Enterprise) Deprecated APIs ^^^^^^^^^^^^^^^ * adaf.File.vjoin, adaf.File.hjoin, adaf.File.vsplit * input_index argument to table.File.vjoin * sylib.synchronize * sylib.time_synchronize_gui Deprecated Nodes/Plugins ^^^^^^^^^^^^^^^^^^^^^^^^ * Figure from Table with Table * VJoin ADAF(s), VJoin ADAFs pairwise * VSplit ADAF * HJoin ADAF(s), HJoin ADAFs pairwise * TimeSync ADAF(s) * Select rows in ADAFs * ADAF(s) import plugin: ATF * ADAF(s) import plugin: ATFX News in 2.2.0 ------------- Bug fixes ^^^^^^^^^ * Exported ADAF files incorrectly expanded the type declaration, causing problems in stand-alone viewer. * Restored ability to select operator in Filter rows in Table. * Tables, ADAFs: fixed Create Empty Entry, action on import failure. * Removed limitation which prevented loading large syx-files. * MDF importer: fixed case where boolean signals would incorrectly become True when the signal byte contained uninitialized data. * Table(s): Limited problems with the MDB/Access database driver, introduced in Office (Build 13127.20296), from affecting import of other types of data. * Fixed issue which could cause Sympathy to exit after 5 minutes on Windows due to an access violation. * Fixed some unintended behaviors of optional subflow ports, that would result in subflow lacking external ports when linked. Node changes ^^^^^^^^^^^^ * Figure: added option to invert axes. * Added ability to control error handling to Parallel Map subflows. API changes: ^^^^^^^^^^^^ * Controller trigger based on port existing or not, see :ref:`controllers`. MongoDB nodes ^^^^^^^^^^^^^ * MongoDB, Set Collection, Find, Insert, Update, Replace, Bulk write. (Sympathy Enterprise) News in 2.1.1 ------------- Bug fixes ^^^^^^^^^ * Changes not saved when configuring linked subflow in some situations. News in 2.1.0 ------------- User interface ^^^^^^^^^^^^^^ * New option to automatically save all open flows. Can be enabled at *Preferences* -> *General* -> *Autosave flows*. * Backup and restore: * If Sympathy crashes, you will be able to recover the last flow changes! * The :ref:`node_config_message_area` (used in Calculators and FilterADAFs) has been improved and is globally configurable: * Configured in *Preferences* -> *Node Configuration* -> *Node Configuration* to be shown, hidden or set to show after first message. * Uses fixed height and shows only one line of text. Viewers ^^^^^^^ * ADAF viewer: Added search for signals in Timeseries view. * Table viewer: * Added configurable, interactive cursors. * Ability to export plotted data or the data between two cursors as a csv file. * Improved icons and tweaked toolbars. New nodes ^^^^^^^^^ * :ref:`org.sysess.sympathy.adaf.renamecolumntable` * :ref:`org.sysess.sympathy.data.adaf.assertequaladaf` Node changes ^^^^^^^^^^^^ * Figure wizards: * Added support for: Histograms, heatmaps, images, and timelines. * Wizard selection shows up when configuring an empty figure node. * Improved DPI handling in Export Figures: * DPI can now only be set when using a rasterized output format and a physical image size. * All old nodes should still produce the same output as before. * Improved Calculator: * Undo/Redo! * More compact layout showing calculations on a single line to require fewer manual adjustments. * Calculations are available as *Results* under Signals and Common functions to make them easier to discover and use in other calculations. * Informative feedback when using forbidden = character in calculation name. * Added option to copy input to :ref:`Lookup Table`. Platform changes ^^^^^^^^^^^^^^^^ * Configuration port for subflows. When enabled, it is implicitly connected to all nodes selected in the subflow's settings. * Added internal index to adaf to speed up computation of names. * Optimized link handling to avoid some unnecessary overhead when propagating input structures to output. Added APIs ^^^^^^^^^^^ * equal_to for table and adaf. Python versions ^^^^^^^^^^^^^^^ * Draft support for Python 3.8. Windows Installer ^^^^^^^^^^^^^^^^^ * Installers bundle the content from Visual C++ 2019 Redistributable instead of the executable installer itself to avoid requiring elevated privileges when installing. Enterprise Edition ^^^^^^^^^^^^^^^^^^ * Moved storage directory under the user folder. News in 2.0.0 ------------- New version scheme that will follow semantic versioning. The file structure has been modernized to allow more seamless use and distribution as a python package. License change ^^^^^^^^^^^^^^ * Using the same license for all sources. License details can be found in the *About Sympathy* window. Installer ^^^^^^^^^ * Install as user by default. * Replaced Sysess directory with Combine. * Replaced Sysess registry key with Combine. Platform changes ^^^^^^^^^^^^^^^^ * Optional subflow ports. See :ref:`subflow_ports`. * Support for Spyder, is limited to installations containing spyder and where the editor plugin is installed. * Documentation: * Our installers now comes with documentation prebuilt * Documentation never starts building automatically * Restructured the outline and shorted many headers to provide better overview. * Library documentation organized by tags. * Better separation between generated documentation for standard platform and third-party libraries. * Plugin base classes and implementations can specify a name used in documentation generation instead of the class name. * Files were restructured: * Top-level package sympathy_app no longer exists. It has been renamed sympathy. * Sources are structured as a setuptools project using setup.py. * Packages sympathy and sylib can be installed via setup.py. These can then be imported without requiring sys.path manipulations. * Tests and other files that are not relevant for end users are excluded from installation. * New library format which is built as a python package. See :ref:`new_library`. These can be defined and installed as setuptools entry points. * Added new option to remove unused port file when nodes are removed in flows or when flows are closed. On by default. * Removed some dependencies that were only required for development of Sympathy itself and completely optional (unused by Sympathy) otherwise. * Settings moved from Sysess/Sympathy_*.ini to combine/Sympathy.ini. Values are copied from the old location when the new settings file is created. * Library view changes * Disk Layout and FlatTag Layout are no longer available * New option to show all hidden nodes * Fixed bug causing inability to save lock status in flows. * Fixed bug causing changes to environment variables to be lost on save. * Fixed bug causing problems when printing unicode characters in the platform itself on Windows. * Added USERPROFILE to worker environment to fix os.path.expanduser on Windows. * Cleanup of compat code for Python 2: __future__ imports and uses of six. User interface ^^^^^^^^^^^^^^ * Documentation: * Online documentation is used if local documentation is unavailable. * Generation is no longer triggered automatically when documentation is missing. * Help menu button to generate documentation has been removed since it was no longer useful. Library developers should use the command-line interface to generate documentation and ensure to make it available to users. * Examples from third-party libraries can be opened from the help menu. * Command line options: * Removed underscores from the command line interface. * --version is now a top level option. -v is no longer used for printing the version. * Top level commands used to run external tools in the Sympathy environment are no longer required and have been removed. That is, spyder, nosetests, coverage, pyflakes, pylint and ipython. These can simply be installed in the same python environment as Sympathy and then run directly. * New top-level command *doc*, for generating documentation. * Improved performance when pasting many elements in the Flow Window. * Library wizard creates new-style libraries by default. * Library and Flow overview highlight gives more predictable feedback. New nodes ^^^^^^^^^ * :ref:`org.sysess.sympathy.getcolumnnamesintable` * :ref:`org.sysess.sympathy.list.grouplist` Node/plugin changes: ^^^^^^^^^^^^^^^^^^^^ * xlsx import plugin has new option to import all sheets. * xlsx export plugin has new option to replace existing sheets which can be used to insert raw data into a template file. * Zip/Tar extractor: fixed issues with output structure. Examples ^^^^^^^^ * Added example for updating sheets in xlsx file. * Updated examples for machine learning. * Updated calculator example flows. * Replaced deprecated nodes in example flows. Removed nodes ^^^^^^^^^^^^^^ * Plot Table(s) * Copy files with Table * Rename file(s) * Rename file with Table * Matlab Calculator * Export Figures with Datasources Removed plugins ^^^^^^^^^^^^^^^ * XLS importer (only legacy Excel format, xlsx importer still works as before) * XLS exporter (only legacy Excel format, xlsx exporter still works as before) * DIVA importer * LAA importer * Matlab Calculator plugin Removed APIs ^^^^^^^^^^^^ * ManagedNode * node_helper.list_node_factory Added APIs ^^^^^^^^^^^ * sympathy.api.library, for new-style libraries. * sympathy.api.component, for adding plugins to nodes. Changed nodes ^^^^^^^^^^^^^ * Improved labels in many machine learning nodes. * :ref:`org.sysess.sympathy.visualize.figuresubplot` can now take global colorbar and global legend from a configurable parent figure instead of always taking it from the last figure in the list. * Filter List with Table: * Return empty output instead of full output when filter lacks rows. * Fail instead of using first column when filter column is missing Removed Node parameters ^^^^^^^^^^^^^^^^^^^^^^^ * Select rows in Table(s): * Use compatible string constraints * Select rows in ADAFs: * Use compatible string constraints * Select rows in Tables with Table * Evaluate values as code Changed APIs ^^^^^^^^^^^^ * The different fields of a list parameters (`list`, `value` and `value_names`) are now always required to match. Any old list parameters where this is not the case are automatically updated from `value_names`. For list parameters that use the standard editors, this should not be a problem. For some custom parameter guis might need to be changed to adhere to this rule. Deprecated APIs ^^^^^^^^^^^^^^^ * node_helper.table_node_factory * node_helper.tables_node_factory * node_helper.adafs_node_factory * node_helper.TableOperation Python versions ^^^^^^^^^^^^^^^ * Removed support for Python 3.6 and older. We now target 3.7+. * Compatibility with bokeh >= 1.3. * Dropped compatibility with matplotlib < 2.2.0. * Dropped compatibility with scikit-image < 0.14.3. * Dropped compatibility with scikit-learn < 0.20.4. * Currently not compatible with numpy >= 0.18.0. * Compatibility with openpyxl >= 3.0.0, replaces XlsxWriter. News in 1.6.2 ------------- License change ^^^^^^^^^^^^^^ * Changed copyright from SysESS to Combine. Platform changes ^^^^^^^^^^^^^^^^ * Instant warning and notice messages from nodes. * Issue reporting, see :ref:`issues`. * Collection of anonymous user statistics. See :ref:`privacynotice` for more details about privacy and data collection in Sympathy. * Fix issues with the histogram plot in table viewer. * Sympathy can now generate file associations and desktop entries on Linux. * Support for spyder, if installed in the same python environment. * Improved performance when working with large number of files. * New options to clean up temporary files when closing flows and the application. * Configuration ports can have any types. * Built in support for creating parameter bindings to configuration ports with table data. * Ability to override temp location using environment variables. See :ref:`env_vars`. User interface ^^^^^^^^^^^^^^ * You can now close flows by clicking the middle mouse button on the respective tab. * Wizard configuration can now be maximized. * Changed menus for creating and removing ports. * New text editor with search support, used for json and text viewers. * Restructured command line arguments: * Help texts should be more consistent. * New options for generating documentation. * Improved feedback when: * Library package fails to load. * Some types of corrupt flows are loaded. * New context menu option to create library node from subflow. Node changes ^^^^^^^^^^^^ * Using combo-boxes for choice parameters with single selection. * New simplified wizards for some plot types in figure node. * Improved the ability of :ref:`org.sysess.sympathy.visualize.figurecompressorgui` and :ref:`org.sysess.sympathy.visualize.figuresubplot` to reproduce their input figures: * Improved getting labels from legends. * Correctly reproduce custom tick labels. * Support for reproducing box plots, rectangles, ellipses, and error bars. * Support for reproducing QuadMesh artist (from matplotlib's pcolormesh function). * Improved error messages when some features could not be reproduced. * New option for sharing axes in :ref:`org.sysess.sympathy.visualize.figuresubplot`. * :ref:`org.sysess.sympathy.visualize.figure` features: * Add edge color and line width properties to scatter plot. * Fix edit issues on MacOS. * Hiding the labels of a pie chart no longer hides its percentages. * New completions interface. * Fixes in :ref:`plugin_tabledataimporterbase.dataimportcsv`: * Add support for a few more obscure line endings (e.g. \r). * Fix regression in handling of incomplete csv-files. * :ref:`org.sysess.sympathy.data.table.calculatorgeneric`: * Improved performance of local_min/local_max. * Fix a problem with masked values not being treated correctly in local_min/local_max. * Removed unnecessary warnings. * Empty calculations are now allowed and do nothing. * :ref:`org.sysess.sympathy.datasources.filedatasource`: Paths can be relative to library directory. * :ref:`org.sysess.sympathy.create.createtable`: * Can now paste multiple values. * Always shows float values in C locale. * :ref:`org.sysess.sympathy.machinelearning.export`: added datasource output. * :ref:`org.sysess.sympathy.datasources.filedatasourcemultiple`: can search for files, directories or both. * :ref:`org.sysess.sympathy.data.table.dropnantable`: able to choose columns. * :ref:`org.sysess.sympathy.data.table.assertequaltable`: can add more ports. * :ref:`org.sysess.sympathy.filters.columnfilternode`: standardized configuration GUI. * :ref:`org.sysess.sympathy.data.table.holdvaluetable`: able to choose columns. * :ref:`org.sysess.sympathy.create.configureparameters`: list choices and editors updated from data. New nodes ^^^^^^^^^ * :ref:`org.sysess.sympathy.texts.splittext` * :ref:`org.sysess.sympathy.data.table.unpivottablenode` and list version. * :ref:`org.sysess.sympathy.setcolumnnamesintablewithtable` and list version. * List versions of several existing nodes: * :ref:`org.sysess.sympathy.json.jsonstolists` * :ref:`org.sysess.sympathy.json.jsonstodicts` * :ref:`org.sysess.sympathy.data.text.tables2texts` * :ref:`org.sysess.sympathy.convert.textstojsons` * :ref:`org.sysess.sympathy.texts.concatenatetextslist` Documentation ^^^^^^^^^^^^^ * Documentation is built as part of the installation process. This means that documenatation should usually be available directly after installation. * Third-party library documentations can be built separately. * Documentation is now by default built for only the platform and the standard library. * Documentation pages are generated for library subflows. The subflows documentation can be edited in the subflow properties dialog. * Building the documentation places it in /Docs/ by default. * Node documentation pages now start with the node's description. * A warning is automatically added if a node is deprecated. * Nodes can now have a list of related nodes, which shows up on the node's documentation page. * A node's docstring is now used as description if the description field is missing or empty. * Basic node documentation for nodes with optional ports. * Updated documentation for available command line arguments. API changes ^^^^^^^^^^^ * More guards against using bad arguments in table and adaf. For example, use of reserved names. Recarrays in place of numpy arrays, etc. * Datetime parameter can set editor display format to show only date and allows fractional second precision. * Expanded names API with another parameter: fields. * Added col_paths as a new kind of names field for accessing data. * Added new completions API. Python versions ^^^^^^^^^^^^^^^ * Compatibility with Python 3.7.3 in virtualenv. * Compatibility with h5py >=2.10 and dask>=2. * Compatibility with scikit-learn >= 0.19.2. * Compatibility with networkx >= 2.4. News in 1.6.1 ------------- Release 1.6.1 is a bug-fix release addressing several bugs present in 1.6.0. Platform changes ^^^^^^^^^^^^^^^^ * Write value and list to the syx file again for ParameterList. This change will have to wait until we have rooted out all nodes that use .value or .list of a list parameter. * Made it possible to open workflow with reference to missing library. * Made it possible to open tabbed subflow configurations containing failed configurations. Node changes ^^^^^^^^^^^^ * Changes in :ref:`org.sysess.sympathy.visualize.figure`: * Add property 'Distance to Axes' to legend. Use this to tweak the position of the legend when placed outside of the Axes. If the legend is placed inside the Axes this property is ignored. * Avoid rare exception when editing some properties. * Improved default values and editors for Annotations. * Changes in :ref:`org.sysess.sympathy.visualize.figurecompressorgui` and :ref:`org.sysess.sympathy.visualize.figuresubplot`: * Fixed a bug when copying axes with unsupported elements. Those elements will now simply be ignored. * Fix colorbar handling. * Fixed python 3 incompatibility in :ref:`org.sysess.sympathy.machinelearning.export`. * Fixed inability to configure and execute Calculator Table(s) if comment contained newlines. * Introduced compatibility option for :ref:`Select rows in Table(s)`. * Fixed issues with configuration related to missing list and value in: * :ref:`org.sysess.sympathy.data.table.indextable` * :ref:`org.sysess.sympathy.data.adaf.adaf2table` * :ref:`org.sysess.sympathy.list.filterlisttable` * Fixed issues with preview/viewer on Windows for: * :ref:`org.sysess.sympathy.html.report` * GeoJSON * Fixed issue in :ref:`org.sysess.sympathy.data.json.jsontotable`. * Fixed use of Fit/Fit Transform with :ref:`org.sysess.sympathy.machinelearning.one_hot_encoder`. News in 1.6.0 ------------- Gui changes ^^^^^^^^^^^ * Panning the flow view is now mapped to middle mouse button (scroll wheel click). There are also toolbar buttons for panning/selection. * New icons from Font Awesome in toolbar, menus and context menus. * Color theme based on the standard palette in Windows 10, is used for all OSs. * Changed keyboard shortcuts for Zoom to selection and Zoom fit all. * Flow Overview look and feel similar to that of Library View. * New splash screen on startup of GUI. * Removed redundant groupings in parameter GUIs, this makes some configuration GUIs more compact. * More clear highlight of invalid parameter values, on Windows. * Double click on node executes or does nothing, will never trigger configure. * Plots in Table viewer shows current resampling. * Figure viewer now supports mouse interactions for 3D axes. * Deprecation warnings are on by default. They can be disabled by unchecking *Display warnings for deprecated nodes* in the *Advanced* section of *Preferences*. Platform changes: ^^^^^^^^^^^^^^^^^ * Supports Python 3.6+. * Supports PySide 2 (Qt for Python). This results in smoother rendering, various improvements. * Asyncio is used in place of twisted, this will make it easier to install Sympathy on some OSs. * Support for installing Sympathy in a Python virtual env, on Windows. * Improved platform stability, undeterministic process crashes when running and on shutdown caused by PySide do not seem to happen anymore. * Improved structure of menus, with new groupings and consistent ordering. * Windows installer produces different package structure, similar to a wheel install. * Added json parameter type. * Added datetime parameter type. * Added dict port type. * Added Html port type. * Added GeoJSON port type. * Added Table editor for json parameter type. * General preview which can easily be added to nodes. * list_node_decorator replaces list_node_factory. * Reduce size of syx files by not writing descriptions etc for all parameters. Pretty print parameter structures for increased legibility. * Selections can be expanded while holding down Ctrl (Cmd on OSX). Ctrl + click toggles selection of individual nodes. * List parameters with static lists are now automatically updated from their definition. * Automatically fix minor inconsistencies between .value and .value_names during update_parameters. Warn about parameters that are still inconsistent during execute/configure. * Documentation improvements: * New instructions for wheel install, now also for Windows. See :ref:`installation`. * New instructions for debugging using PyCharm. See :ref:`pycharm_debug`. * Show labels for parameters in node documentation. * Improved documentation about node libraries and their structure. * No longer supported: * Python 2. Python 3.5 and older. * PySide. * Built-in debugging and editing using Spyder. * Built-in benchmark of the platform. * Add maintainer information to library.ini metadata. Useful when maintainer and original author are not the same. * Node copyright notice is now inherited from library.ini metadata if unspecified in the node. Allows less repetition when creating nodes. Node/plugin changes: ^^^^^^^^^^^^^^^^^^^^ * Datasource can supply URL to importer nodes. * Support for URL datasource in import nodes and new :ref:`org.sysess.sympathy.files.downloadfile`. * VJoin, Transpose: improved feedback when constructing unsupported columns * Select columns in Table uses new preview feature. * New :ref:`org.sysess.sympathy.visualize.figure`: * Many different improvements to the configuration gui and to the code. * Allow setting major/minor ticks. * Support for legend outside of axes. * Enabling and disabling individual plots. * Uses new preview. * The node no longer outputs its configuration in the form of a Table. * Improved support for plotting datetimes * Improved documentation. * The old figure node can still be used, but will be deprecated at some point in the near future. * Transpose can set column names even when there are no rows * Consistently using "regex" to refer to regular expressions in node configurations. * Calculator should be able to detect res-dependencies that use other quoting than ''. * Improved reading of typed input value: * Mask nodes. * Replace value in Table * Select rows in Table * :ref:`F(x)` can be configured without external file. New nodes/plugins: ^^^^^^^^^^^^^^^^^^ * :ref:`org.sysess.sympathy.html.report` * GeoJSON * :ref:`org.sysess.sympathy.html.htmltotext` * :ref:`org.sysess.sympathy.datasources.renames` * :ref:`org.sysess.sympathy.files.downloadfile` * :ref:`org.sysess.sympathy.data.json.selectkeyjson` * :ref:`org.sysess.sympathy.data.json.removekeyjson` * :ref:`org.sysess.sympathy.data.json.splitonkeyjson` * :ref:`org.sysess.sympathy.dict.insert` * :ref:`org.sysess.sympathy.dict.update` * :ref:`org.sysess.sympathy.dict.getitem` * :ref:`org.sysess.sympathy.dict.fromitems` * :ref:`org.sysess.sympathy.dict.values` * :ref:`org.sysess.sympathy.dict.keys` * :ref:`org.sysess.sympathy.dict.items` * :ref:`org.sysess.sympathy.json.jsontodict` * :ref:`org.sysess.sympathy.json.dicttojson` * :ref:`org.sysess.sympathy.json.jsontolist` * :ref:`org.sysess.sympathy.json.listtojson` Removed nodes: ^^^^^^^^^^^^^^ * Propagate Input * Repeat Item to List * Elementwise ADAFs to Tables * Filter Image (deprecated) Deprecated nodes/plugins: ^^^^^^^^^^^^^^^^^^^^^^^^^^ * Convert specific columns in Table(s) * Copy files with Datasources * Rename file(s) * Rename Files with Table * Jinja2 Template (deprecated) * Plot Table(s) * Export Figures with Datasources * Matlab Calculator * DIVA import plugin * LAA import plugin API changes: ^^^^^^^^^^^^ * sympathy.api.dtypes includes handling for reading typed values from text input * Deprecated ManagedNode base class. * Removed deprecated methods from table, adaf, table_wrapper and adaf_wrapper. * New (compatible) format for F(x) script, see :ref:`F(x)`. * Deprecate support for ${signal0} syntax in calculator plugins. * Deprecate imports() from calculator plugins. News in 1.5.3 ------------- Platform changes: ^^^^^^^^^^^^^^^^^ * Improved behavior of locked subflows: * Child nodes that have finished executing will give feedback immediately instead of after the whole locked flow has completed. * Child nodes that has executed successfully, in memory, will show up in purple color to indicate that their output port data is unavailable. * Output from locked child nodes will be shown like for other nodes, in the Messages view. * Modifying locked flows will no longer reload all nodes. * Showing progress from child nodes. * Locked subflows and lambdas (Apply, Map) will continue executing as much as possible instead of aborting after the first node has failed. * Subflow configuration wizard copes with nodes executed in locked subflows and executes them when needed to ensure that progress can be made. * Toggle-able filter for list parameters with mode selection saves vertical space. * Included brief description of current node state in tooltips. * Documentation improvements: * Links to example flows in the node documentation for many nodes, see :ref:`example_flows`. * Documented how to create custom library tags for third party libraries, see :ref:`library_tags`. * Documented how to create plugins for calculators, importer and exporter nodes, see :ref:`pluginwriting`. * Improved documentation for Tables with more modern APIs and descriptions of attributes etc. * Windows installer generates links and file associations that ignore local user site - this makes the bundled Python more stand-alone and helps avoid issues in case of package conflicts. Figure and plotting updates: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * Figure(s) can now take any sympathy object or lists of object instead of just tables. The configuration output table is now optional and need to be added explicitly to be used. Arguments can now be accessed using "arg" instead of "table" in expressions, eg. arg['X'] gives column X if the argument is a table. * Improved the Figure node with new plot types and drawing primitives: * Boxplots calculating and visualizing median, average, quantiles and outliers. * Pie-charts. * Timeline plots for drawing events or states. * Sets of unconnected lines based on starting/ending XY points. * Rectangles and Ellipses. * Text and arrows (annotations) to point out or label parts of data. * Images (using the generic Figure inputs). * Added extended example "Figure Gallery" with 12 plots to show how the new plot types and options can be used. * Added options for drawing error-bars on scatter-plots and bar-plots. * Frames and axes can now toggle visibility and the XY spines of the axes can be drawn at a predetermined XY data coordinate (eg. with the axis through origo), or at a given point on the screen (eg. center). * Fixed bug showing tooltips for plot parameters. Node/plugin changes: ^^^^^^^^^^^^^^^^^^^^ * Added error strategies to :ref:`org.sysess.sympathy.list.getitemlist`, when the list is empty. * Add different strategies for handling lists of different lengths in :ref:`org.sysess.sympathy.tuple.ziptuple2`. * Added median filtering to :ref:`Overlay Images` and :ref:`Overlay Images List` * Added :ref:`Colormap lookup` to explicitly create colors from values, useful when creating many figures where exact control of the colors are needed. New color-maps added for discrete categories of objects. * Fixed bug in :ref:`Image to List` missing one of the extracted objects. * Improved :ref:`Generic Calculator` with: * More robust handling of dependencies between calculations. * Compacted layout with a larger portion available for preview. Input signals are shown in in the tree where functions are shown. * Drag and drop of calculations. * Ability to set custom attributes for each calculation. * Error handling for :ref:`Extract lambda nodes` with choice between skip and error. * :ref:`Match Tables Nodes`: support for masked arrays. * :ref:`Filter ADAFs`: will only auto-refresh when it has been enabled. * :ref:`Adaf(s) to Table(s)`: shows preview of available signals. * Many nodes have toggle-able filter for their column selection editors. * :ref:`org.sysess.sympathy.examples.daskvisualize`: improved detection of Graphviz installation. * :ref:`org.sysess.sympathy.data.table.ensuretablecolumns`: can create type-dependent zero elements for missing columns. * MDF importer plugin has better support for reading partially unsupported or broken files. * Avoid infinite loop when loading scikit-learn 0.20. * Excel importer plugin copes with date values outside of supported range by assigning masked values. New nodes/plugins: ^^^^^^^^^^^^^^^^^^ * :ref:`Rename datasource with regex`: helps to simplify automatically creating new datasource names based on existing file names. * ADAF importer plugin for LabVIEW TDMS files. * :ref:`org.sysess.sympathy.list.transposelist` swaps the outer two layers of a nested list. * :ref:`org.sysess.sympathy.keyvaluecalculation`: calculates simple descriptive statistics for all columns in a table. API changes: ^^^^^^^^^^^^ * Generalized interface for import and export nodes and their plugins. * Opt-in support for toggle-able filter for single-select parameters with combo box editors. * Public APIs for creating importer and exporter plugins. * Datasource: added getitem. Example use: ``datasource['type']``. * Table: added setter for name. Example use: ``table.name = 'some name'``. * New exception: SyColumnTypeError, raised when creating table columns of unsupported type. News in 1.5.2 ------------- Platform changes: ^^^^^^^^^^^^^^^^^ * Fixed problem causing the OK button to be disabled (grey) when configuring subflows containing certain nodes (for example, :ref:`org.sysess.sympathy.data.adaf.interpolateadaf` and Convert Specific Columns in Table using the Wizard or Tabbed dialog. News in 1.5.1 ------------- Platform changes: ^^^^^^^^^^^^^^^^^ * Improved performance overhead of nodes that simply select some columns of the incoming data and pass them to the output. * Significantly reduced file sizes of lambdas on disk. * More careful disconnection of signals to improve stability and performance. * Added option to *Unlink* linked subflows. * Confirmation dialog when canceling subflow configurations with unsaved changes. * Better feedback and handling of node errors when configuring subflows. * Improved ability to open (somewhat corrupt) flows with duplicate identifiers, instead of failing early. * Cleared up separation between flows linked from library and normal linked subflows. * Disabled automatic documentation generation. * Cleaned up command line options, "-" is used to separate words instead of "_". * Added *Recent* libraries, to simplify switching between libraries in use. * Restructured documentation format for Nodes. * New installation instructions for Linux and Mac OS. * Saving warning filter between node executions to avoid interference. * Improved deprecation warnings for nodes, on by default. * Fixed some regressions causing library view not to update. Node/plugin changes: ^^^^^^^^^^^^^^^^^^^^ * Improved feedback and validation for many scalar parameters. Incorrect values are displayed in red and the tooltip shows why. * Added progress and current list index, in case of error - to many list nodes. * In some cases, improved performance for :ref:`Replace values in Table`. * Improved documentation regarding handling of unmatched values for :ref:`Lookup Table`. * :ref:`Select categories in ADAFs` now respects the choice made in the combo box for raster selection. * :ref:`Datasource` nodes store UNIX paths in their configuration, but show paths in native format. * MDF Importer, improved performance when importing unsorted files and added option to attempt to read incomplete files without error. * ADAF exporter, improved GUI. * CSV exporter, improved performance. * @ca.changed()@, @ca.changed_up()@, and @ca.changed_down()@ can now all handle masked arrays with scalar masks. * Filtered several expected warnings produced by nodes. New nodes: ^^^^^^^^^^^ * :ref:`org.sysess.sympathy.create.configureparameters` * :ref:`org.sysess.sympathy.create.createparameters` * :ref:`org.sysess.sympathy.data.table.createindextable` New flows: ^^^^^^^^^^ * :ref:`org.sysess.lambda.map2flow` * :ref:`org.sysess.lambda.map4flow` * :ref:`org.sysess.lambda.configurablemap4flow` News in 1.5.0 ------------- Sympathy for Data version 1.5.0 offers several improvements and new features, such as workflows in the library and routing points for connections. Nodes and flows from 1.3 and 1.4 should for the most part be compatible with 1.5.0. .. * Connections are rendered as lines. This can be changed to *Spline* in .. *Preferences -> General: connection shape* if you prefer the original look. Platform changes: ^^^^^^^^^^^^^^^^^ * Markdown_ is now supported in text fields. * Mark unavailable items in list views. * list views and combobox views can be edited when customized with editor attribute edit=True. This makes it possible to create selections that includes columns that do not exist in the input data or without executing previous nodes. * Shorter tab labels for flows, only showing parent flow names as necessary. * Only nodes that are in the current libraries can be used. Current libraries depend on the configured libraries for the current flow and the global selection from preferences. * Allow subflows to specify custom svg icons. * Route points for connections, see :ref:`route_points`. * 0 based indexing is used more consistently, that is, 0 is the index of the first element, etc. This behavior is more consistent with python code. * English/US locale is used regardless of the host language settings. * Generalized text-output port on all nodes. * Flows can be added to library, see :ref:`flows_in_library`. .. _Markdown: http://daringfireball.com/projects/markdown API changes: ^^^^^^^^^^^^ * Improved backwards compatibility of ParameterLists during execution. * ParameterLists return copies of its internal state disallowing direct mutation. * Ability to get and set dask arrays in adaf and table APIs. See example nodes: :ref:`Dask max example`, :ref:`Dask stack example` and :ref:`Dask tail example`. * Made getitem, setitem usable as a shorthand way of working with arrays columns in tables. For example table1['x'] = table2['y']. Node/plugin changes: ^^^^^^^^^^^^^^^^^^^^ * Improved masked arrays support in input data to :ref:`Select rows nodes` get normal numpy arrays instead of pandas Series. * :ref:`Select rows in Table with Table` understands literal comparison operators such as '==' or '>' in addition to their old text representations. * :ref:`HJoin Tables` gives consistent names when renaming duplicate columns * :ref:`Figure` allows you to rotate bin labels in bar plots. * New parameter in :ref:`Select rows with Table(s)` * :ref:`Select rows nodes` * Faster execution of :ref:`Table` and :ref:`Select categories in ADAFs` * Responsive preview for :ref:`Calculator List` and :ref:`Calculator` API changes ^^^^^^^^^^^ * Added MATLAB API for writing scripts executed by the Matlab node * Added update method to Attributes class. (ADAF API) * Added support for placeholder text in :ref:`lineedit_editor` in parameter helper * Added visibility and enable/disable slots to ParameterValueWidget Bug fixes ^^^^^^^^^ * Fixed name and type of output port of :ref:`Report Apply` nodes * Fixed a bug where save file dialog wouldn’t show up at all when trying to save subflow on Windows, if the subflow label contained some specific unallowed characters * Made sure that aborting a subflow doesn't also abort nodes outside of the subflow * Fixed a bug where linked subflows were sometimes inserted with absolute path Stability ^^^^^^^^^ * Improved reliability when working with lambdas, maps and apply nodes Deprecated nodes ^^^^^^^^^^^^^^^^ Deprecated nodes don't show up in the library view, but can still be used in workflows. * Type specific versions of list operation nodes (such as Get Item Table and Append ADAF). * Old FilterADAFs node News in 1.3 series ------------------ Sympathy for Data version 1.3.0 offers several new and prominent features such as generic types, higher order functions and much improved support for linked subflows. Many small improvements were made to the standard node library. Nodes will often cope better with empty input data and deliver informative, but less detailed, feedback. Nodes from 1.2.x should be compatible with 1.3.0 but there are new, more succinct, ways of writing nodes for 1.3.x that are not backwards compatible with 1.2.x. When writing new nodes, consider which older versions of the platform that will be used. New features ^^^^^^^^^^^^ * :ref:`Generic types` * :ref:`Higher order functions`: Lambda, Map and Apply * Official, and much improved, support for :ref:`linked subflows` * Official support for :ref:`locked_subflows` * New library structure using tags New nodes ^^^^^^^^^ * New generic versions of all list operations * :ref:`Ensure columns in Tables with Table` * :ref:`Conditional Propagate` * :ref:`Extract Lambdas` builtin node for reading lambda functions from existing workflows User interface ^^^^^^^^^^^^^^ * Right-click on an empty part of the flow to insert :ref:`higher order functions`. * New command in context menu for inserting a subflow as a :ref:`link`. * Improved file dialogs in node configurations, by using native dialog when asking for an existing directory and starting file dialogs from currently selected file path. API changes ^^^^^^^^^^^ * Simpler APIs for writing nodes. See :ref:`nodewriting` * New method in :ref:`adafapi`: ``Group.number_of_rows`` * Configuration widgets can expose a method called save_parameters which is called before the gui is closed. See :ref:`custom_gui` * Added API (parameter helper): List parameter widgets emit ``valueChanged`` signal * Improved slicing of (sy)table with slice object with negative or undefined stride * Automatically update order, label, and description for parameters when the node’s definition changes * :ref:`NodeContext ` is no longer a named tuple * Added new method: :meth:`NodeContext.manage_input`. A managed input will have its lifetime decided outside of the node Linked/locked subflows ^^^^^^^^^^^^^^^^^^^^^^ * Include subflows relative to path of parent flow, not relative to root flow. This affects where sympathy searches for linked subflows inside linked subflows and should hopefully feel more natural than the old system * Allow opening of flows with broken links * Import and export nodes can now be used inside locked subflows and lambdas * Made it impossible for flows below a locked flow to themselves be locked * Improved abort for locked subflows Node changes ^^^^^^^^^^^^ * :ref:`Report` framework: histogram2d graph layer is now called heatmap and can handle different reduction functions (count, mean, median, max, min). * Improved XLS(X) :ref:`import`/:ref:`export`. Especially handling of dates, times, and mixed types. Cells formatted as Time are now imported as timedeltas. * Renamed Sort Table(s) to :ref:`Sort rows in Table(s)` * :ref:`Calculator List`: chooses columns case-sensitively on Windows too. * :ref:`Calculator List`: shows number of output rows in preview in calculator gui. * :ref:`VSplit Table`: Removed constraint that the index should be sorted. The elements will be grouped by the first occurrence of each unique value. * :ref:`Convert columns in Table`: Added conversion path between datetime and float. * :ref:`Select columns in ADAF with structure Table` now works as expected when *Remove selected* has been checked. * :ref:`Select rows in Table with Table` offers a choice of reduction function between rows in config table. Previously it only read first row of the config table. * "Slice List of ADAFs/Tables": Basic integer indexing now works as expected. * Improve handling of one sample signals in :ref:`Interpolate ADAF(s) ` * :ref:`Report Apply nodes ` output datasources to created files * Improved :ref:`CSV import
`. Can now handle empty input, input with only one row, with or without trailing newline, and files with errors towards the end. It also features a new option for how to handle errors when importing a file. Header row has been made independent of the other input boxes, and no longer affects the data row. When read to end of file is selected, the number of footer rows is ignored. Delimiter detection was improved * Fixed issues with nesting of higher order functions (:ref:`Map `, :ref:`Lambda` and :ref:`Apply`) * Improvements to :ref:`reporting `: Improved bin placement and x-axis extent of 1d histograms. Automatically set axes labels from data source if they are empty. Added option "Lift pen when x decreases" to line graph layer. Added vline layer in reporting tool. * Several nodes are better at forwarding attributes, table names, etc. to output `Slice data Table (deprecated)`, :ref:`Select columns in ADAF(s) with structure Table(s)`, :ref:`Interpolate ADAF(s) `, and :ref:`Rename columns nodes ` * Many nodes are better at handling missing or incomplete input data: :ref:`Filter Rows in Table`, :ref:`Replace values in Tables`, :ref:`Detrend ADAF(s) `, :ref:`ADAF(s) to Table(s) `, :ref:`Select Report Pages`, :ref:`Scatter nodes `. * Added 'calculation' attribute on all output columns from :ref:`Calculator List` node * :ref:`Export Tables` and :ref:`Archive files` create missing folders * Fixed :ref:`Export Texts` Other improvements ^^^^^^^^^^^^^^^^^^ * Added :ref:`default workflow environment variables ` ``SY_FLOW_FILEPATH``, ``SY_FLOW_DIR`` and ``SY_FLOW_AUTHOR``. All flows have these and they can't be set or deleted. * Subflows can define :ref:`workflow variables `. Each subflow specializes the variables of its parent flow, so that the parent flows vars are accessible in the subflow but not vice versa. * Improve performance by skipping validation of any nodes that don’t implement :meth:`verify_parameters` * Improve performance by changing compression settings for sydata files, compression is faster but compresses slightly less * Pretty print workflow xml files, making diffs possible New requirements: * Requiring pandas version 0.15 for the CSV import, for versions before 0.15 down to 0.13 it will still work but may behave slightly differently in edge cases with blank rows News in 1.2 series ------------------ Sympathy for Data version 1.2 is a significant minor release for Sympathy for Data. It features several prominent new features, improved stability and more. It is however not redesigned and with only a few small modifications, all existing nodes and flows should work as well as in 1.1. The bundled python installation has been upgraded with new versions of almost every package. Added to the packages is scikit-learn, used for machine learning. Our investigations suggest that the new package versions are reasonably compatible with old nodes and cause no significant differences for the standard library. New features ^^^^^^^^^^^^ * Added support for using environment variables, and per installation/workflow variables. The variables which can have a default value are used in string fields of configuration widgets to enable parametrization. See :ref:`env_vars`. * Added support for profiling, with the ability to produce graphs if Graphviz is available. See :ref:`profiling`. * Added support for debugging single nodes with data available from Sympathy using spyder. See :ref:`spyder_debug`. * Added new Node Wizard for generating new nodes. See :ref:`node_wizard`. * Added support for configuring subflows by aggregating selected node configurations. See :ref:`subflow_config`. * Improved support for plugins in third party libraries. It is no longer necessary to add the folder with the plugin to python path in preferences * Support for adding custom data types in third party libraries. See :ref:`create_type`. * Significantly improved handling of unicode paths including the ability to install Sympathy and third party libraries in a path with unicode characters Nodes and plugins ^^^^^^^^^^^^^^^^^ * Added CarMaker type 2 ERG ADAF importer plugin called “CM-ERG” * Plugins can now export to non-ascii filenames * Fixed MDF export of boolean signals * Added generating nodes for empty Table, Tables ADAF and ADAFs. * Convert column nodes can convert to datetime * Calculator node can produce compact output for length matched output * Lookup nodes handles both event column and other columns with datetimes * Time Sync nodes “SynchronizeLSF” strategy should work as expected again. The Vjoin index option is now only used for the ”Sync parts” strategy New command line options ^^^^^^^^^^^^^^^^^^^^^^^^ See :ref:`start_options` for more info. * Added new command line option, '--generate-documentation' for generating documentation from CLI * Added 'exit-after-exception' argument which is activated by default in CLI. It makes Sympathy exit with error status if an unhandled exception occurs in a signal handler. * Added separate flag: --node-loglevel, for controlling the log output from nodes. * Made it possible to set the number of workers using --num-worker-processes n. API changes ^^^^^^^^^^^ * Libraries must now have only a single python package in their Common folders. See :ref:`nodewriting`. In the Standard Library this package is called sylib * Removed ``has_parameter_view`` from node interface. See :ref:`custom_gui`. * Changed default unit for time series to empty string instead of ``'unknown'``. * Added ``has_column`` method in sytable and added corresponding method in ``table.File`` * Accessing an ADAF basis which does not exist will raise a KeyError * Improved node error handling, making it possible for nodes to issue user friendly error messages as well as warnings. See :ref:`node_errors`. * Expanded and improved documentation, including API references for all default data types, and documentation on how to create your own data type * Improved error handling in many data type API functions User interface ^^^^^^^^^^^^^^ * Improved selection and context menu handling * "Help" in node context menus will now also build documentation if necessary. * Allow connections to be made by dragging from an input to an output port * Added zoom with Ctrl/Cmd + scroll wheel * Added working stop button. * Improved the presentation of data in the viewer with a clearer font and better size handling as well as coloring of columns by data type * Improved undo/redo functionality, making more operations available in the undo history Stability ^^^^^^^^^ * Avoid hanging on Windows when too much output is produced during startup * Avoid infinite wait during node validation