.. This file is part of Sympathy for Data. .. Copyright (c) 2010-2024 Combine Control Systems AB .. .. SYMPATHY FOR DATA COMMERCIAL LICENSE .. You should have received a link to the License with Sympathy for Data. News in 6.0.0 ------------- User interface ^^^^^^^^^^^^^^ * Snap connections to nearby ports. When making a new connection between two ports, the connection now snaps to nearby eligible ports. This will make it easier to hit the ports. Also change cursor when hovering a port with incompatible type to make it more clear that it will not result in a connection. * Drop or paste data files in flow to add import nodes. * Double-clicking an input port will now open a data viewer for the data that is being passed to that port. Previously only output ports could show data viewers. * Fix width of y-signal selection list in Table viewer. You should now be able to see the full column names even for for very long column names. * Limit check actions (check all, uncheck all, and invert checked) for multi-select list parameters by the filter, when it is active. * Clear filter after closing search (magnifying glass icon) to make avoid hidden state in multi-select lists and combo-box editor parameters. * Show some advanced node/flow properties using the JSON viewer and disable automatic syntax highlighting for large items to improve responsiveness. * Add *Reload Workers* as a faster alternative to *Reload Library* for code changes that do not affect node interfaces. See :ref:`reload_code` for more information. * Add basic support for plotting of complex numbers to the Table viewer. A complex column `X` appear for selection as four separate columns: `X` (real), `X.complex`, `X.abs`, and `X.angle` (-pi, pi), in plot combo-boxes. * Fix a rare bug that could sometimes make it so that some optional ports for a node couldn't be removed. * Detect when a syx file contains multiple connections to a single input port and automatically remove one of the connections. Such flows should not be possible to create in Sympathy, but only by editing the syx file outside of Sympathy. **Migrating:** In these cases one arbitrary connection is selected from among the conflicting connections and is kept in the flow. This means that executing a flow with such conflicting connections could now run different nodes that before 6.0.0. To fix this edit the flow and make sure that only the correct node is connected to the input port. Platform changes ^^^^^^^^^^^^^^^^ * Cache result of library scanning for faster start-up and loading of flows that add libraries. Use :ref:`reload library ` after updating or making changes to a library for the changes to have effect. * Improve loading time for big flows by caching some results to avoid repeated work. * Don't write version, description, author or copyright for nodes in syx files. Also, only write positions for flowios in the linked flow, never in the parent. All of these fields were already ignored when reading the flow but could lead to large useless diffs. This change will also create diffs, but should at least pave the way for fewer such diffs in the future. * License key can now be set in HKEY_LOCAL_MACHINE and not just in HKEY_CURRENT_USER as was the case before. * Add version number to app name in "Open with..." menu, for gui and viewer. * Compress sydata files more eagerly, resulting in smaller files. * Nodes with list ports should no longer keep data in memory more that necessary when running in locked mode. This could reduce memory usage somewhat when running such flows. * Fix a bug where error and exception messages were logged at the wrong level in the cli. They are now correctly logged at the error level and should now show up with the default logging configuration. * Port scheme is no longer used to determine actual data format. It is still written in syx files for compatibility with older versions. * Existing nodes always use the port configuration as stored in flows. This means that updating a node's default ports should be safe and will not impact existing flows. For some nodes saved by older versions this can lead to lost ports, to avoid this you can first load and then save the flow using version 5.x or simply replace them with new ones from the library. * Settings file are no longer copied from its old location. The copying was introduced in version 2.0.0 to ease the transition, but is now more likely to surprise than help. Node changes ^^^^^^^^^^^^ * Improvements to :ref:`Parquet Table Importer `: * Allows importing only a selection of columns and rows instead of always importing the full file. * Improved performance of the configuration gui and its preview. * Importer now sets filename without path as table name. * New node: :ref:`org.sysess.sympathy.visualize.bokehfigureparts`, used to embed bokeh plots in custom html templates. * Make datasource port opt-in for new :ref:`F(x) nodes `, see :ref:`node_section_ports` for more information about nodes with configurable number of ports. * Executing Spectral (short time fourier) transform with no column selected, now gives an error. **Migrating:** Manually configure the node and select the column by name. * Output columns with unsupported types unfiltered instead of raising error in :ref:`Filter ADAFs`. * Add parameter "stratify" to node `Split Dataset (Experimental)` and remove "random_state" parameter to align with :ref:`Simple Train-Test Split`. * Remove deprecated metrics :guilabel:`kulsinski` and :guilabel:`matching` from Density based spatial clustering (DBSCAN). **Migrating:** A migration exists which will choose another metric if one of the removed metrics was selected. * Skip parameter change instead of removing connections when the user configures any of :ref:`org.sysess.builtin.extractlambdas`, :ref:`org.sysess.builtin.extractflowslambdas`, and :ref:`org.sysess.builtin.empty` to a type that doesn't match the connection :ref:type. * Improve signal selection gui in :ref:`Iterpolate ADAF(s)`:, by removing the :guilabel:`Resample all signal` checkbox and just using the mode drop-down from the signal parameter like most other list parameters. Existing copies of this node are updated automatically via a migration. * Deprecated a behavior in :ref:`org.sysess.sympathy.data.adaf.interpolateadafswithtable` where multiple specifications were in some cases permitted for the the same output raster. This now results in a warning and will become an error in 7.0.0. **Migrating:** Remove such duplicate specifications from the node configuration before version 7.0.0. * Add migration from the old `JSON to Table (deprecated)` node to the new :ref:`Json to Table` node. Apply it in the node's context menu: :menuselection:`Refactor > Migrate node`. * Take extra care to ensure that imported/exported XLSX files are closed after reading/writing. * Fix bug where table name is not passed to dataset when converted using `Tables to Dataset (Experimental)``. * Fix a bug in :ref:`org.sysess.sympathy.generate.signaltable` where the optional noise added was not centered around zero, and so would displace the center of the generated signals. All generated signals are now properly centered at zero regardless of noise. * Fix a bug where the ADAF viewer wouldn't update the list of rasters after changing system. * MDF3 exporter will now always write mdf version 3.30 files. Before this change an Adaf with a faulty value in MDF_version could result in exported MDFs with incorrect version numbers which made some programs unable to open the exported file. * Remove unused parameter :guilabel:`Pass through X` in Predict Dataset (Experimental). To remove the parameter from existing copies of the node, remove the node from the flow and add a new copy from the the node library. * Fix a bug where non-experimental machine learning models couldn't be applied to datasets in Fit Dataset (Experimental). * Fix a bug where column selection wasn't taken into account in Select columns in Table Dataset (Experimental). * Fix a bug with the :guilabel:`PolynomialFeatures` in Transform Table Dataset (Experimental) where it would only look at a single feature at a time not create any interaction features. * Removed deprecated nodes: **Migrating:** See :ref:`deprecations` for replacements for these nodes. * Remove *Slice Data Table(s) (deprecated)* * Remove *Slice List (deprecated)* * Remove *JSON to Table(s) (deprecated)* * Remove *Json to Dicts* * Remove *Json to Lists* * Remove *Select key JSON (deprecated)* * Remove *Remove key JSON (deprecated)* * Remove *Split on key JSON (deprecated)* Python and package versions ^^^^^^^^^^^^^^^^^^^^^^^^^^^ * Update bundled Python version to 3.11. * Update most bundled Python packages. * Add support for Python 3.12. * Remove support for Python 3.9 and older. * The packages and versions in Sympathy's Python environment are no longered validated on each startup. API changes ^^^^^^^^^^^ * Table columns from node input ports are now read-only and may not be changed inplace. To modify the data make a copy of it first. This also affects other types that use tables internally (e.g., ADAF). **Migrating:** Example of a situation with inplace modification:: def execute(self, node_context): col_data = node_context['input_table']['column_name'] col_data += 1 # <-- This will fail in 6.0.0 node_context['output_table']['column_name'] = col_data This can be changed to not modify the input column:: def execute(self, node_context): col_data = node_context['input_table']['column_name'] col_data = col_data + 1 node_context['output_table']['column_name'] = col_data In some cases the easiest fix can be to explicitly copy the column data with ``col_data.copy()``. * :ref:`sympathy.api.node_helper.node_list_decorator` can now handle optional ports. Previously, telling it to automatically loop over an optional port would result in an exception. * Deprecate type module functions for checking file type. Use ``is_type`` from the corresponding ``TypeAlias`` (or ``FileList``) subclass instead. **Migrating:** For example, replace ``table.is_table`` with ``table.Table.is_type`` (and ``table.is_tables`` with ``table.FileList.is_type``). Affects the following type modules: adaf, bokeh, datasource, figure, report, table and text. The deprecated functions will be removed in 7.0.0. * Remove deprecated ``sympathy.api.qt``. **Migrating:** Use ``sympathy.api.qt2`` instead or import ``PySide6`` directly. * Remove deprecated ``sympathy.api.figure.File.rotate_xlabels_for_dates`` and ``sympathy.api.figure.SyAxes.rotate_xlabels_if_date``. * Parameter API: * Defining node parameters as a dictionary now gives an error. **Migrating:** Use ``sympathy.api.parameters()`` factory function instead. * Remove deprecate ``ParameterEntity.value_changed``, ``ParameterGroup.trigger_value_changed`` and ``ParameterGroup.add_handler_to_subgroup``. **Migrating:** The use case is now covered by ``ParameterWidget.parameter_changed`` on the parameter gui. * :ref:`Type APIs`: * Remove deprecated aliases for port constructors: ``MongoDB``, ``Html``, ``Image``, ``Images``, ``Report``, ``Reports``. **Migrating:** Replacement names end with ``Port``: ``MongoDBPort``, ``HtmlPort``, etc. * Added classmethod ``Text.from_str()`` for easily creating Text objects from python strings. * Added ``bokeh.Bokeh.source()`` implementation. * :ref:`Table API`: * Raise error in ``table.Table.set_column_from_series`` when object series contains mixed, incompatible types. * Fix possible overflow in ``table.Table.from_dataframe``. The size of the datatype in the resulting dataframe is now influenced by the actual values instead of using the default size for the current platform. * ``figure.Figure.save_figure`` will now raise a ``SyDataError`` for unsupported file formats instead of just printing a warning. * Remove ``list_wrapper`` attribute from fx_wrapper and its subclasses in table_wrapper and adaf_wrapper. * Deprecated ``.data`` and ``.attributes`` on interactive node instances. Instead just access the parameters normally, directly in ``node.parameters``. See documentation for :ref:`interactive mode`. Documentation ^^^^^^^^^^^^^ * Add page in appendices documenting :ref:`return codes`. * Add cli option to remove the docs cache folder: ``sympathy clear --docs``. * Third party node libraries can now customize their documentation theme using css. The custom css is placed in a file called *custom.css* in their documenatation root. * Change location of docs storage folder from /Nodes/py3-6.0.0/doc/ to /Docs/6.0.0/.