.. This file is part of Sympathy for Data.
.. Copyright (c) 2010-2012 Combine Control Systems AB
..
.. SYMPATHY FOR DATA COMMERCIAL LICENSE
.. You should have received a link to the License with Sympathy for Data.

.. _nodewriting:

Node writing
============
Sympathy's standard library contains a lot of useful nodes and it is also
possible to add complete third-party libraries without writing any code
yourself. But sometimes you might come to a point when the node that you need
simply has not been written yet. One option is to write your own node.

Sympathy nodes are written in Python_. To create and edit nodes you will need
some text editor or Python IDE. If you do not already have a favorite
editor/IDE, we recommend `Visual Studio Code <https://code.visualstudio.com/>`__
or `PyCharm <https://www.jetbrains.com/pycharm/>`__ (PyCharm Community Editon is
free to use).

.. _Python: https://python.org


.. _`node_wizard`:

Creating new nodes
------------------
The easiest way to get started writing your own node is to use the node wizard.
It will create an outline of a node code for you, so you can start right at
implementing the actual functionality of the node.

To start the node wizard go to *File* -> *Wizards* -> *New Node*. The wizard
will guide you to enter the information used to create a new node:

#. Choose Library, for the new node
#. Node Location, in the chosen library
#. Base Information, for example, *Name* and *Description*
#. Ports, inputs and outputs
#. Summary, with preview of the code output

Enter information to make the node easy to find and understand. See the section
:ref:`node_meta` for details about the different fields. When finishing
the wizard, the node will be written to your chosen location.

.. warning::
    You can in theory add new nodes to Sympathy's standard library (by moving
    the python files manually) or to some third-party library and have them
    appear in the Library view in Sympathy. This is not recommended though as
    it makes it much more difficult to manage library updates and such. In order
    to place nodes under a certain folder in the library view, read the section
    "Library tags" below.


.. _`node_code`:


.. _`reload_code`:

Reload after changes
--------------------

After making changes to a library, you need to :ref:`reload the libraries
<reload_library>` in Sympathy. For changes to python code not affecting node
interfaces, it is enough to reload workers (*Control* -> *Reload Workers*),
which is faster than reloading the libraries.

The node code
-------------
Nodes are loaded from their definition files when Sympathy is started, and only
Python files with names starting with ``node_`` and ending with ``.py`` will
generate nodes. You can place the nodes in subfolders to group related nodes
together. Now, create a file called ``node_helloworld.py`` and open it in your
editor of choice.

Without further ado let us look at the code for a simple example node::

    from sympathy.api import node as synode
    from sympathy.api.nodeconfig import Port, Ports, Tag, Tags


    class HelloWorld(synode.Node):
        name = 'Hello world!'
        description = 'An amazing node!'
        nodeid = 'com.example.boblib.helloworld'
        tags = Tags(Tag.Development.Example)

        def execute(self, node_context):
            print('Hello world!')

Copy this code into the file ``node_helloworld.py``, :ref:`reload the libraries
<reload_code>` and add the node to a new workflow.


A node is defined as a Python class which inherits from
``sympathy.api.node.Node``. The name of the class is irrelevant. The
class definition starts with a description of the node, then you have to
define some variables that contain meta data about the node. Lastly, you
write the method that actually controls the behavior of the node (such as
``execute``). For all the details of what goes in a node class, please refer to
the :ref:`node_reference`.

You can place several such classes in the same python file, but only do this if
they are clearly related to one another.


Library tags
------------
In the example above you may have spotted the ``tags`` variable. Each node should
have a library tag. This specific one::

    tags = Tags(Tag.Development.Example)

will place the node into the *Development->Example* tag in the library
hierarchy.

To see what different tags are available have a look in
*sylib/librarytag_sylib.py* or look at the code of any specific
node which uses the tag that you are interested in. If you don't specify a tag
the node will be shown under a folder called "Unknown" in the library view.


.. _node_ports:

Adding input and output ports
-----------------------------
The possibilities for a node with neither input nor output ports are quite
limited. To add a single Table output port to your node, add the class variable
``outputs`` as follows::

    import numpy as np

    from sympathy.api import node as synode
    from sympathy.api.nodeconfig import Ports, Port, Tags, Tag

    class FooTableNode(synode.Node):
        """Creates a foo Table"""

        name = 'Create foo Table'
        nodeid = 'com.example.boblib.footable'
        tags = Tags(Tag.Development.Example)

        outputs = Ports([Port.Table('Table of foo', name='foo')])

        def execute(self, node_context):
            output_port = node_context.output['foo']
            output_port['foo column'] = np.array([1, 2, 3])

Also notice the new `import` statements at the head of the file. :ref:`Reload
the library <reload_code>` and add a new instance of your node to a workflow. You
can see that it now has an output port of the Table type.

Writing to the output port is as easy as adding those two lines to your
``execute`` method.

The object ``output_port`` which is used in the example is of the class
:class:`sympathy.api.table.Table`. Once again, :ref:`reload the libraries
<reload_code>`, add the node to a flow, and execute it. With these changes the
node will produce an output table with a single column called *foo column*
containing the values 1, 2, and 3.

Inspect the output by double clicking on the output port of your node. It will
open in Sympathy's internal data viewer.

If you want your output to be a modified version of the input you can use the
``source`` method::

    import numpy as np

    from sympathy.api import node as synode
    from sympathy.api.nodeconfig import Ports, Port, Tags, Tag

    class AddBarNode(synode.Node):
        """Adds a bar column to a Table."""

        name = 'Add bar column'
        nodeid = 'com.example.boblib.addbar'
        tags = Tags(Tag.Development.Example)

        inputs = Ports([Port.Table('Input Table', name='foo')])
        outputs = Ports([Port.Table('Table with some added bar', name='foobar')])

        def execute(self, node_context):
            input_port = node_context.input['foo']
            output_port = node_context.output['foobar']
            output_port.source(input_port)
            number_of_rows = input_port.number_of_rows()
            output_port['bar'] = np.arange(number_of_rows, dtype=int)

All the other basic port data types are also available in the ``Port`` class,
such as ``ADAF``, ``Datasource``, and ``Text``. Try changing your port to some
other type and add it again to a flow (do not forget to reload libraries first)
to see the port data type change. You can also just as easily add several input
or output ports to a node::

    inputs = Ports([Port.Datasource('Input foo file', name='foofile'),
                    Port.ADAFs('All the data', name='alldata')])
    outputs = Ports([Port.Table('Table with baz', name='baz'),
                     Port.ADAF('The best data', name='outdata')])

Note though that the different data types have different APIs whose references
can be found here: :ref:`datatypeapis`.

If you need ports of some type which does not have its own method in
:class:`Port` (such as generic types or lambdas) see :ref:`custom_ports`.


.. _node_parameters:

Adding a configuration GUI
--------------------------
When writing a node we want to make it as generally useful as possible. This
usually includes adding a configuration gui where the user can tweak what the
node does.

Going back to the original Hello world node, let us add a choice of what
greeting to print by adding a configuration gui to the node.

Parameters are defined in the class variable ``parameters``. Create a new
parameters object by calling the function ``synode.parameters``. Then add all
the parameters with methods such as ``set_string``. In our example it would
look something like this::

    from sympathy.api import node as synode
    from sympathy.api.nodeconfig import Tags, Tag


    class HelloWorldNode(synode.Node):
        """Prints a custom greeting to the node output."""

        name = 'Hello world!'
        description = 'An amazing node!'
        nodeid = 'com.example.boblib.helloworld'
        tags = Tags(Tag.Development.Example)

        parameters = synode.parameters()
        parameters.set_string(
            'greeting',
            value='Hello world!',
            label='Greeting:',
            description='Choose what kind of greeting the node will print.')

        def execute(self, node_context):
            greeting = node_context.parameters['greeting'].value
            print(greeting)

Once again try reloading the library and readding the node to a flow. You will
notice that you can now configure the node. A configuration GUI has been
automatically created from your parameter definition. As you can see the
``label`` argument is shown next to the line edit field and the ``description``
is shown as a tooltip. Try changing the greeting in the configuration

.. figure:: screenshot_hello_parameter.png
   :alt: Parameter gui example
   :align: center

You can add parameters of other types than strings as well by using the methods
``set_boolean``, ``set_integer``, ``set_float``, ``set_list``. Most of them
have the same arguments as ``set_string``, but lists are a bit different. A
simple example of storing a list might look like this::

    parameters.set_list(
        'toppings', label='Pizza toppings',
        description='Choose what toppings you want on your pizza.',
        list=['Cheese', 'Tomato sauce', 'Pineapple',
              'Ham', 'Anchovies', 'Mushrooms'],
        value_names=['Cheese', 'Tomato sauce'],
        editor=synode.editors.multilist_editor())

This list is named "toppings" and has the available options specified by the
``list`` argument. The ``value_names`` argument specifies which options in the
list that are selected by default. The ``editor`` argument is used to specify
that we want this list to be shown in a list view with multiple selection.

See :ref:`parameter_helper_reference` for more details or see
:ref:`All Parameters Example` for more examples of how to use all the different
parameter types and editors.


.. _node_errors:

Errors and warnings
-------------------
Any uncaught exceptions that occur in your code will be shown as *Exceptions*
in the error view. The stack traces in the details can be very valuable while
developing nodes, but are pretty incomprehensible for most users. Because of
this you should always try to eliminate the possibility of such uncaught
exceptions. If an error occurs which the node cannot recover from you should
instead try to raise an instance of one of the classes defined in
``sympathy.api.exceptions``. Here is an example that uses
``SyConfigurationError``::

    from sympathy.api.exceptions import SyConfigurationError
    from sympathy.api import node as synode
    from sympathy.api.nodeconfig import Tags, Tag

    class HelloWorldNode(synode.Node):
        """Prints a custom greeting to the node output."""

        name = 'Hello world!'
        description = 'An amazing node!'
        nodeid = 'com.example.boblib.helloworld'
        tags = Tags(Tag.Development.Example)

        parameters = synode.parameters()
        parameters.set_string(
            'greeting',
            value='Hello World!',
            label='Greeting:',
            description='Choose what kind of greeting the node will print.')

        def execute(self, node_context):
            greeting = node_context.parameters['greeting'].value
            if len(greeting) >= 200:
                raise SyConfigurationError('Too long a greeting!')
            print(greeting)

This will produce a more user friendly error message.

If you simply want to warn the user of something that *might* be a concern but
which does not stop the node from performing its task, use the function
``sympathy.api.exceptions.sywarn``::

    from sympathy.api.exceptions import sywarn
    from sympathy.api import node as synode
    from sympathy.api.nodeconfig import Tags, Tag

    class HelloWorldNode(synode.Node):
        """Prints a custom greeting to the node output."""

        name = 'Hello world!'
        description = 'An amazing node!'
        nodeid = 'com.example.boblib.helloworld'
        tags = Tags(Tag.Development.Example)

        parameters = synode.parameters()
        parameters.set_string(
            'greeting',
            value='Hello world!',
            label='Greeting:',
            description='Choose what kind of greeting the node will print.')

        def execute(self, node_context):
            greeting = node_context.parameters['greeting'].value
            if len(greeting) >= 100:
                sywarn("That's a very long greeting. Perhaps too wordy?")
            print(greeting)

See :ref:`error window` for more info about how the error view shows different
types of output. See the :ref:`Error Example` node for another example.

.. _documenting_nodes:

Documenting nodes
-----------------
Documentation for your library can be built into a convenient HTML format by
Sympathy. The automatic node documentation uses docstrings from node classes and
description fields (top-level node description, node port descriptions and
descriptions of node parameters). Since Sympathy uses `Sphinx
<http://www.sphinx-doc.org/en/master/index.html>`_ for producing HTML
documentation for nodes and the platform, it requires docstrings to be written
in Sphinx compliant reStructuredText format. See also the :ref:`node_reference`
for more information.

Additionally, documentation for other functions and classes can benefit from
using numpy docstring format, see `A Guide to NumPy/SciPy Documentation
<https://docs.scipy.org/doc/numpy/docs/howto_document.html>`_.

See :ref:`documenting_libraries` for information about documenting libraries and
building library documentation.

Reusable nodes
--------------
Follow these simple guidelines to make sure that your node is as reusable as
possible.

- Break down the task into the smallest parts that are useful by themselves and
  write nodes for each of those, instead of writing one monolithic "fix
  everything" node. Take some inspiration from the Unix philosophy; every node
  should "do only one thing, and do it well".
- Try to work on the most natural data type for the problem that you are trying
  to solve. When in doubt go with Table since it is the simplest and most
  widely applicable data type.
- Do not hard code site specific stuff into your nodes. Instead add
  preprocessing steps or configuration options as needed.
- Add documentation for your node, describing what the node does, what the
  configuration options are, and whether there any constraints on the input
  data.
- When you write the code for your node, remember that how you write it can
  make a huge difference. If others can read and easily understand what your
  code does it can continue to be developed by others. As a starting point you
  should try to follow the Python style guide (PEP8_) as much as possible.

.. _PEP8: https://www.python.org/dev/peps/pep-0008/

If your nodes are very useful and do not include any secrets you may be able to
donate it to Combine_ for inclusion in the standard library. This is only
possible if the node is considered reusable.

.. _Combine: https://www.sympathyfordata.com