(closes #3432) Transformation functionality to split and propagate kwargs to subtransformations

changelog

@@ -1,3 +1,6 @@
+   20) PR #3433 for 3432. Introduces initial support for sub transformations
+   and option handling.
+
    19) PR #3404 for 3403. Modernises the remaining runme examples.
 
    18) PR #3410 towards #2668. Update more Transformations to use kwargs.

doc/developer_guide/transformations.rst

@@ -567,4 +567,48 @@ the ``ParallelLoopTrans`` class for reference):
 7. Repeat this process for any classes that the class inherits from.
 
 
+Transformations options and meta-transformations
+================================================
+
+Sometimes it is useful to implement transformation that in turn call one or
+multiple other transformations. In these cases we often want to propagate the
+kwargs not only to the superclass (by inheritance) but also to the internally
+used transformations, but we cannot pass the whole kwargs everywhere because
+many options will only be valid for certain transformations.
+
+To easily decide which options provide to each transformation we are developing
+the concept of ``_SUB_TRANSFORMATIONS``. Populating this class attribute allows
+`self.split_kwargs(**kwargs)` to return the set of valid kwargs for itself and
+each of the listed transformations, while maintaining the `validate_options`
+functionality. Typically both the apply and the validate need to split the
+kwargs as in the example below:
+
+.. code-block:: python
+
+    class TestMetaTrans(Transformation):
+        ''' MetaTrans Example'''
+        _trans1 = Called1Trans
+        _trans2 = Called2Trans
+        _SUB_TRANSFORMATIONS = [Called1Trans, Called2Trans]
+
+        def validate(self, node, **kwargs):
+            self_kwargs, tr1_kwargs, tr2_kwargs = self.split_kwargs(**kwargs)
+            self._trans1().validate(node, **tr1_kwargs)
+            self._trans2().validate(node, **tr2_kwargs)
+            self.validate_options(**self_kwargs)
+            super().validate(node, **self_kwargs)
+
+		def apply(self, node, my_option):
+		    # Omitted code before using the subtransformations...
+            _, tr1_kwargs, tr2_kwargs = self.split_kwargs(
+                my_option=my_options, **kwargs)
+            self._trans1().apply(node, **tr1_kwargs)
+            self._trans2().apply(node, **tr2_kwargs)
+
+Note that the TestMetaTrans docstring won't mention the SUB_TRANSFORMATIONS
+options, therefore it is currently recommended to explicitly mention the
+relevant internal transformations in the apply docstring with a
+``:py:class:`` tag (to create a link). TODO #3330 will explore automating
+this step.
+
 .. footbibliography::

doc/user_guide/profiling.rst

@@ -107,9 +107,9 @@ libraries that come with PSyclone:
     for each MPI process), and not thread-safe.
 
 ``lib/profiling/dl_timer``
-    This wrapper uses the apeg-dl_timer library. In order to use
+    This wrapper uses the dl_timer library. In order to use
     this wrapper, you must download and install the dl_timer library
-    from https://bitbucket.org/apeg/dl_timer. This library has
+    from https://github.com/stfc/dl_timer. This library has
     various compile-time options and may be built with MPI or OpenMP
     support. Additional link options might therefore be required
     (e.g. enabling OpenMP, or linking with MPI).

examples/gocean/eg5/profile/Makefile

@@ -108,9 +108,9 @@ tau: $(INF_LIB)
 	        tau_lib $(NAME).tau
 
 # The dl-timer library is available from:
-# https://bitbucket.org/apeg/dl_timer/src/master/
+# https://github.com/stfc/dl_timer
 # As default assume dl_timer is installed 'next' to PSyclone.
-DL_TIMER_ROOT?=$(ROOT_DIR)/../apeg-dl_timer
+DL_TIMER_ROOT?=$(ROOT_DIR)/../dl_timer
 # Needs to link with openmp because the dl_timer library needs openmp.
 dl_timer:
 	$(MAKE) WRAPPER_DIR="$(PROFILE_DIR)/dl_timer"                     \

examples/gocean/eg5/profile/README.md

@@ -56,8 +56,8 @@ The location of the dl_esm_inf infrastructure library, it defaults to
 which is the version included in PSyclone.
 
 ### DL_TIMER_ROOT:
-The location of the apeg-dl_timer library. It defaults to
-``../../../../../apeg-dl_timer``, i.e. it is assumed that apeg-dl_timer
+The location of the dl_timer library. It defaults to
+``../../../../../dl_timer``, i.e. it is assumed that dl_timer
 is installed next to PSyclone.
 Note that until Issue #730 is complete, executing this example
 will fail as the labels produced by PSyclone are longer than

examples/nemo/eg3/Makefile

@@ -37,7 +37,7 @@
 # Makefile for the 3rd NEMO example. Uses PSyclone (which must
 # be installed) to generate Fortran with OpenACC directives for
 # the tracer-advection benchmark. The 'compile' target requires
-# the dl_timer library (bitbucket.org/apeg/dl_timer) which must
+# the dl_timer library (github.com/stfc/dl_timer) which must
 # be installed and the location specified in DL_TIMER_ROOT below.
 #
 # The compiler to use must be specified via the F90 environment

examples/nemo/eg3/README.md

@@ -49,8 +49,8 @@ directives using PSyclone with the NEMO API. It is not intended to
 demonstrate how to obtain good performance.
 
 Since `tra_adv.F90` is instrumented for use with the dl_timer library,
-this library is also required. It is available from
-[bitbucket.org/apeg/dl_timer](https://bitbucket.org/apeg/dl_timer).
+this library is also required. It is available on
+[github](https://github.com/stfc/dl_timer).
 
 Once dl_timer has been downloaded, the supplied Makefile must be
 edited to supply the location of the library. The compiler and flags

examples/nemo/scripts/utils.py

@@ -499,6 +499,7 @@ class MaximalProfilingOutsideDirectivesTrans(MaximalRegionTrans):
         # (which would create unclosed hooks).
         _allowed_contiguous_statements = (Assignment, Call, CodeBlock)
         _transformation = ProfileTrans
+        _SUB_TRANSFORMATIONS = [ProfileTrans]
 
         def _satisfies_minimum_region_rules(self, region: list[Node]) -> bool:
             '''Returns whether the provided node list satisfies the

lib/profiling/README.md

@@ -52,7 +52,7 @@ Example output:
 This is a wrapper library that maps the [PSyclone profiling API](
 https://psyclone.readthedocs.io/en/latest/user_guide/profiling.html#profiling) to the
 dl_timer API. A copy of dl_timer can be downloaded from
-https://bitbucket.org/apeg/dl_timer.
+https://github.com/stfc/dl_timer.
 
 The PSyclone dl_timer wrapper library uses the ``ProfileData`` type and
 dl_timer's timer_register function to store the module/region name and

lib/profiling/dl_timer/Makefile

@@ -38,11 +38,11 @@
 # Compiler and compiler flags
 F90 ?= gfortran
 F90FLAGS ?=
-# Path to the 'apeg/dl_timer' installation. It defaults to the assumed
-# location of the apeg/dl_timer library next to a clone of PSyclone
+# Path to the 'dl_timer' installation. It defaults to the assumed
+# location of the dl_timer library next to a clone of PSyclone
 # repository. Overwrite for a custom location.
 DL_TIMER_ROOT ?= $(PSYDATA_LIB_DIR)/../../dl_timer
-# Path to the 'apeg/dl_timer' "include" files. Overwrite for a custom location.
+# Path to the 'stfc/dl_timer' "include" files. Overwrite for a custom location.
 DL_TIMER_INCLUDE ?= $(DL_TIMER_ROOT)/src
 # Path to the PSyclone wrapper libraries. It defaults to the relative path to
 # the top-level 'lib' directory. Overwrite for a custom location.

lib/profiling/dl_timer/README.md

@@ -7,7 +7,7 @@ dl_timer API. This library is **thread-safe**.
 ## Dependencies
 
 The library dl_timer must be installed, which can be downloaded from
-https://bitbucket.org/apeg/dl_timer.
+https://github.com/stfc/dl_timer.
 
 This profiling library uses the [PSyData API](
 https://psyclone.readthedocs.io/en/latest/user_guide/psy_data.html) to interface with
@@ -37,7 +37,7 @@ To compile the PSyclone wrapper library for dl_timer, one of the
 following two ``Makefile`` variables must be set to specify the path to
 the dl_timer installation:
 
-- ``DL_TIMER_ROOT``, the path to the ``apeg-dl_timer`` directory in which
+- ``DL_TIMER_ROOT``, the path to the ``stfc/dl_timer`` directory in which
   dl_timer is compiled. It defaults to ``./../../../../dl_timer`` in
   the ``Makefile`` (i.e., it assumes dl_timer is installed next to a
   PSyclone clone).

lib/profiling/dl_timer/dl_timer.f90

@@ -34,7 +34,7 @@
 
 
 !> An implementation of the PSyData API for profiling, which wraps the use
-!> of the dl_timer library (https://bitbucket.org/apeg/dl_timer).
+!> of the dl_timer library (https://github.com/stfc/dl_timer).
 
 module profile_psy_data_mod
 

src/psyclone/psyGen.py

@@ -2344,6 +2344,11 @@ class Transformation(metaclass=abc.ABCMeta):
         "User guide for more details."
     )
 
+    #: List of transformations called inside this one that need to be
+    #: considered by the split_kwargs infrastructure when propagating
+    #: transformation options
+    _SUB_TRANSFORMATIONS = []
+
     @property
     def name(self):
         '''
@@ -2353,6 +2358,34 @@ def name(self):
         '''
         return type(self).__name__
 
+    def split_kwargs(self, **kwargs) -> tuple[dict[str, Any]]:
+        '''
+        :param kwargs: the list of kwargs to split.
+
+        :returns: a tuple of the kwargs dictionaries that are valid for this
+            transformation and every other transformation listed in the
+            _SUB_TRANSFORMATIONS list. The first kwargs (the ones for itself)
+            will also include any key that is not valid in any of the other
+            transformation (this is done to ensure one of the validate_options
+            reports invalid options when those are provided).
+        '''
+        # The first kwargs starts with all the items
+        first_dict = dict(kwargs)
+        # The following kwargs start empty
+        other_dicts = [{} for _ in self._SUB_TRANSFORMATIONS]
+
+        # Now copy each valid item into the transformation-specific kwargs
+        # and delete them from the first one if they are valid somewhere
+        # else but not in the self options
+        for key in kwargs:
+            for idx, trans in enumerate(self._SUB_TRANSFORMATIONS):
+                if key in trans.get_valid_options():
+                    other_dicts[idx][key] = kwargs[key]
+                    if key not in type(self).get_valid_options():
+                        del first_dict[key]
+
+        return first_dict, *other_dicts
+
     @abc.abstractmethod
     def apply(self, node, options=None, **kwargs):
         '''Abstract method that applies the transformation. This function
@@ -2509,10 +2542,18 @@ def validate_options(self, **kwargs):
             for invalid in invalid_options:
                 invalid_options_detail.append(f"'{invalid}'")
             invalid_options_list = ", ".join(invalid_options_detail)
+            extra_options = ""
+            if self._SUB_TRANSFORMATIONS:
+                sub_trans_names = [
+                    tr.__name__ for tr in self._SUB_TRANSFORMATIONS
+                ]
+                extra_options = (
+                    f" or any other options supported in {sub_trans_names}"
+                )
             raise ValueError(f"'{type(self).__name__}' received invalid "
                              f"options [{invalid_options_list}]. "
                              f"Valid options are "
-                             f"'{list(valid_options.keys())}.")
+                             f"{list(valid_options.keys())}{extra_options}.")
         if len(wrong_types.keys()) > 0:
             wrong_types_detail = []
             for name in wrong_types.keys():

src/psyclone/psyir/transformations/maximal_omp_parallel_region_trans.py

@@ -65,6 +65,7 @@ class MaximalOMPParallelRegionTrans(MaximalRegionTrans):
     the discussion on #3205 for more detail.'''
     # The type of parallel transformation to be applied to the input region.
     _transformation = OMPParallelTrans
+    _SUB_TRANSFORMATIONS = [OMPParallelTrans]
     # Tuple of statement nodes allowed inside the _transformation
     _allowed_contiguous_statements = (
         OMPTaskwaitDirective,

src/psyclone/psyir/transformations/maximal_region_trans.py

@@ -31,12 +31,13 @@
 # ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
 # POSSIBILITY OF SUCH DAMAGE.
 # -----------------------------------------------------------------------------
-# Authors A. B. G. Chalk, STFC Daresbury Lab
+# Author: A. B. G. Chalk, STFC Daresbury Lab
+# Modified: S. Siso, STFC Daresbury Lab
 
 '''This module contains the MaximalRegionTrans.'''
 
 import abc
-from typing import Union
+from typing import Union, Any
 
 from psyclone.psyir.nodes import (
     Node,
@@ -148,13 +149,15 @@ def _can_be_in_region(self, node: Node) -> bool:
     def _compute_transformable_sections(
             self, node_list: list[Node],
             trans: Transformation,
+            trans_kwargs: dict[str, Any]
     ) -> list[list[Node]]:
         '''
         Computes the sections of the input node_list to apply the
         transformation to.
 
         :param node_list: The node_list passed into this Transformation.
         :param trans: The transformation applied to the regions found.
+        :param trans_kwargs: The kwargs applied to the transformation.
         :returns: The list of node_lists to apply this class'
             _transformation class to.
         '''
@@ -168,7 +171,7 @@ def _compute_transformable_sections(
                 # Check that validation still succeeds if we add this child
                 # to the current block.
                 try:
-                    trans.validate(current_block + [child])
+                    trans.validate(current_block + [child], **trans_kwargs)
                     current_block.append(child)
                 except TransformationError:
                     # If validation now fails, then don't add this to the
@@ -189,17 +192,17 @@ def _compute_transformable_sections(
                 # Need to recurse on some node types
                 if isinstance(child, IfBlock):
                     if_blocks = self._compute_transformable_sections(
-                            child.if_body, trans
+                            child.if_body, trans, trans_kwargs
                     )
                     all_blocks.extend(if_blocks)
                     if child.else_body:
                         else_blocks = self._compute_transformable_sections(
-                            child.else_body, trans
+                            child.else_body, trans, trans_kwargs
                         )
                         all_blocks.extend(else_blocks)
                 if isinstance(child, (Loop, WhileLoop)):
                     loop_blocks = self._compute_transformable_sections(
-                        child.loop_body, trans
+                        child.loop_body, trans, trans_kwargs
                     )
                     all_blocks.extend(loop_blocks)
         # If any nodes are left in the current block at the end of the
@@ -220,7 +223,8 @@ def validate(self, nodes: Union[Node, Schedule, list[Node]], **kwargs):
                                      same parent and aren't consecutive.
         '''
 
-        self.validate_options(**kwargs)
+        self_kwargs, _ = self.split_kwargs(**kwargs)
+        self.validate_options(**self_kwargs)
         node_list = self.get_node_list(nodes)
 
         node_parent = node_list[0].parent
@@ -247,11 +251,13 @@ def apply(self, nodes: Union[Node, Schedule, list[Node]], **kwargs):
 
         # Call validate.
         self.validate(nodes, **kwargs)
+        _, tr_kwargs = self.split_kwargs(**kwargs)
 
         par_trans = self._transformation()
 
-        all_blocks = self._compute_transformable_sections(node_list, par_trans)
+        all_blocks = self._compute_transformable_sections(
+            node_list, par_trans, tr_kwargs)
 
         # Apply the transformation to all of the blocks found.
         for block in all_blocks:
-            par_trans.apply(block)
+            par_trans.apply(block, **tr_kwargs)