cfipython file changes in 14_1_0_pre4 (compared to pre3) lead to failures in ConfDb parsing

[Martin-Grunewald]

Between pre3 and pre4 of 14_1_0, there is the following change in fillDescriptions-generated cfipython files, for example:

pre3:

cat $CMSSW_RELEASE_BASE/cfipython/el8_amd64_gcc12/Alignment/CommonAlignment/apvModeFilter_cfi.py 
import FWCore.ParameterSet.Config as cms

apvModeFilter = cms.EDFilter('APVModeFilter',
  apvMode = cms.untracked.string('deco'),
  mightGet = cms.optional.untracked.vstring
)

while in pre4:

cat $CMSSW_RELEASE_BASE/cfipython/el8_amd64_gcc12/Alignment/CommonAlignment/apvModeFilter_cfi.py 
import FWCore.ParameterSet.Config as cms

from .APVModeFilter import APVModeFilter

apvModeFilter = APVModeFilter()

The pre4 version breaks ConfDb parsing with the error message:

Scanning package: /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre4//cfipython/el8_amd64_gcc12/Alignment/CommonAlignment
Traceback (most recent call last):
  File "/data/user/gruen/ConfDb/CMSSW_14_1_0_pre4/hlt-confdb/parser/./ConfdbLoadParamsFromConfigsPy3.py", line 1879, in <module>
    main(sys.argv[1:])
  File "/data/user/gruen/ConfDb/CMSSW_14_1_0_pre4/hlt-confdb/parser/./ConfdbLoadParamsFromConfigsPy3.py", line 191, in main
    confdbjob.BeginJob()
  File "/data/user/gruen/ConfDb/CMSSW_14_1_0_pre4/hlt-confdb/parser/./ConfdbLoadParamsFromConfigsPy3.py", line 483, in BeginJob
    self.ExtendTheCfi(validatedpyfile, validatedpydir, allowmultiplecfis, usepythonsubdir)
  File "/data/user/gruen/ConfDb/CMSSW_14_1_0_pre4/hlt-confdb/parser/./ConfdbLoadParamsFromConfigsPy3.py", line 717, in ExtendTheCfi
    exec(importcommand)
  File "<string>", line 1, in <module>
  File "/cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre4//cfipython/el8_amd64_gcc12/Alignment/CommonAlignment/apvModeFilter_cfi.py", line 3, in <module>
    from .APVModeFilter import APVModeFilter
ImportError: attempted relative import with no known parent package

Clearly this is a problem.

It seems the issue is with from .APVModeFilter import APVModeFilter, the leading '.' syntax.
Can it be reverted or done differently?
For example, defining the parent package explicitly in the py file, or giving a full path in the import statement?

[Martin-Grunewald]

assign core

[cmsbuild]

New categories assigned: core

@Dr15Jones,@makortel,@smuzaffar you have been requested to review this Pull request/Issue and eventually sign? Thanks

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @Martin-Grunewald.

@antoniovilela, @rappoccio, @Dr15Jones, @smuzaffar, @sextonkennedy, @makortel can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[mmusich]

From a quick look, seems #44424 and #44363 caused this.

[Martin-Grunewald]

The generated cfipython files no longer seem to be self-contained or fully-defined python files:

pre4:

python3 $CMSSW_RELEASE_BASE/cfipython/el8_amd64_gcc12/Alignment/CommonAlignment/apvModeFilter_cfi.py
Traceback (most recent call last):
  File "/cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_0_pre4/cfipython/el8_amd64_gcc12/Alignment/CommonAlignment/apvModeFilter_cfi.py", line 3, in <module>
    from .APVModeFilter import APVModeFilter
ImportError: attempted relative import with no known parent package

while there is no error in pre3:

python3 $CMSSW_RELEASE_BASE/cfipython/el8_amd64_gcc12/Alignment/CommonAlignment/apvModeFilter_cfi.py

I hope this can be fixed!

[Martin-Grunewald]

The following makes it work:

cat apvModeFilter_cfi.py

import FWCore.ParameterSet.Config as cms

from Alignment.CommonAlignment.APVModeFilter import APVModeFilter

apvModeFilter = APVModeFilter()

ie, with a fully qualified import path name, python3 apvModeFilter_cfi.py does not show an error.

[mmusich]

presumably a fix is needed here then:

cmssw/FWCore/ParameterSet/src/ConfigurationDescriptions.cc

Line 274 in ab77a26

outFile << "from ." << pythonName << " import " << pythonName << "\n\n";

(changes introduced at #44363)

[Dr15Jones]

It seems the issue is with from .APVModeFilter import APVModeFilter, the leading '.' syntax.
Can it be reverted or done differently?

The problem is ConfDB is not properly following the python rules for doing an import. If one were going to use a subset of numpy you would not go into some numpy directory and doimport bar and expect that to work. The python rules require that you must do import numpy.foo.bar.

What I believe needs to be changed is this line in ConfDB
https://github.com/cms-sw/hlt-confdb/blob/6ade11b6afa09b185585b73ea47bbfc1c60607c4/parser/ConfdbLoadParamsFromConfigsPy3.py#L705

which should be like the line two above it
https://github.com/cms-sw/hlt-confdb/blob/6ade11b6afa09b185585b73ea47bbfc1c60607c4/parser/ConfdbLoadParamsFromConfigsPy3.py#L703

@Martin-Grunewald how do I checkout and run ConfDB to see if that change is all that is needed?

[Martin-Grunewald]

Sorry, your reply is confusing: the py file is not even working with python3 <file>, because it does NOT contain
from Alignment.CommonAlignment.APVModeFilter import APVModeFilter
which is what I understand you advocate (import numpy.foo.bar).

Could you please clarify?

[Martin-Grunewald]

Can we make python3 file working first?

[Dr15Jones]

So I did

source /cvmfs/cms.cern.ch/cmsset_default.csh 
scram project CMSSW_14_1_0_pre4
cd CMSSW_14_1_0_pre4
cmsenv
python3
Python 3.9.14 (main, Apr 29 2024, 23:37:22) 
[GCC 12.3.1 20230527] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from Alignment.CommonAlignment.APVModeFilter import APVModeFilter
>>> print(APVModeFilter)
<function APVModeFilter at 0x7f5c4e148af0>

[Dr15Jones]

This also works

[cdj@cmslpc-el8-heavy01 CMSSW_14_1_0_pre4]$ python3
Python 3.9.14 (main, Apr 29 2024, 23:37:22) 
[GCC 12.3.1 20230527] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from Alignment.CommonAlignment.modules import APVModeFilter
>>> print(APVModeFilter)
<FWCore.ParameterSet.ModulesProxy._ModuleProxy object at 0x7fae552169d0>